{"id":2597454,"url":"https://alion.io/job/photon-qa-lead-automationperformance-dallas-tx","title":"QA Lead (Automation+Performance)- Dallas, TX","company":{"id":3186451,"name":"Photon","domain":"photon.com","url":"https://alion.io/company/photon-com","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Oracle","truth_index":{"grade":"D","score":51,"open_postings":701,"ghost_share":0.812,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-10-24T06:02:35Z"}},"role":"QA","role_family":"QA","seniority":"lead","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Dallas, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":59000,"max_usd":131000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":152},"experience_years_min":10,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"CI/CD","optional":false},{"name":"Claude","optional":false},{"name":"Cypress","optional":false},{"name":"DeepEval","optional":false},{"name":"Function Calling","optional":false},{"name":"GPT-4","optional":false},{"name":"Hallucination","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"Playwright","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Promptfoo","optional":false},{"name":"Pytest","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Selenium","optional":false},{"name":"SQL","optional":false},{"name":"Multi-Agent Systems","optional":true}],"status":"live","first_seen_at":"2026-04-15T00:50:13Z","employer_posted_date":"2026-04-15","last_verified_at":"2026-10-24T21:36:48Z","board_verified":true,"closed_at":null,"days_open":192,"trust":{"level":"ghost","repost_count":0,"flags":["stale","company_stale"],"days_open":192},"description":"We are seeking a QA Automation Lead who is ready to move beyond traditional \"Pass/Fail\" testing. In this role, you will design and build automation frameworks specifically for Agentic AI products. You will focus on evaluating the performance of autonomous agents, ensuring they follow logical reasoning paths, call the correct tools, and provide accurate, safe outputs.\nYour mission is to build the \"evaluations\" (Evals) that define what high-quality AI behavior looks like, moving the needle from unpredictable experiments to production-grade software.\nKey Responsibilities\nNon-Deterministic Testing: Develop automation strategies for probabilistic outputs, using model-based evaluation to \"test the tester.\"\nBuilding \"Eval\" Pipelines: Create and maintain \"Golden Datasets\" to benchmark agent performance across different versions of prompts and models.\nTool-Use Validation: Build automated tests to verify that agents call the correct functions/APIs with the right parameters in complex multi-step workflows.\n Regression Testing for Prompts: Monitor how subtle changes in prompt engineering or model updates (e.g., moving from GPT-4 to Claude 3.5) affect the product’s reliability.\nLatency & Token Monitoring: Integrate performance testing into the CI/CD pipeline to track agent reasoning time and cost-efficiency.\nHallucination Detection: Develop automated checks to identify and report AI hallucinations, bias, or \"jailbreak\" attempts.\nCollaboration: Work closely with AI Engineers to translate \"vague\" business requirements into measurable, automated test cases.\nRequired Skills & Qualifications\nExperience: 10+ years in QA Automation, with a recent focus on AI/ML or LLM-based applications.\n Python Proficiency: Expert-level Python skills (the industry standard for AI testing) and experience with testing frameworks like Pytest.\nAI Testing Tools: Familiarity with AI evaluation frameworks such as LangSmith, DeepEval, RAGAS, or Promptfoo.\nAPI & Backend Testing: Deep experience with Playwright, Selenium, or Cypress for UI, but a heavy focus on API-level testing and database validation.\n Statistical Mindset: Understanding that AI testing often requires \"scoring\" (e.g., 85% accuracy) rather than a simple binary pass/fail.\n Data Skills: Ability to work with SQL and JSON to validate data retrieved by agents during RAG (Retrieval-Augmented Generation) processes.\nPreferred Qualifications\nExperience testing Multi-Agent Systems (where one agent tests another).\nKnowledge of Prompt Engineering and how it influences software behavior.\nBackground in Investment Banking or Fintech (if applicable) to understand high-stakes data accuracy.\nCompensation, Benefits and Duration\nMinimum Compensation: USD 38,000\nMaximum Compensation: USD 133,000\nCompensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role.\nMedical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees.\nThis position is not available for independent contractors\nNo applications will be considered if received more than 120 days after the date of this post","description_format":"text","description_chars":3201,"description_truncated":false,"requirements":{"experience_years_min":10,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["401k plan","Retirement plans"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["IT Consulting & Digital Transformation"],"lifecycle":[{"event":"open","at":"2026-10-15T07:30:57Z"}],"visa":[],"liveness":{"score":5,"band":"cold","label":"Long shot","p_open":1,"p_active":0.188,"p_room":0.28,"age_days":192,"expected_fill_days":23,"reasons":["conf:1","stale_co","velocity","ghost","win:tail","crowd:brand"],"computed_at":"2026-10-24T06:02:35Z"},"pay":null,"html_url":"https://alion.io/job/photon-qa-lead-automationperformance-dallas-tx","json_url":"https://alion.io/job/photon-qa-lead-automationperformance-dallas-tx.json","meta":{"generated_at":"2026-10-24T21:45:57Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":31954,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-25T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3186451},"rest":"https://alion.io/mcp/rest/get_company?id=3186451"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fphoton-qa-lead-automationperformance-dallas-tx"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fphoton-qa-lead-automationperformance-dallas-tx"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fphoton-qa-lead-automationperformance-dallas-tx"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/photon-qa-lead-automationperformance-dallas-tx\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fphoton-qa-lead-automationperformance-dallas-tx"}]}