{"id":4222696,"url":"https://alion.io/job/unitedhealth-group-ai-or-ml-engineer-python-qa-testing","title":"AI or ML Engineer - Python, QA Testing","company":{"id":1043512,"name":"UnitedHealth Group","domain":"unitedhealthgroup.com","url":"https://alion.io/company/unitedhealthgrp","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Radancy","truth_index":{"grade":"B","score":75,"open_postings":178,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-11-12T10:38:37Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Hyderabad, India"],"countries":["IN"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agile","optional":false},{"name":"AI Agents","optional":false},{"name":"CI/CD","optional":false},{"name":"Function Calling","optional":false},{"name":"LLM","optional":false},{"name":"Playwright","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Pytest","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"A/B Testing","optional":true},{"name":"Arize Phoenix","optional":true},{"name":"AWS","optional":true},{"name":"Azure","optional":true},{"name":"BrowserStack","optional":true},{"name":"DeepEval","optional":true},{"name":"Dynatrace","optional":true},{"name":"GCP","optional":true},{"name":"Gemini","optional":true},{"name":"HIPAA","optional":true},{"name":"JavaScript","optional":true},{"name":"JMeter","optional":true},{"name":"Langfuse","optional":true},{"name":"LangSmith","optional":true},{"name":"Model Context Protocol","optional":true},{"name":"OpenTelemetry","optional":true},{"name":"OWASP Top 10","optional":true},{"name":"Ragas","optional":true},{"name":"Selenium","optional":true},{"name":"SonarQube","optional":true},{"name":"TypeScript","optional":true},{"name":"Vertex AI","optional":true}],"status":"live","first_seen_at":"2026-11-13T06:07:55Z","employer_posted_date":"2026-11-12","last_verified_at":"2026-11-13T06:07:55Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.\nWe are looking for an AI Quality Engineer to help ensure our agentic AI systems are accurate, reliable, secure, and safe. You will define and execute a quality strategy spanning manual testing, test automation, release evaluation, and production monitoring. You will build an automation framework for agent responses, retrieval, tool use, and multi-step workflows, and work with engineering and product teams to turn failures into lasting improvements.\nPrimary Responsibilities:\nDefine and execute a risk-based test strategy covering functional, API, integration, end-to-end, regression, performance, reliability, safety, and security testing for AI agents\nBuild and maintain automated evaluation and test frameworks for prompts, model outputs, retrieval, tool calls, agent handoffs, and multi-step workflows\nPerform manual exploratory and scenario-based testing of new capabilities, ambiguous requests, edge cases, and real-world failure modes\nCreate and maintain representative test datasets, expected outcomes, and evaluation rubrics. Use deterministic assertions, LLM-as-judge, and human review where each is appropriate\nRun quality checks for every release and for material prompt, model, tool, or data-source change. Integrate repeatable tests into CI/CD pipelines as release gates\nMonitor production quality using measures such as task completion, answer accuracy, groundedness, hallucinations, tool-call success, latency, cost, and user feedback\nDetect drift and quality regressions, set alert thresholds, investigate failures, and expand test coverage based on production findings\nEvaluate risks including prompt injection, sensitive-data exposure, bias, and unauthorized or unsafe agent actions\nPublish quality dashboards and release-readiness reports. Partner with engineering, product, and SRE teams on defect triage, root-cause analysis, and corrective actions\nComply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements\nRequired Qualifications:\nBachelor's degree in Computer Science or a related field, or equivalent practical experience\n5+ years of experience in software quality engineering or test automation, including 2+ years testing AI/ML, generative AI, or LLM-based systems\nHands-on experience evaluating LLM applications with test datasets, rubrics, groundedness or relevance checks, and a combination of automated and human review\nExperience designing and executing manual and automated functional, API, integration, end-to-end, regression, and risk-based tests\nExperience integrating automated quality checks into CI/CD pipelines and testing APIs, databases, and distributed applications\nExperience using logs, traces, dashboards, alerts, and user feedback to investigate production quality issues\nSolid Python skills and experience building or extending test automation frameworks, such as Pytest or Playwright\nWorking knowledge of QA methods, defect lifecycle management, Agile delivery, and the software development lifecycle\nUnderstanding of RAG, prompt engineering, tool use, and multi-step agent workflows\nDemonstrated solid analytical, problem-solving, communication, and collaboration skills\nPreferred Qualifications:\nExperience with evaluation or observability tools such as LangSmith, Langfuse, Arize, DeepEval, Ragas, Vertex AI evaluation, or OpenTelemetry\nExperience with GCP, Vertex AI, or Gemini; experience testing applications on AWS or Azure\nExperience with JavaScript/TypeScript or Java, Selenium, advanced Playwright framework development, or browser testing platforms such as BrowserStack or Sauce Labs\nExperience with adversarial testing, responsible AI, and the OWASP Top 10 for LLM Applications\nExperience with A/B testing or canary evaluation of prompt and model changes\nExperience testing conversational AI, multi-agent or MCP-enabled solutions, event-driven services, or API orchestration\nExperience using AI to assist with test generation, failure analysis, or quality reporting\nExperience with performance and load testing tools such as JMeter or LoadRunner, or operational tools such as Dynatrace or SonarQube\nExperience with cloud migration, data migration, or data reconciliation testing\nHealthcare or other regulated-industry experience, including knowledge of HIPAA, data privacy, and security requirements. Experience with claims, eligibility, prior authorization, provider, contact center, or digital health systems\nAt UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes - an enterprise priority reflected in our mission.","description_format":"text","description_chars":5772,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":true},"security_clearance":false,"languages":[]},"benefits":["Flexible schedule"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Hospitals & Health Systems","Pharmacies","Health Insurance & Benefits"],"lifecycle":[{"event":"open","at":"2026-11-13T06:07:55Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":28,"reasons":["conf:1","win:early","comp:brand"],"computed_at":"2026-11-13T07:43:49Z"},"pay":null,"html_url":"https://alion.io/job/unitedhealth-group-ai-or-ml-engineer-python-qa-testing","json_url":"https://alion.io/job/unitedhealth-group-ai-or-ml-engineer-python-qa-testing.json","meta":{"generated_at":"2026-11-13T07:43:49Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":16126,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-14T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1043512},"rest":"https://alion.io/mcp/rest/get_company?id=1043512"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Funitedhealth-group-ai-or-ml-engineer-python-qa-testing"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Funitedhealth-group-ai-or-ml-engineer-python-qa-testing"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Funitedhealth-group-ai-or-ml-engineer-python-qa-testing"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/unitedhealth-group-ai-or-ml-engineer-python-qa-testing\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Funitedhealth-group-ai-or-ml-engineer-python-qa-testing"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":4222696}},{"tool":"search_jobs","arguments":{"company":"unitedhealthgroup.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":4222696},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}