{"id":2141649,"url":"https://alion.io/job/evolent-senior-data-scientist-ai-evaluation-improvement","title":"Senior Data Scientist (AI Evaluation & Improvement)","company":{"id":1040035,"name":"Evolent","domain":"evolent.com","url":"https://alion.io/company/evolent","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":null},"role":"Data Science","role_family":"Data Science","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":135000,"max":165000,"currency":"USD","period":"year","gross":null,"usd_annual":165000},"salary_estimate":null,"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"A/B Testing","optional":false},{"name":"Hypothesis","optional":false},{"name":"Langfuse","optional":false},{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"Python","optional":false},{"name":"SQL","optional":false}],"status":"live","first_seen_at":"2026-10-08T00:00:00Z","employer_posted_date":"2026-10-15","last_verified_at":"2026-10-22T22:16:46Z","board_verified":true,"closed_at":null,"days_open":15,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":15},"description":"Your Future Evolves Here\nEvolent partners with health plans and providers to achieve better outcomes for people with most complex and costly health conditions. Working across specialties and primary care, we seek to connect the pieces of fragmented health care system and ensure people get the same level of care and compassion we would want for our loved ones.\nEvolent employees enjoy work/life balance, the flexibility to suit their work to their lives, and autonomy they need to get things done. We believe that people do their best work when they're supported to live their best lives, and when they feel welcome to bring their whole selves to work. That's one reason why diversity and inclusion are core to our business.\nJoin Evolent for the mission. Stay for the culture.\nWhat You’ll Be Doing:\nCore Responsibilities\nOwn the diagnostic loop for LLM-based clinical services: take failure modes surfaced by clinical reviewers and product managers, form root-cause hypotheses (prompt design, context assembly, retrieval, guideline encoding, model behavior, upstream data), and design experiments that isolate the cause.\nTest candidate fixes - prompt and configuration variants, context changes, model alternatives - and verify improvements with structured evaluations, not anecdotes; confirm fixes don't regress other behavior.\nOwn the evaluation roadmap and quality bar for the team's AI services: which metrics gate deployment, how golden sets and regression suites grow, and what \"good enough to ship\" means in evidence.\nUse and extend the team's evaluation platform: build golden sets and regression suites, define metrics (accuracy, guideline adherence, grounding/faithfulness), run and interpret eval batteries. Fluency in *using* modern eval tooling matters; the platform exists - extending it thoughtfully is in scope, rebuilding it is not.\nPartner with clinical reviewers (medical directors) to turn review findings into labeled evidence and executable evaluation criteria; partner with product managers to prioritize which failure modes matter most.\nMentor others in evaluation methods and grow the evals function as it scales, including readiness to take direct reports as the team expands.\nSupport the annual clinical-guidelines update cycle with regression evaluation as guidelines, prompts, and models change; ramp with the team's senior data scientists in Q4 2026.\nDocument the diagnostic playbook: failure taxonomies, experiment templates, variant history - a method others can run, not a private intuition. This is a practicing role - the diagnostic loop is the job, at every level of seniority.\nHandle clinical data (including PHI) according to organizational security, privacy, and compliance requirements.\nMinimum Requirements\nBachelor's degree in Data Science, Computer Science, Statistics, or a related quantitative field - or equivalent experience.\n5+ years of data science or applied machine learning experience, including shipping and maintaining models or AI systems in production.\n2+ years of recent, hands-on experience evaluating and improving LLM-based systems: structured error analysis, prompt/configuration iteration, experiment design, metrics interpretation - as a practitioner, not only as a reviewer of others' work.\nDemonstrated experience with LLM evaluation methods and tools - golden/regression sets, LLM-as-judge with validation, tracing and observability tooling.\nStrong Python and solid data-analysis skills (SQL a plus); comfort computing and reasoning about metrics such as sensitivity, specificity, and PPV.\nHypothesis-driven working style: the instinct to isolate variables and prove a fix, rather than tweak and hope.\nStrong written communication - findings and go/no-go evidence must be legible to engineers, clinicians, and leadership.\nPreferred Requirements\nHealthcare experience: utilization management, prior authorization, clinical documentation, or clinical/claims data; comfort reading clinical guideline and medical-policy content.\nExperience mentoring data scientists or leading small technical workstreams; interest in growing into people leadership as a function scales.\nExperience working with clinical reviewers or other domain experts to convert expert judgment into labeled data and evaluation criteria.\nExperience with LLM observability/telemetry stacks (OpenTelemetry-based tracing, Logfire, Langfuse, or similar).\nStatistics or experimentation background (A/B testing, statistical significance, sample-size reasoning).\nMaster's degree in a quantitative field.\nTo ensure a secure hiring process we have implemented several identity verification steps, including submission of a government issued photo ID. We conduct identity verification during interviews, and final interviews may require onsite attendance. All candidates must complete a comprehensive background check, in-person I-9 verification, and may be subject to drug screening prior to employment. The use of artificial intelligence tools during interviews is prohibited and monitored. Misrepresentation will result in immediate disqualification from consideration.\nTechnical Requirements:\nWe require that all employees have the following technical capability at their home: High speed internet over 10 Mbps and, specifically for all call center employees, the ability to plug in directly to the home internet router.\nEvolent is an equal opportunity employer and considers all qualified applicants equally without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran status, or disability status. If you need reasonable accommodation to access the information provided on this website, please contact  for further assistance.\nThe expected base salary/wage range for this position is $135,000 - 165,000. As part of our total compensation package, Evolent is proud to offer comprehensive benefits (including health insurance benefits) to qualifying employees. All compensation determinations are based on the skills and experience required for the position and commensurate with experience of selected individuals, which may vary above and below the stated amounts.","description_format":"text","description_chars":6166,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":["Health insurance"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Care Coordination & Population Health","Clinical Decision Support","Health Insurance & Benefits"],"lifecycle":[{"event":"open","at":"2026-10-09T03:20:26Z"}],"visa":[],"liveness":{"score":44,"band":"fade","label":"Fading","p_open":1,"p_active":0.804,"p_room":0.55,"age_days":14,"expected_fill_days":12,"reasons":["conf:13","velocity","win:tail"],"computed_at":"2026-10-22T06:10:10Z"},"pay":{"stated_usd_annual":165000,"is_top_pay":true},"html_url":"https://alion.io/job/evolent-senior-data-scientist-ai-evaluation-improvement","json_url":"https://alion.io/job/evolent-senior-data-scientist-ai-evaluation-improvement.json","meta":{"generated_at":"2026-10-23T05:14:04Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":8617,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-24T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":1040035},"rest":"https://alion.io/mcp/rest/get_company?id=1040035"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fevolent-senior-data-scientist-ai-evaluation-improvement"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fevolent-senior-data-scientist-ai-evaluation-improvement"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fevolent-senior-data-scientist-ai-evaluation-improvement"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/evolent-senior-data-scientist-ai-evaluation-improvement\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fevolent-senior-data-scientist-ai-evaluation-improvement"}]}