{"id":3711663,"url":"https://alion.io/job/sedgwick-ai-evaluation-engineer","title":"AI Evaluation Engineer","company":{"id":5707,"name":"Sedgwick","domain":"sedgwick.com","url":"https://alion.io/company/sedgwick","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Workday","truth_index":{"grade":"B","score":84,"open_postings":52,"ghost_share":0,"stale_share":0.635,"repost_share":0.019,"time_to_fill_p50_days":24,"computed_at":"2026-11-04T06:34:25Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"middle","employment_type":null,"work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"posting_text","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["United States"],"countries":["US"],"hiring_countries":["US"],"hiring_countries_total":1,"salary":null,"salary_estimate":null,"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Arize Phoenix","optional":false},{"name":"AWS","optional":false},{"name":"Azure","optional":false},{"name":"CI/CD","optional":false},{"name":"DeepEval","optional":false},{"name":"Function Calling","optional":false},{"name":"GCP","optional":false},{"name":"Hallucination","optional":false},{"name":"Langfuse","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"LLMOps","optional":false},{"name":"MLFlow","optional":false},{"name":"NIST AI RMF","optional":false},{"name":"NLP","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Promptfoo","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Ragas","optional":false},{"name":"Synthetic Data","optional":false}],"status":"live","first_seen_at":"2026-11-03T23:29:37Z","employer_posted_date":"2026-11-03","last_verified_at":"2026-11-05T00:43:39Z","board_verified":true,"closed_at":null,"days_open":1,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":1},"description":"By joining Sedgwick, you'll be part of something truly meaningful. It’s what our 33,000 colleagues do every day for people around the world who are facing the unexpected. We invite you to grow your career with us, experience our caring culture, and enjoy work-life balance. Here, there’s no limit to what you can achieve.\nNewsweek Recognizes Sedgwick as America’s Greatest Workplaces National Top Companies\nCertified as a Great Place to Work®\nFortune Best Workplaces in Financial Services & Insurance\nAI Evaluation EngineerPRIMARY PURPOSE: The AI Evaluation Engineer designs, builds, and operates the evaluation systems that determine whether our GenAI solutions are accurate, safe, reliable, and ready for production. You will serve multiple product pods, establishing consistent evaluation standards, automated pipelines, and quality gates, and you will help teams measure and improve LLM, RAG, and agentic applications throughout their lifecycle.\nFully remote role, open to U.S. based candidates.\nESSENTIAL FUNCTIONS AND RESPONSIBILITIES\nEstablishes standardized, automated evaluation frameworks for LLMs and traditional ML models, including LLM as a judge patterns, ROUGE, METEOR, and statistical validation to ensure performance, reliability, and readiness prior to production release.\nDeploys comprehensive observability solutions to monitor model drift, hallucination risk, latency, throughput, and token consumption, balancing model quality with infrastructure cost and operational efficiency.\nDefine evaluation methodologies and metrics for LLM, RAG, and agent-based applications (e.g., faithfulness, groundedness, relevance, task completion, tool-use correctness, safety).\nBuild reusable evaluation frameworks, templates, and libraries that pods can adopt with minimal effort.\nPartner with product owners and SMEs to translate business requirements into measurable quality criteria and acceptance thresholds.\nDevelop automated evaluation pipelines integrated into CI/CD, with regression testing and quality gates for prompt, model, and retrieval changes.\nImplement offline evaluation (benchmark and golden datasets) and online evaluation (production monitoring, sampling, drift and quality alerts).\nSupport A/B and shadow testing for model, prompt, and architecture changes.\nDesign, validate, and maintain LLM-as-a-judge evaluators, including rubric design, bias mitigation (position, verbosity, self-preference), and calibration against human labels.\nEstablish human annotation workflows, labeling guidelines, and inter-annotator agreement practices with SMEs.\nTrack judge reliability over time and re-calibrate as models and use cases change.\nBuild and curate golden datasets, adversarial and edge-case suites, and synthetic test data generation approaches.\nApply approved data-handling practices (de-identification, synthetic data, access controls) for any dataset derived from sensitive or regulated data.\nImplement red-teaming, hallucination, toxicity, PII leakage, prompt-injection, and jailbreak testing.\nProduce evaluation evidence and reporting that supports security, privacy, compliance, and model risk reviews.\nContribute to responsible AI standards and release-readiness criteria.\nEmbed with pods to onboard them to evaluation practices, then coach teams toward ownership of their own eval suites.\nBuild dashboards and reporting that give engineering and business leaders transparent quality and cost/latency trends.\nEvaluate and recommend eval tooling and platforms; document standards and best practices.\nADDITIONAL FUNCTIONS and RESPONSIBILITIES\nPerforms other duties as assigned.\nTravel as required.\nQUALIFICATIONS\nEducation & Licensing\nMaster’s degree in a computer science or a quantitative discipline from an accredited college or university preferred.\nExperience\n4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications (adjust to your market).\nSkills & Knowledge\nProfessional experience with evaluation and observability frameworks like LangSmith, Arize Phoenix, or WhyLabs\nA strong understanding of statistical validation, model evaluation metrics, and the ability to partner with Data Scientists to optimize model performance\nStrong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review).\nPractical experience evaluating LLM/RAG systems, including retrieval metrics (recall, precision, MRR/nDCG) and generation metrics.\nHands-on experience with LLM-as-a-judge design, prompt engineering, and judge validation.\nWorking knowledge of statistics and experimental design: sampling, confidence intervals, significance testing, inter-rater agreement.\nFamiliarity with at least one eval or observability tool (e.g., Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or equivalent).\nExperience with cloud platforms (Azure, AWS, or GCP) and their managed LLM services.\nClear communication skills and the ability to work with engineers, product owners, SMEs, and risk stakeholders.\nExcellent oral and written communication skills, including presentation skills\nAbility to work in a team environment\nAbility to meet or exceed Performance Competencies\nPreferred\nExperience evaluating agentic systems (multi-step reasoning, tool calling, trajectory evaluation).\nBackground in ML evaluation, search/relevance, NLP, or QA automation.\nExperience with LLMOps, monitoring and observability, and production feedback loops.\nFamiliarity with AI risk and governance frameworks (e.g., NIST AI RMF, ISO/IEC 42001) and secure development practices in regulated environments.\nExperience with data annotation workflows and working with SMEs.\nKnowledge of enterprise controls: SSO, RBAC, private networking, logging, and encryption.\nSUCCESS MEASURES (first 12 months)\nStandard evaluation framework adopted by pods.\nAutomated quality gates in CI/CD for production-bound solutions.\nCalibrated LLM judges with documented agreement against human labels.\nReduction in regressions reaching production and in time to validate model or prompt changes.\nWORK ENVIRONMENT\nWhen applicable and appropriate, consideration will be given to reasonable accommodations.\nMental: Clear and conceptual thinking ability; excellent judgment, troubleshooting, problem solving, analysis, and discretion; ability to handle work-related stress; ability to handle multiple priorities simultaneously; and ability to meet deadlines\nPhysical: Computer keyboarding, travel as required\nAuditory/Visual: Hearing, vision and talking\nThe statements contained in this document are intended to describe the general nature and level of work being performed by a colleague assigned to this description. They are not intended to constitute a comprehensive list of functions, duties, or local variances. Management retains the discretion to add or to change the duties of the position at any time.\nSedgwick is an Equal Opportunity Employer and a Drug-Free Workplace.\nIf you're excited about this role but your experience doesn't align perfectly with every qualification in the job description, consider applying for it anyway! Sedgwick is building a diverse, equitable, and inclusive workplace and recognizes that each person possesses a unique combination of skills, knowledge, and experience. You may be just the right candidate for this or other roles.","description_format":"text","description_chars":7328,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"master","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Business Process Outsourcing (BPO)","Insurance Brokers & Agencies"],"lifecycle":[{"event":"open","at":"2026-11-03T23:29:37Z"}],"visa":[],"liveness":{"score":63,"band":"ok","label":"Likely open","p_open":1,"p_active":0.632,"p_room":1,"age_days":0,"expected_fill_days":24,"reasons":["conf:1","stale_co","velocity","win:early","comp:brand"],"computed_at":"2026-11-04T06:34:25Z"},"pay":null,"html_url":"https://alion.io/job/sedgwick-ai-evaluation-engineer","json_url":"https://alion.io/job/sedgwick-ai-evaluation-engineer.json","meta":{"generated_at":"2026-11-05T02:49:40Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":3768,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-06T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":5707},"rest":"https://alion.io/mcp/rest/get_company?id=5707"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsedgwick-ai-evaluation-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsedgwick-ai-evaluation-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsedgwick-ai-evaluation-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/sedgwick-ai-evaluation-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsedgwick-ai-evaluation-engineer"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3711663}},{"tool":"search_jobs","arguments":{"company":"sedgwick.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3711663},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}