Eval Engineer

Responsibilities:

• Design and own evaluation systems for agent domains.
• Take architectural responsibility for eval coverage across one or more capability domains: multi-turn conversation evals, tool-selection accuracy, goal completion, policy compliance, and safety boundary probing.
• Coverage spans behavioural equivalence, SLOs, idempotency, chaos-as-test, context and retrieval quality, and tool/dependency failure modes; each is a distinct failure surface that functional correctness evals alone do not cover.
• Define what "evaluated" means for the domain, not just execute a checklist.

Requirements:

• 5+ years of software engineering experience, with at least 2 years building evaluation, measurement, or quality systems for AI/ML products.
• Engineering-parallel fundamentals: you can design, build, and operate evaluation infrastructure as production-grade distributed systems (services, pipelines, storage, observability), not scripts and spreadsheets.
• Deep hands-on LLM evaluation experience: LLM-as-judge design and calibration, human annotation pipelines, golden dataset governance, and systematic error analysis from raw traces.
• Demonstrated ability to define acceptance criteria for nondeterministic behaviour, turning fuzzy notions of quality into thresholds an organisation can operate against.
• Experience diagnosing agentic systems across the full execution path: able to attribute a failure to prompt, context, tooling, or model capability, and defend the attribution with evidence.
• Experience wiring evaluation into deployment flows as a hard gate, blocking bad changes before they reach production rather than reporting on them after.
• Statistical fluency applied to evaluation: sample size design, confidence intervals, and significance testing, enough to defend a regression call when a team disputes it.
• Proficiency in Python and/or Java at the level needed to build and maintain a shared eval framework others depend on.
• Experience operating in environments where AI failures have real business or compliance consequences.

Skills

Agent, Agentic AI, Evaluation, LLM, Large Language Model, Python