Data Scientist - Agent Evaluations & Quality
This a Full Remote job, the offer is available from: California (USA)
About the Role
We're an early-stage AI company building autonomous agents that handle real work - email, calendar, browser, business software, and more. We're hiring a Data Scientist focused on Agent Evaluations & Quality to measure, understand, and continuously improve the quality of our agent capabilities.
Your mission is to translate ambiguous product behavior into measurable definitions of success, build representative evaluation datasets, design reliable graders and metrics, analyze failures, and create the feedback loops that guide engineering and product decisions. This is applied data science at the intersection of evaluation design, statistics, experimentation, production Python, and deep understanding of how LLM agents behave in real products.
This is a full-time, on-site role based in Palo Alto, CA. Visa sponsorship is not available.
What You'll Do
Architect and maintain automated evaluation pipelines that measure agent quality across capabilities and product surfaces.
Translate capabilities into explicit success criteria - including pass, partial-pass, and failure definitions for complex multi-step tasks.
Build representative gold datasets and regression suites covering common workflows, ambiguous requests, long-tail behavior, edge cases, and adversarial scenarios.
Define and track metrics such as task success, partial completion, tool-selection accuracy, tool-use correctness, instruction adherence, factual consistency, user corrections, latency, cost, and reliability.
Design deterministic graders, model-based graders, and human-review processes; calibrate LLM-as-a-judge systems and measure false positives, false negatives, variance, and grader agreement.
Analyze traces, tool calls, model outputs, user context, and production outcomes to identify root causes and build a useful failure taxonomy.
Compare models, prompts, tools, and capability implementations using rigorous offline experiments and production evidence.
Build dashboards, reports, and release-quality signals that make evaluation results understandable and actionable for engineering, product, and leadership.
Partner with capability engineers to recommend improvements and verify that fixes raise quality without unacceptable regressions in cost, latency, or reliability.
What We're Looking For
Required - Dealbreakers:
5+ years of experience in data science, machine learning, or analytics roles building or delivering evaluation systems, metrics frameworks, or quality measurement solutions for production systems.
Demonstrated experience designing and implementing evaluation frameworks, metrics, and grading systems for ML/AI systems in production.
Production-quality Python and SQL proficiency with the ability to build automated data pipelines and analysis code at scale.
Required Skills & Experience:
Experience designing evaluation methodologies: success criteria definition, dataset construction, metric selection, and distinguishing useful benchmarks from misleading ones.
Statistical and experimental design knowledge: sampling, variance, uncertainty quantification, bias detection, confounding variables, and significance testing for non-deterministic systems.
Experience with ground-truth data development: labeling guideline design, annotation quality control, ambiguity resolution, and dataset maintenance as product behavior evolves.
Working knowledge of LLM behavior - including model-based graders, tool use, retrieval systems, multi-step execution, partial completion, and practical failure modes.
Analytical debugging ability: connecting quantitative patterns to individual system traces and identifying failure origins across model, prompt, context, tools, data, and application logic.
Experience building dashboards, reports, and communicating evaluation results, methodology, uncertainty, and trade-offs to both technical and non-technical stakeholders.
Nice to Have:
Experience with LLM-as-a-judge systems, calibration, and measurement of grader agreement, false positives, and false negatives.
Prior work on evaluation or benchmarking platforms for AI systems.
Experience with agentic systems, multi-step task execution, or tool-use evaluation.
Experience working on customer-facing consumer software or production ML systems with real-world user outcomes.
Location & Work Arrangement
On-site in Palo Alto, CA
Visa sponsorship is not available