Research Engineer, Benchmarks
Confirmed on the employer's own hiring board on Sep 27, 2026. First seen by Alion on Sep 26, 2026. Clera scores B on the Alion truth index.
About the Role
This is a hands-on research engineering role focused on designing and owning high-quality benchmarks that evaluate frontier AI agents on realistic, domain-specific workflows. You will sit within a small, highly technical team and play a critical part in ensuring evaluations are rigorous, credible, and trusted by leading AI labs and customers.
What You'll Do
Design, implement, and own the quality of internal benchmarks for evaluating frontier agents on domain-specific tasks.
Partner with subject-matter experts to define realistic workflows and translate them into benchmark tasks and evaluation criteria.
Build and operate reliable infrastructure to run models and agents against benchmark tasks at scale.
Develop metrics and statistical analyses that measure benchmark difficulty, reliability, and failure modes.
Validate that benchmark performance correlates with real-world evaluations, customer needs, and frontier lab expectations.
Write clear technical documentation and benchmark reports for research and engineering audiences.
What We're Looking For
2 to 4 years of experience in software engineering, ML engineering, or research roles, with a focused track record in AI benchmarks or evaluation infrastructure.
Strong proficiency in Python, Docker, and Linux environments.
Demonstrated experience designing, implementing, and running benchmarks or evaluation environments for AI agents or large language models.
Experience building infrastructure to reliably run AI models or agents against benchmark or evaluation tasks.
Ability to analyze and model workflows across diverse technical or business domains to support task design.
Sharp attention to detail with a habit of spotting subtle inconsistencies and edge cases.
Comfort reasoning from first principles about task design, scoring, and failure modes.
Strong written communication skills; experience producing technical documentation or benchmark reports.
Ability to thrive in unstructured problem spaces at an early-stage startup.
Bonus: experience with reinforcement learning pipelines, data generation, or RL agent evaluation; published work on AI benchmarking or model evaluation.
Compensation & Benefits
Salary range: USD 150,000 to 250,000 annually. Visa sponsorship is available.
Location
On-site in Singapore.