Full-Stack Engineer, Agent Evaluation & Reliability
Confirmed on the employer's own hiring board on Nov 4, 2026. First seen by Alion on Nov 1, 2026.
About the Role
Join a small, technical AI company building infrastructure that helps enterprise AI agents improve through production feedback. This backend-leaning full-stack role focuses on reliable evaluation systems and includes ownership of product features from design through delivery.
What You'll Do
Build backend systems for agent evaluation and continuous improvement.
Design data models for evaluation datasets, experiment runs, agent traces, knowledge versions, findings, and customer-defined quality criteria.
Develop and operate enterprise integrations, including authentication, OAuth, webhooks, incremental syncs, backfills, rate limits, retries, and monitoring.
Take features from technical design through implementation, deployment, monitoring, customer feedback, and iteration.
Work with customers to diagnose system failures and turn specific needs into reusable product capabilities.
Build product interfaces across the full stack while contributing to architecture and product decisions.
What We're Looking For
Two to four years of experience building and operating production software, including experience at a fast-paced startup.
Hands-on machine learning engineering experience that includes model development.
Strong production experience with TypeScript and Node.js or Bun, and the ability to work across the full stack.
Strong PostgreSQL knowledge and practical experience with Redis, job queues, background workers, or workflow systems.
Understanding of why evaluating nondeterministic AI agent systems differs from conventional software testing.
High ownership, fast learning, and the ability to take projects from start to finish.
Compensation & Benefits
Base salary: $120,000 to $180,000 annually. Visa sponsorship is available.
Location
On-site in San Francisco, California, United States.