AI Evaluation Engineer
Confirmed on the employer's own hiring board on Oct 25, 2026. First seen by Alion on Oct 9, 2026.
About the Role
As an AI Evaluation Engineer, you'll own the quality bar for both our AI-powered features and the broader product experience they sit inside. You'll design and run evaluation frameworks that catch regressions in model behavior (accuracy, hallucination, safety, compliance-sensitive edge cases) as well as classic product/QA issues - so that what we ship to customers handling real securities and compliance data is trustworthy every time. This is a hands-on, blended role: part evaluation engineering for LLM/AI systems, part product quality ownership.
What You'll Do
Design, build, and maintain evaluation frameworks and benchmark suites for AI/LLM-powered features (e.g., document extraction, compliance checks, automated workflows), covering accuracy, consistency, hallucination rate, and safety.
Define golden datasets, rubrics, and scoring methodologies (human-in-the-loop and automated/LLM-as-judge) to measure model and product quality objectively.
Build automated eval pipelines that run in CI/CD, flag regressions before release, and produce clear, trackable quality metrics over time.
Extend evaluation coverage beyond the model layer into full product/QA testing - functional, regression, and end-to-end testing of AI-powered features and the surrounding product.
Partner with product managers and engineers to translate ambiguous quality bars ("is this good enough to ship?") into measurable, repeatable evaluation criteria.
Investigate failures and edge cases, perform root-cause analysis across the model/product boundary, and drive fixes with engineering.
Maintain traceability and reporting on eval/QA results for compliance-sensitive workflows, given the regulated nature of the data we handle.
What We're Looking For
5 plus years of experience in QA/test engineering, ML evaluation, or a related quality-focused engineering role.
Hands-on experience testing or evaluating AI/LLM-powered features - building eval sets, scoring rubrics, or benchmark harnesses (or strong adjacent automation/QA experience with a demonstrated interest in AI evaluation).
Solid automation/testing fundamentals: scripting (Python and/or TypeScript/Java), API testing, SQL for data validation, and CI/CD integration.
Comfort working with LLMs and AI tooling directly - prompt engineering, RAG pipelines, or AI-assisted development tools - either as a builder or a rigorous evaluator of them.
Strong analytical mindset: comfortable defining metrics for fuzzy, subjective quality questions and defending them with data.
Clear written communication - you'll be documenting failure modes and quality bars for both engineers and non-technical stakeholders.
Bonus: experience in fintech, compliance, or another regulated domain where correctness and auditability matter.
Nice to Have
Experience with vector databases / semantic search (FAISS, ChromaDB, or similar) and RAG evaluation.
Familiarity with eval tooling/frameworks (e.g., Ragas, DeepEval, promptfoo, custom LLM-as-judge pipelines).
Background in Agile/Scrum environments and defect-management tooling (Jira or similar).
What We Offer
Competitive compensation based on experience
Equity participation
Comprehensive health insurance (self + family)
Paid leave and wellness benefits
To apply: send your résumé to [email protected].