First seen by Alion on Nov 20, 2026.
Our banking client is seeking a Senior QE Engineer – AI Evaluation (10+ years of experience) to build evaluation frameworks and regression testing using Python for LLM, RAG and agentic AI applications
Build an engineering-grade evaluation platform that helps teams determine whether changes to models, prompts, retrieval systems, tools and agent workflows improve or degrade application quality. This specialized role prioritizes hands-on AI evaluation engineering over AI application development, spanning reusable frameworks, large-scale test harnesses, custom evaluators and evaluation dashboards. The contract duration is 12 months.
Contract, Toronto, Working Hours: EST
Must Haves
- AI/LLM evaluation engineering: frameworks and methodologies, dataset-driven evaluation, regression testing, custom evaluators, LLM-as-a-Judge, RAG/retrieval evaluation and agent/tool-calling evaluation.
- Python development, with Pytest, API testing and integration testing.
- SQL and evaluation dataset management.
- CI/CD, Git, Docker, MLflow or similar evaluation/experiment-tracking platforms, and tracing and observability for AI applications.
- React, TypeScript and JavaScript exposure for dashboard/UI development.
Nice to Have
- Hands-on AI/LLM application development.
- Azure OpenAI/OpenAI APIs.
- Agentic application development.
- RAG application development.
- FastAPI or similar backend frameworks.
Responsibilities
- Architect reusable evaluation frameworks for AI and agentic applications.
- Build dataset-driven evaluation and test harnesses capable of running hundreds or thousands of scenarios.
- Develop custom evaluators for domain-specific correctness and quality.
- Implement model/prompt version comparison and regression detection.
- Evaluate RAG retrieval quality, relevance, grounding and correctness, alongside agent decision paths, tool selection and multi-step workflows.
- Integrate evaluation pipelines and automated release gates into CI/CD.
- Build dashboards and UI components for evaluation results, trends, regressions and traces.
AI may be used in evaluating candidates.
This posting is for an existing vacancy.

