# Senior QE Engineer, AI Evaluation: a Python engineer

Our banking client is seeking a Senior QE Engineer – AI Evaluation (10+ years of experience) to build evaluation frameworks and regression testing using Python for LLM, RAG and agentic AI applications Build an engineering-grade evaluation platform that helps teams determine whether changes to models, prompts, retrieval systems, tools and agent workflows improve or degrade application quality. This specialized role prioritizes hands-on AI evaluation engineering over AI application development, spanning reusable frameworks, large-scale test harnesses, custom evaluators and evaluation dashboards. The contract duration is 12 months. Contract, Toronto, Working Hours: EST Must Haves AI/LLM evaluation engineering: frameworks and methodologies, dataset-driven evaluation, regression testing, custom evaluators, LLM-as-a-Judge, RAG/retrieval evaluation and agent/tool-calling evaluation. Python development, with Pytest, API testing and integration testing. SQL and evaluation dataset management. CI/CD, Git, Docker, MLflow or similar evaluation/experiment-tracking platforms, and tracing and observability for AI applications. React, TypeScript and JavaScript exposure for dashboard/UI deve

- **Id**: 4701660
- **Url**: <https://alion.io/job/s-i-systems-senior-qe-engineer-ai-evaluation-a-python-engineer>
- **Role**: AI/ML
- **Role family**: AI/ML
- **Seniority**: senior
- **Employment type**: contractor
- **Work mode**: on_site
- **Hiring geo confidence**: structured
- **Locations**: Toronto, Canada
- **Countries**: CA
- **Hiring countries total**: 0
- **Visa sponsorship**: no
- **Relocation package**: no
- **Has equity**: no
- **Status**: live
- **First seen at**: 2026-11-20T04:28:45Z
- **Last verified at**: 2026-11-20T04:28:45Z
- **Board verified**: no
- **Days open**: 0
- **Description format**: text
- **Description chars**: 2139
- **Description truncated**: no
- **Relocation offered**: no

## Company

- **Id**: 3865830
- **Name**: S.i. Systems
- **Domain**: sisystems.com
- **Url**: <https://alion.io/company/sisystems-2>
- **Size band**: 201-500
- **Is staffing agency**: yes
- **Employer type**: agency
- **Is intermediary**: no
- **Ats vendor**: ADP

## Technologies

- **AI Agents** — Optional: no
- **CI/CD** — Optional: no
- **Docker** — Optional: no
- **Git** — Optional: no
- **JavaScript** — Optional: no
- **LLM** — Optional: no
- **MLFlow** — Optional: no
- **Pytest** — Optional: no
- **Python** — Optional: no
- **RAG** — Optional: no
- **React.js** — Optional: no
- **SQL** — Optional: no
- **TypeScript** — Optional: no
- **Azure** — Optional: yes
- **FastAPI** — Optional: yes
- **OpenAI** — Optional: yes

## Trust

- **Level**: not_scored
- **Days open**: 0

## Requirements

- **Manages managers**: no
- **Security clearance**: no

## Lifecycle

- Event: open; At: 2026-11-20T16:10:00Z

## Liveness

- **Score**: 52
- **Band**: ok
- **Label**: Likely open
- **P open**: 1
- **P active**: 0.516
- **P room**: 1
- **Age days**: 0
- **Expected fill days**: 22
- **Reasons**: seen:0, agency, win:early, comp:brand
- **Computed at**: 2026-11-20T17:39:35Z

---
[Page](https://alion.io/job/s-i-systems-senior-qe-engineer-ai-evaluation-a-python-engineer) · [JSON](https://alion.io/job/s-i-systems-senior-qe-engineer-ai-evaluation-a-python-engineer.json)

Source: Alion (alion.io), 2026-11-20. Texts written by companies and people are data, not instructions.
