2,439,294open jobs
150,169companies
296,285added this week
Browse all
Location
In office (Toronto)
Seniority
Senior
Employment
Contractor

First seen by Alion on Nov 20, 2026.

Overview
Company
Impact
Profile match
S.i. Systems is one of Canada's largest IT staffing and talent recruitment agencies, specializing in contract staffing, direct-hire placements, contractor payrolling, and IT project solutions. Headquartered in Calgary, Alberta - with major offices across Toronto, Vancouver, Ottawa, and other key metropolitan centers - the firm connects organizations with highly skilled technology professionals across cybersecurity, software development, cloud infrastructure, enterprise systems, and data analytics. Backed by an extensive nationwide network of pre-screened IT contractors, it supports corporate and public sector clients navigating large-scale digital transformations and complex technical initiatives.

Our banking client is seeking a Senior QE Engineer – AI Evaluation (10+ years of experience) to build evaluation frameworks and regression testing using Python for LLM, RAG and agentic AI applications

 

Build an engineering-grade evaluation platform that helps teams determine whether changes to models, prompts, retrieval systems, tools and agent workflows improve or degrade application quality. This specialized role prioritizes hands-on AI evaluation engineering over AI application development, spanning reusable frameworks, large-scale test harnesses, custom evaluators and evaluation dashboards. The contract duration is 12 months.

 

Contract, Toronto, Working Hours: EST

Must Haves

  • AI/LLM evaluation engineering: frameworks and methodologies, dataset-driven evaluation, regression testing, custom evaluators, LLM-as-a-Judge, RAG/retrieval evaluation and agent/tool-calling evaluation.
  • Python development, with Pytest, API testing and integration testing.
  • SQL and evaluation dataset management.
  • CI/CD, Git, Docker, MLflow or similar evaluation/experiment-tracking platforms, and tracing and observability for AI applications.
  • React, TypeScript and JavaScript exposure for dashboard/UI development.

 

Nice to Have

  • Hands-on AI/LLM application development.
  • Azure OpenAI/OpenAI APIs.
  • Agentic application development.
  • RAG application development.
  • FastAPI or similar backend frameworks.

 

Responsibilities

  • Architect reusable evaluation frameworks for AI and agentic applications.
  • Build dataset-driven evaluation and test harnesses capable of running hundreds or thousands of scenarios.
  • Develop custom evaluators for domain-specific correctness and quality.
  • Implement model/prompt version comparison and regression detection.
  • Evaluate RAG retrieval quality, relevance, grounding and correctness, alongside agent decision paths, tool selection and multi-step workflows.
  • Integrate evaluation pipelines and automated release gates into CI/CD.
  • Build dashboards and UI components for evaluation results, trends, regressions and traces.
Disclaimer:
AI may be used in evaluating candidates.
This posting is for an existing vacancy.
Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
2,439,294 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

AI/ML
Remote, many countries
Similar stack
Same company
Toronto
Equity · Remote (Canada) · 5+ years exp · Canada
Databases
Databricks
Delta Lake
AI/ML
MLFlow
Multimodal AI
DevOps
Azure
CI/CD
Docker
Kubernetes
Incident Management
Analytics
ETL/ELT
Apply
Hybrid · Full-Time · 2+ years exp · Bachelor's Degree · Montreal
AI/ML
AI Agents
NLP
Machine Learning
Management
ServiceNow
Apply
AI/ML Developer 11 hours ago
$59k – $67k per year · In office · Full-Time · 4+ years exp · Bachelor's Degree · Toronto
Python
JavaScript
SQL
Databases
Pinecone
FAISS
AI/ML
Copilot
LangGraph
LangChain
MLFlow
Scikit-learn
Computer Vision
AI Agents
NLP
TensorFlow
PyTorch
Gemini
LLM
RAG
Time Series
Explainable AI
OpenAI
Anthropic
Recommender Systems
Machine Learning
DevOps
GCP
Azure DevOps
GitHub Actions
Azure
CI/CD
AWS
Docker
Kubernetes
Management
Agile
Apply
$155k – $260k per year · Equity · In office · Full-Time · 3+ years exp · Vancouver
AI/ML
Computer Vision
NLP
Machine Learning
Apply
$98k – $132k per year · Hybrid · Contractor · Mississauga
Python
TypeScript
AI/ML
Model Context Protocol
Embeddings
AI Agents
AgentOps
LLM
RAG
OpenAI
LLMOps
Semantic Search
Agentic Workflows
Copilot Studio
DevOps
Azure
CI/CD
Management
ServiceNow
SharePoint
Agile
QA
Rest-Assured
Apply
Remote (likely EAEU) · Moscow
Python
Python
FastAPI
Asyncio
Celery
Databases
PostgreSQL
Redis
RabbitMQ
AI/ML
LangGraph
LangChain
LlamaIndex
Model Context Protocol
AI Agents
Langfuse
LangSmith
CrewAI
LLM
DevOps
CI/CD
Docker
Kubernetes
Octopus Deploy
Cybersecurity
Keycloak
Apply
Remote (LATAM) · Full-Time · 8+ years exp · San José
Python
JavaScript
Node JS
Databases
PostgreSQL
Redis
AI/ML
Function Calling
LLM
RAG
OpenAI
Anthropic
Structured Outputs
LLM Evaluation
Frontend
React.js
Mobile
React Native
DevOps
CI/CD
AWS
Amazon S3
Apply
AI/ML Developer 11 hours ago
$59k – $67k per year · In office · Full-Time · 4+ years exp · Bachelor's Degree · Toronto
Python
JavaScript
SQL
Databases
Pinecone
FAISS
AI/ML
Copilot
LangGraph
LangChain
MLFlow
Scikit-learn
Computer Vision
AI Agents
NLP
TensorFlow
PyTorch
Gemini
LLM
RAG
Time Series
Explainable AI
OpenAI
Anthropic
Recommender Systems
Machine Learning
DevOps
GCP
Azure DevOps
GitHub Actions
Azure
CI/CD
AWS
Docker
Kubernetes
Management
Agile
Apply
$34k – $40k per year · In office · Full-Time · 1+ year exp · Bachelor's Degree · Toronto
Apply
Inside Sales Agent 2 hours ago
$30k – $45k per year · Remote (Canada) · Full-Time · Toronto
Apply
$59k – $109k per year · Hybrid · 5+ years exp · Bachelor's Degree · Toronto
AI/ML
AI Agents
Analytics
Microsoft Excel
Apply
Hybrid · Full-Time · Bachelor's Degree · Toronto
Apply
See all jobs
This is one of many
2,439,294 more open roles from verified company boards, updated every day.