AI Agent Evaluation Engineer - Remote

YO AI Labs • Artificial Intelligence • LLM & Generative AI • Python • Java • Rust • TypeScript • C++ • 22 days ago
Smart Apply
Similar jobs
1,828,223open jobs
105,310companies
275,979added this week
Browse all
Salary
$114k – $170k per year
Location
In office (Montreal)

Confirmed on the employer's own hiring board on Oct 18, 2026. First seen by Alion on Sep 27, 2026. YO AI Labs scores A on the Alion truth index.

Overview
Company
Impact
Profile match

YO AI Labs

YO AI Labs is an artificial intelligence platform specializing in AI model training, evaluation, and data annotation services. Headquartered in San Francisco, California, United States, the company connects domain experts and remote specialists across the globe to review, test, and benchmark generative AI model outputs. Through its decentralized workforce solutions and structured evaluation workflows, it enables frontier technology developers to enhance the accuracy, safety, and reliability of next-generation artificial intelligence systems.
yoailabs.com • HQ: San Francisco, United States • AI Evaluation & Observability • AI Training Data & Annotation • 5000+ employees
Tech stack of YO AI Labs

Senior Software Engineer

Job Type: Contractor (~15 hours/week)

Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

  • Create reinforcement learning environments for software engineering tasks.
  • Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
  • Build deterministic verification systems and golden reference solutions.
  • Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
  • Develop realistic, reproducible software engineering scenarios.
  • Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
  • Document solutions and provide clear technical feedback.

Required Skills

  • Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
  • Strong understanding of algorithms and data structures.
  • Experience with bug fixing and debugging complex software issues.
  • Proven experience in feature implementation and codebase refactoring.
  • Strong knowledge of performance optimization and tuning.
  • Excellent written and verbal communication.
  • Strong attention to detail.

Preferred Qualifications

  • Experience working with large-scale or distributed codebases.
  • Familiarity with AI/ML systems is a plus but not required.
  • Experience with rigorous code reviews and software engineering best practices.
  • Experience working effectively in remote or cross-functional teams.

Hiring Process

  • Submit an application and screening questions.
  • Complete an AI interview (~30 minutes).
  • Complete a technical assessment, if required.
  • Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24-48 hours of completing onboarding.

Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
1,828,223 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

AI/ML
Similar stack
Same company
Montreal
≈ $89k – $178k per year (Estimated) • Hybrid • Full-Time • High School Diploma • Mirabel
AI/ML
AI Agents
Human-in-the-Loop
Management
Agile
Apply
MLOps Technical Lead 6 hours ago
≈ $85k – $169k per year (Estimated) • In office • Mississauga
Python
AI/ML
MLFlow
Kubeflow
Machine Learning
DevOps
Terraform
GitHub Actions
CircleCI
CloudFormation
Fluentd
Prometheus
GitLab CI
CI/CD
Jenkins
Git
AWS
Grafana
Bitbucket
GitHub
GitLab
Apply
≈ $60k – $121k per year (Estimated) • Hybrid • Full-Time • 3+ years exp • Waterloo
Python
SQL
PowerShell
Python
pySpark
Databases
Databricks
AI/ML
Copilot
Claude
Spark
Vertex AI
Prompt Engineering
AI Agents
Gemini
OpenAI Codex
DevOps
GCP
Git
Management
Google Workspace
Agile
Apply
≈ $117k – $240k per year (Estimated) • In office • 7+ years exp • Toronto
Python
Go
TypeScript
Python
FastAPI
AI/ML
LangGraph
LangChain
Claude
Spark
LlamaIndex
Model Context Protocol
Fine-tuning
Embeddings
AI Agents
LiteLLM
AWS Bedrock
TensorFlow
PyTorch
LLM
RAG
Synthetic Data
OpenAI
Anthropic
LLMOps
Structured Outputs
LLM Guardrails
Machine Learning
DevOps
AWS
Platform Engineering
Cybersecurity
Okta
Apply
$202k – $224k per year • Equity • In office • 5+ years exp • Bachelor's Degree • Sunnyvale
Python
Go
Java
Scala
AI/ML
Spark
Machine Learning
Apply
Equity • In office • Full-Time • 7+ years exp • Bachelor's Degree
Python
Java
C++
DevOps
Linux
TCP/IP
Cybersecurity
Wireshark
Apply
$94k – $179k per year • Remote (United States) • Full-Time • Dallas • Kansas City • Des Moines
Python
DevOps
Rest API
Ansible
GCP
Azure
AWS
VPN
BGP
OSPF
Apply
$119k – $139k per year • Hybrid • 5+ years exp • Bachelor's Degree • Syracuse
Python
SQL
Databases
PostgreSQL
Snowflake
Oracle
AI/ML
dbt
DevOps
Terraform
GitHub Actions
Azure
CI/CD
Git
Linux
Analytics
ETL/ELT
Apply
$145k – $155k per year • Hybrid • 8+ years exp • Bachelor's Degree • Plano
Python
AI/ML
LangGraph
LangChain
Prompt Engineering
AI Agents
Semantic Kernel
AWS Bedrock
RAG
OpenAI
Amazon SageMaker
DevOps
Terraform
Ansible
Chef
CloudFormation
Azure
CI/CD
Jenkins
Git
AWS
Bitbucket
Linux
Management
Agile
Apply
≈ $63k – $149k per year (Estimated) • Remote (Brazil) • 10+ years exp • Bachelor's Degree • Rio de Janeiro
Python
JavaScript
SQL
Node JS
Databases
PostgreSQL
Databricks
Delta Lake
AI/ML
AutoGen
LangChain
Prompt Engineering
AI Agents
CrewAI
RAG
Multi-Agent Systems
Frontend
React.js
DevOps
Terraform
Ansible
Azure
CI/CD
AWS
Docker
Kubernetes
GitLab
Apply
≈ $190k – $415k per year (Estimated) • In office • New York
Python
Java
Rust
TypeScript
C++
Apply
≈ $194k – $424k per year (Estimated) • In office • New York
Python
Java
Rust
TypeScript
C++
AI/ML
Model Context Protocol
Reinforcement Learning
AI Agents
Apply
≈ $191k – $419k per year (Estimated) • In office • New York
Python
AI/ML
Scikit-learn
TensorFlow
PyTorch
Machine Learning
Apply
$120k – $200k per year • In office • Phoenix
JavaScript
C++
AI/ML
CUDA Toolkit
CUDA
Frontend
WebGPU
Game Dev
GLSL
Apply
$120k – $200k per year • In office • Dallas
JavaScript
C++
AI/ML
CUDA Toolkit
CUDA
Frontend
WebGPU
Game Dev
GLSL
Apply
$67k – $84k per year • Hybrid • Montreal
Python
AI/ML
Copilot
Prompt Engineering
LLM
Machine Learning
DevOps
Azure
Management
Power Apps
Agile
Apply
Sales Manager 1 day ago
≈ $50k – $95k per year (Estimated) • In office • 5+ years exp • Bachelor's Degree • Montreal
Management
Microsoft Office
Apply
≈ $130k – $246k per year (Estimated) • Remote (United States, United Kingdom, Canada) • Contractor • Los Angeles • Austin • Montreal • Manchester
Apply
$56k – $72k per year • In office • Full-Time • 2+ years exp • High School Diploma • Montreal
DevOps
AWS
Management
Microsoft Office
Apply
$42k – $54k per year • In office • Full-Time • 1+ year exp • Montreal
DevOps
AWS
Apply
See all jobs
This is one of many
1,828,223 more open roles from verified company boards, updated every day.
Browse all jobs Get matched roles