AI Agent Evaluation Engineer - Remote

YO AI Labs • Artificial Intelligence • LLM & Generative AI • Python • Java • Rust • TypeScript • C++ • 24 days ago
Smart Apply
Similar jobs
1,865,820open jobs
110,134companies
261,649added this week
Browse all
Salary
$160k – $240k per year
Location
In office (Atlanta)

Confirmed on the employer's own hiring board on Oct 20, 2026. First seen by Alion on Sep 26, 2026. YO AI Labs scores A on the Alion truth index.

Overview
Company
Impact
Profile match

YO AI Labs

YO AI Labs is an artificial intelligence platform specializing in AI model training, evaluation, and data annotation services. Headquartered in San Francisco, California, United States, the company connects domain experts and remote specialists across the globe to review, test, and benchmark generative AI model outputs. Through its decentralized workforce solutions and structured evaluation workflows, it enables frontier technology developers to enhance the accuracy, safety, and reliability of next-generation artificial intelligence systems.
yoailabs.com • HQ: San Francisco, United States • AI Evaluation & Observability • AI Training Data & Annotation • 5000+ employees
Tech stack of YO AI Labs

Senior Software Engineer

Job Type: Contractor (~15 hours/week)

Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

  • Create reinforcement learning environments for software engineering tasks.
  • Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
  • Build deterministic verification systems and golden reference solutions.
  • Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
  • Develop realistic, reproducible software engineering scenarios.
  • Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
  • Document solutions and provide clear technical feedback.

Required Skills

  • Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
  • Strong understanding of algorithms and data structures.
  • Experience with bug fixing and debugging complex software issues.
  • Proven experience in feature implementation and codebase refactoring.
  • Strong knowledge of performance optimization and tuning.
  • Excellent written and verbal communication.
  • Strong attention to detail.

Preferred Qualifications

  • Experience working with large-scale or distributed codebases.
  • Familiarity with AI/ML systems is a plus but not required.
  • Experience with rigorous code reviews and software engineering best practices.
  • Experience working effectively in remote or cross-functional teams.

Hiring Process

  • Submit an application and screening questions.
  • Complete an AI interview (~30 minutes).
  • Complete a technical assessment, if required.
  • Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24-48 hours of completing onboarding.

Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
1,865,820 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

AI/ML
Similar stack
Same company
Atlanta
$180k – $200k per year • In office • Full-Time • 3+ years exp • Bachelor's Degree • San Francisco
Python
TypeScript
AI/ML
AI Agents
Apply
$170k – $200k per year • In office • Full-Time • New York
Python
TypeScript
Databases
Snowflake
Databricks
AI/ML
Model Context Protocol
Function Calling
AI Agents
LLM
Tool Use
DevOps
Helm
GitHub Actions
Azure
CI/CD
AWS
Docker
Kubernetes
Platform Engineering
GitHub
Apply
$150k – $300k per year • In office • Full-Time • New York
Python
TypeScript
AI/ML
AI Agents
LLM
Apply
$197k – $225k per year • In office • Full-Time • 4+ years exp • Bachelor's Degree • New York
Python
Go
Java
C++
Scala
C++
TensorFlow C++
PyTorch C++
AI/ML
Spark
Reinforcement Learning
Scikit-learn
Transformers
TensorFlow
Pandas
NumPy
PyTorch
Ray
Explainable AI
Machine Learning
DevOps
GCP
Azure
CI/CD
AWS
Kubernetes
Management
Agile
Apply
AI Platform Engineer 6 hours ago
$70k – $90k per year • Remote (United States) • Full-Time • Bachelor's Degree • Alpharetta
Python
SQL
AI/ML
Model Context Protocol
Prompt Engineering
AI Agents
Multi-Agent Systems
DevOps
Ansible
DigitalOcean
GitHub Actions
CI/CD
Jenkins
Git
AWS
Docker
Platform Engineering
Linux
Apply
$136k – $166k per year • Equity • In office • Full-Time • Belmont
Python
Bash
Mobile
MVP
DevOps
Ansible
Zabbix
VictoriaMetrics
Grafana
Linux
TCP/IP
BGP
OSPF
MPLS
Cybersecurity
Tcpdump
Apply
≈ $16k – $26k per year (Estimated) • In office • 1+ year exp • Bachelor's Degree • Novosibirsk
Python
SQL
DevOps
Linux
Design
Figma
IoT
OPC UA
Apply
PL SQL - Manager 7 days ago
≈ $14k – $36k per year (Estimated) • Hybrid • Full-Time • Bachelor's Degree • Bengaluru
Java
SQL
Perl
Java
Hibernate
Databases
Oracle
DevOps
Linux
Unix
Analytics
ETL/ELT
Dimensional Modeling
Apply
≈ $19k – $40k per year (Estimated) • Hybrid • Full-Time • Bachelor's Degree • Pune • Mumbai
Python
AI/ML
Model Context Protocol
Fine-tuning
Prompt Engineering
AI Agents
LLM
RAG
Machine Learning
DevOps
AWS
Apply
≈ $20k – $53k per year (Estimated) • Hybrid • Full-Time • Bachelor's Degree • Pune • Mumbai
Python
AI/ML
LangGraph
AutoGen
LangChain
Model Context Protocol
Fine-tuning
Quantization
Prompt Engineering
Knowledge Distillation
AI Agents
LLM
RAG
Multi-Agent Systems
Model Distillation
Machine Learning
DevOps
CI/CD
AWS
Apply
≈ $190k – $419k per year (Estimated) • In office • New York
Python
Java
Rust
TypeScript
C++
Apply
≈ $193k – $427k per year (Estimated) • In office • New York
Python
Java
Rust
TypeScript
C++
AI/ML
Model Context Protocol
Reinforcement Learning
AI Agents
Apply
≈ $191k – $422k per year (Estimated) • In office • New York
Python
AI/ML
Scikit-learn
TensorFlow
PyTorch
Machine Learning
Apply
$120k – $200k per year • In office • Phoenix
JavaScript
C++
AI/ML
CUDA Toolkit
CUDA
Frontend
WebGPU
Game Dev
GLSL
Apply
$120k – $200k per year • In office • Dallas
JavaScript
C++
AI/ML
CUDA Toolkit
CUDA
Frontend
WebGPU
Game Dev
GLSL
Apply
$200k – $400k per year • Remote (United States) • 3+ years exp • Master's Degree • Atlanta
AI/ML
RLHF
Prompt Engineering
Machine Learning
Apply
Merchandiser 6 hours ago
In office • Full-Time • Atlanta
Apply
Apply
≈ $40k – $104k per year (Estimated) • In office • Bachelor's Degree • Atlanta
Apply
≈ $40k – $88k per year (Estimated) • Remote (United States) • Atlanta
Apply
See all jobs
This is one of many
1,865,820 more open roles from verified company boards, updated every day.
Browse all jobs Get matched roles