AI Agent Evaluation Engineer - Remote

YO AI Labs • AI Training Data & Annotation • AI Evaluation & Observability • Python • Java • Rust • TypeScript • C++ • 12 days ago
Smart Apply
Similar jobs
1,983,706open jobs
117,337companies
249,537added this week
Browse all
Salary
$160k – $240k per year
Location
Remote (United States)

Confirmed on the employer's own hiring board on Oct 26, 2026. First seen by Alion on Oct 14, 2026. YO AI Labs scores A on the Alion truth index.

Overview
Company
Impact
Profile match

YO AI Labs

YO AI Labs is an artificial intelligence platform specializing in AI model training, evaluation, and data annotation services. Headquartered in San Francisco, California, United States, the company connects domain experts and remote specialists across the globe to review, test, and benchmark generative AI model outputs. Through its decentralized workforce solutions and structured evaluation workflows, it enables frontier technology developers to enhance the accuracy, safety, and reliability of next-generation artificial intelligence systems.
yoailabs.com • HQ: San Francisco, United States • AI Evaluation & Observability • AI Training Data & Annotation • 5000+ employees
Tech stack of YO AI Labs

Senior Software Engineer

Job Type: Contractor (~15 hours/week)

Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

  • Create reinforcement learning environments for software engineering tasks.
  • Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
  • Build deterministic verification systems and golden reference solutions.
  • Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
  • Develop realistic, reproducible software engineering scenarios.
  • Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
  • Document solutions and provide clear technical feedback.

Required Skills

  • Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
  • Strong understanding of algorithms and data structures.
  • Experience with bug fixing and debugging complex software issues.
  • Proven experience in feature implementation and codebase refactoring.
  • Strong knowledge of performance optimization and tuning.
  • Excellent written and verbal communication.
  • Strong attention to detail.

Preferred Qualifications

  • Experience working with large-scale or distributed codebases.
  • Familiarity with AI/ML systems is a plus but not required.
  • Experience with rigorous code reviews and software engineering best practices.
  • Experience working effectively in remote or cross-functional teams.

Hiring Process

  • Submit an application and screening questions.
  • Complete an AI interview (~30 minutes).
  • Complete a technical assessment, if required.
  • Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24-48 hours of completing onboarding.

Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
1,983,706 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

AI/ML
Similar stack
Same company
Seattle
≈ $78k – $150k per year (Estimated) • Hybrid • Full-Time • 6+ years exp • Bachelor's Degree • San Diego • Rolling Meadows
Python
TypeScript
AI/ML
LangGraph
LangChain
Claude Code
Vertex AI
Fine-tuning
Prompt Engineering
AI Agents
LangSmith
AWS Bedrock
CrewAI
LLM
RAG
Whisper
Few-Shot Learning
OpenAI Codex
AWS Strands Agents
Google AI Studio
Lovable
Deepgram
Semantic Search
Agentic Workflows
Vertex AI Agent Builder
DevOps
GCP
WebRTC
Azure
CI/CD
Git
AWS
Docker
Robotics
Apollo
Apply
Remote (likely EAEU) • Moscow
Python
SQL
AI/ML
LLM
Feature Store
Structured Outputs
Machine Learning
DevOps
Docker
Apply
$54k – $64k per year • Remote (EU, United States, CET hours) • Full-Time • 5+ years exp • New York
Python
Go
Databases
PostgreSQL
AI/ML
LangGraph
AutoGen
LangChain
LoRA
Fine-tuning
Prompt Engineering
Function Calling
AI Agents
PEFT
Transformers
CrewAI
LLM
RAG
Reranking
Human-in-the-Loop
LLM Evaluation
Multi-Agent Systems
Machine Learning
Frontend
GraphQL
DevOps
gRPC
AWS
AWS Lambda
Amazon EC2
Amazon ECS
Apply
≈ $63k – $107k per year (Estimated) • Remote (Poland) • Contractor • Warsaw
AI/ML
Copilot
Cursor
LangGraph
LangChain
Claude Code
Model Context Protocol
LLM
RAG
OpenAI Codex
DevOps
CI/CD
Apply
$184k – $249k per year • Equity • In office • Full-Time • 6+ years exp • Master's Degree • Seattle • Arlington • New York
Python
Java
C++
AI/ML
Machine Learning
Apply
$11k per year (net) • In office • Full-Time • Ufa
C++
C++
CMake
DevOps
Git
Linux
TCP/IP
Apply
≈ $9k – $22k per year (Estimated) • In office • Full-Time • 2+ years exp • Jakarta
Python
SQL
AI/ML
Machine Learning
Apply
In office • 1+ year exp • Tashkent
Python
SQL
DevOps
Rest API
Management
Jira
QA
Postman
Apply
≈ $14k – $32k per year (Estimated) • In office • Full-Time • 4+ years exp • Bachelor's Degree • Jakarta
Python
SQL
Databases
Presto
AI/ML
Spark
Apply
≈ $31k – $83k per year (Estimated) • In office • 7+ years exp • Bengaluru
JavaScript
TypeScript
Node JS
Node JS
Express
Databases
PostgreSQL
AI/ML
Fine-tuning
LLM
RAG
Frontend
Angular
React.js
DevOps
GitHub Actions
GitLab CI
CI/CD
Jenkins
Apply
ML Engineer - Remote 11 days ago
≈ $183k – $406k per year (Estimated) • Remote (United States) • Master's Degree • New York
Python
AI/ML
llama.cpp
vLLM
JAX
SciPy
SGLang
Transformers
NumPy
PyTorch
Hugging Face
Machine Learning
Apply
$120k – $200k per year • Remote (United States) • United States
JavaScript
C++
AI/ML
CUDA
Frontend
WebGPU
Game Dev
GLSL
Apply
$120k – $200k per year • Remote (United States) • Boston
JavaScript
C++
AI/ML
CUDA
Frontend
WebGPU
Game Dev
GLSL
Apply
$120k – $170k per year • Remote (United States) • San Francisco
JavaScript
C++
AI/ML
LLM
CUDA
Frontend
WebGPU
DevOps
HPC
Game Dev
GLSL
Apply
$120k – $170k per year • Remote (United States) • Seattle
JavaScript
C++
AI/ML
LLM
CUDA
Frontend
WebGPU
DevOps
HPC
Game Dev
GLSL
Apply
≈ $143k – $277k per year (Estimated) • Equity • In office • Full-Time • 5+ years exp • Bachelor's Degree • Seattle • Dallas
ABAP
ABAP
SAP Fiori
SAP BTP
Databases
SAP HANA
AI/ML
AI Agents
Mobile
Clean Architecture
Analytics
SAP BusinessObjects
Apply
$116k – $160k per year • Equity • In office • Full-Time • 5+ years exp • Master's Degree • Seattle • Sunnyvale
DevOps
AWS
Apply
COOK (FULL TIME) 1 day ago
$50k – $52k per year • In office • Full-Time • Seattle
Apply
$221k – $277k per year • In office • 8+ years exp • Salt Lake City • New York • San Francisco • Seattle
Apply
≈ $45k – $88k per year (Estimated) • In office • 4+ years exp • Seattle
Apply
See all jobs
This is one of many
1,983,706 more open roles from verified company boards, updated every day.
Browse all jobs Get matched roles