AI Agent Evaluation Engineer - Remote

YO AI Labs • Artificial Intelligence • LLM & Generative AI • Python • Java • Rust • TypeScript • C++ • 24 days ago
Smart Apply
Similar jobs
1,851,491open jobs
108,759companies
258,783added this week
Browse all
Salary
$160k – $240k per year
Location
In office (United States)

Confirmed on the employer's own hiring board on Oct 20, 2026. First seen by Alion on Sep 26, 2026. YO AI Labs scores A on the Alion truth index.

Overview
Company
Impact
Profile match

YO AI Labs

YO AI Labs is an artificial intelligence platform specializing in AI model training, evaluation, and data annotation services. Headquartered in San Francisco, California, United States, the company connects domain experts and remote specialists across the globe to review, test, and benchmark generative AI model outputs. Through its decentralized workforce solutions and structured evaluation workflows, it enables frontier technology developers to enhance the accuracy, safety, and reliability of next-generation artificial intelligence systems.
yoailabs.com • HQ: San Francisco, United States • AI Evaluation & Observability • AI Training Data & Annotation • 5000+ employees
Tech stack of YO AI Labs

Senior Software Engineer

Job Type: Contractor (~15 hours/week)

Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

  • Create reinforcement learning environments for software engineering tasks.
  • Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
  • Build deterministic verification systems and golden reference solutions.
  • Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
  • Develop realistic, reproducible software engineering scenarios.
  • Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
  • Document solutions and provide clear technical feedback.

Required Skills

  • Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
  • Strong understanding of algorithms and data structures.
  • Experience with bug fixing and debugging complex software issues.
  • Proven experience in feature implementation and codebase refactoring.
  • Strong knowledge of performance optimization and tuning.
  • Excellent written and verbal communication.
  • Strong attention to detail.

Preferred Qualifications

  • Experience working with large-scale or distributed codebases.
  • Familiarity with AI/ML systems is a plus but not required.
  • Experience with rigorous code reviews and software engineering best practices.
  • Experience working effectively in remote or cross-functional teams.

Hiring Process

  • Submit an application and screening questions.
  • Complete an AI interview (~30 minutes).
  • Complete a technical assessment, if required.
  • Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24-48 hours of completing onboarding.

Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
1,851,491 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

AI/ML
Similar stack
Same company
United States
In office • Moscow
Python
SQL
Python
Asyncio
HTTPX
AI/ML
Model Context Protocol
Function Calling
AI Agents
Langfuse
LLM
Tool Use
DevOps
GitLab CI
CI/CD
Git
Docker
QA
Pytest
Apply
$48k – $72k per year • Remote (LATAM, Europe) • TS/SCI • Full-Time • Bachelor's Degree
AI/ML
Claude
Claude Code
AI Agents
Apply
≈ $25k – $66k per year (Estimated) • In office • Full-Time • 8+ years exp • Bengaluru
Python
JavaScript
TypeScript
AI/ML
LLM
Machine Learning
QA
Selenium
Playwright
Appium
Apply
AI Developer 1 day ago
Hybrid • Full-Time • Mumbai
Python
JavaScript
TypeScript
Python
Flask
FastAPI
Databases
Weaviate
Chroma
FAISS
AI/ML
LangGraph
LangChain
LlamaIndex
Prompt Engineering
LangSmith
LLM
RAG
Semantic Search
Semantic Search
Apply
Vision AI Engineer 1 day ago
≈ $18k – $49k per year (Estimated) • Hybrid • Full-Time • 4+ years exp • Bachelor's Degree • India
Python
AI/ML
OpenCV
Computer Vision
ONNX
TensorRT
TensorFlow
PyTorch
Synthetic Data
OCR
Machine Learning
Robotics
Digital Twin
Apply
In office • Full-Time • 3+ years exp • Bachelor's Degree • Christchurch
Python
C
C++
MATLAB
C
Embedded C
DevOps
Git
Linux
Robotics
Sensor Fusion
Apply
≈ $59k – $164k per year (Estimated) • In office • Full-Time • Sydney
Java
TypeScript
C#
C#
.NET
Databases
Snowflake
Robotics
Digital Twin
Apply
$108k – $117k per year • Equity • Hybrid • Full-Time • 2+ years exp • Bachelor's Degree • Austin
Java
Apex
Apex
Lightning Web Components
AI/ML
Copilot
Agentforce
DevOps
CI/CD
Bamboo
Bitbucket
GitHub
Management
Jira
Agile
Apply
≈ $25k – $55k per year (Estimated) • In office • Tianjin
Python
SQL
Management
Agile
Apply
$113k – $193k per year • Hybrid • 8+ years exp • Eden Prairie
Python
SQL
AI/ML
Hadoop
Spark
NLP
Amazon SageMaker
Machine Learning
DevOps
AWS
Amazon S3
Management
Agile
Apply
≈ $190k – $419k per year (Estimated) • In office • New York
Python
Java
Rust
TypeScript
C++
Apply
≈ $193k – $427k per year (Estimated) • In office • New York
Python
Java
Rust
TypeScript
C++
AI/ML
Model Context Protocol
Reinforcement Learning
AI Agents
Apply
≈ $191k – $422k per year (Estimated) • In office • New York
Python
AI/ML
Scikit-learn
TensorFlow
PyTorch
Machine Learning
Apply
$120k – $200k per year • In office • Phoenix
JavaScript
C++
AI/ML
CUDA Toolkit
CUDA
Frontend
WebGPU
Game Dev
GLSL
Apply
$120k – $200k per year • In office • Dallas
JavaScript
C++
AI/ML
CUDA Toolkit
CUDA
Frontend
WebGPU
Game Dev
GLSL
Apply
≈ $51k – $99k per year (Estimated) • Hybrid • Full-Time • 6+ years exp • Bachelor's Degree • Irving • New York • Washington
DevOps
AWS
Management
Agile
Scrum
Apply
≈ $65k – $132k per year (Estimated) • Hybrid • Public Trust • Full-Time • 6+ years exp • Bachelor's Degree • Ashburn • Washington
Apply
$104k – $156k per year • Hybrid • Full-Time • Dallas
Databases
Google BigQuery
BigQuery
Apply
≈ $140k – $260k per year (Estimated) • Remote (United States) • Full-Time • 2+ years exp • United States
AI/ML
Machine Learning
Marketing
Salesforce
HubSpot
Marketo
Apply
$91k – $155k per year • Equity • Remote (United States) • Full-Time • 6+ years exp • Bachelor's Degree • United States
Python
JavaScript
PHP
TypeScript
SQL
C#
Node JS
C#
.NET
Frontend
Angular
React.js
DevOps
GCP
Azure
AWS
Docker
Kubernetes
Apply
See all jobs
This is one of many
1,851,491 more open roles from verified company boards, updated every day.
Browse all jobs Get matched roles