AI Agent Evaluation Engineer - Remote

YO AI Labs · Artificial Intelligence · LLM & Generative AI · Python · Java · Rust · TypeScript · C++ · 2 months ago

Smart Apply

Similar jobs

2,093,777open jobs

128,564companies

243,634added this week

Browse all

Salary

$114k – $172k per year

Location

In office (Ottawa)

Confirmed on the employer's own hiring board on Nov 4, 2026. First seen by Alion on Sep 27, 2026.

Overview

Company

Impact

Profile match

YO AI Labs

YO AI Labs is an artificial intelligence platform specializing in AI model training, evaluation, and data annotation services. Headquartered in San Francisco, California, United States, the company connects domain experts and remote specialists across the globe to review, test, and benchmark generative AI model outputs. Through its decentralized workforce solutions and structured evaluation workflows, it enables frontier technology developers to enhance the accuracy, safety, and reliability of next-generation artificial intelligence systems.

yoailabs.com · HQ: San Francisco, United States · AI Evaluation & Observability · AI Training Data & Annotation · 5000+ employees

Tech stack of YO AI Labs

Senior Software Engineer

Job Type: Contractor (~15 hours/week)

Location: Remote

Job Summary

We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools.

You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required.

Key Responsibilities

Create reinforcement learning environments for software engineering tasks.
Design tasks involving bug fixing, feature development, refactoring, and performance optimization.
Build deterministic verification systems and golden reference solutions.
Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.
Develop realistic, reproducible software engineering scenarios.
Ensure tasks accurately measure coding ability, problem-solving, and tool usage.
Document solutions and provide clear technical feedback.

Required Skills

Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.
Strong understanding of algorithms and data structures.
Experience with bug fixing and debugging complex software issues.
Proven experience in feature implementation and codebase refactoring.
Strong knowledge of performance optimization and tuning.
Excellent written and verbal communication.
Strong attention to detail.

Preferred Qualifications

Experience working with large-scale or distributed codebases.
Familiarity with AI/ML systems is a plus but not required.
Experience with rigorous code reviews and software engineering best practices.
Experience working effectively in remote or cross-functional teams.

Hiring Process

Submit an application and screening questions.
Complete an AI interview (~30 minutes).
Complete a technical assessment, if required.
Hiring Manager review.

Compensation

Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply.

Availability

Selected experts should be prepared to begin their first tasks within 24-48 hours of completing onboarding.

Free account

Stop reading job ads. Get the ones that fit.

One free account turns this page into a shortlist built around your stack, your level and your pay.

Match on every job. Stack, seniority, pay and location, scored against your profile.

2,093,777 open roles. Read straight off company career pages, refreshed every day.

Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.

3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.

Create a free account Continue with Google

Free forever. No card. Under a minute.

Your match

How well do you fit this role?

Two answers are enough for a real match. No account needed.

Check my fit

Answers stay in this browser until you create an account.

Recommended for you based on this role

AI/ML

Similar stack

Same company

Ottawa

Ingénieur(e) des connaissances | IA générative, recherche sémantique et gouvernance des contenus 27 days ago

Hybrid · Canada

Databases

Databricks

DevOps

Azure

Apply