2,300,357open jobs
145,988companies
269,708added this week
Browse all
Location
Remote (United States)
Seniority
Junior · 2+ years exp

Confirmed on the employer's own hiring board on Nov 16, 2026. First seen by Alion on Nov 5, 2026.

Overview
Company
Impact
Profile match
YO AI Labs is an artificial intelligence platform specializing in AI model training, evaluation, and data annotation services. Headquartered in San Francisco, California, United States, the company connects domain experts and remote specialists across the globe to review, test, and benchmark generative AI model outputs. Through its decentralized workforce solutions and structured evaluation workflows, it enables frontier technology developers to enhance the accuracy, safety, and reliability of next-generation artificial intelligence systems.

Job Title: AI Agent Tester - Real-World Tasks

Role Type: Contractor

Location: United States

Job Summary

We are seeking detail-oriented, U.S.-based AI Agent Testers to evaluate how AI assistants handle everyday tasks. You will test two AI agents using your own accounts and smartphone, assess their performance, and provide feedback to improve their real-world usability.

Key Responsibilities

  • Test AI agents on tasks such as shopping, bookings, bill payments, returns, email follow-ups, travel, and online forms.
  • Approve or stop agents at important decision points.
  • Screen-record every task attempt.
  • Score performance based on completion, quality, user control, satisfaction, and time.
  • Provide clear written feedback on the overall user experience.

Required Qualifications

  • Must be based in the United States.
  • Bachelor's degree required.
  • 2-3 years of professional experience, preferably in engineering, business management, or operations.
  • Regular use of AI tools in daily life.
  • Strong attention to detail, written communication, and documentation skills.
  • An active personal AI assistant account (Plus tier), active Muse and Dots access, a smartphone with screen-recording capability, and a payment card for reimbursable test purchases.

No coding or AI technical background is required. Generalists are welcome, and domain experts are preferred.

Project Details

  • Kickoff: Project is already active.
  • Onboarding: Within 24-48 hours of selection.
  • Work Schedule: Flexible, entirely from your phone.
  • Task Duration: Approximately 1.5 hours per run, including evidence and scoring.

Selection Process

  • Apply and complete the screening questions.
  • Attend a 15-30-minute interview.
  • Complete the profile review.
  • Receive onboarding instructions from the Talent Operations team.

Why Join?

  • Work with AI tools and evaluate real-world performance.
  • Enjoy flexible, phone-based work.
  • Help shape and improve next-generation AI agents.
Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
2,300,357 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Create a free account Continue with Google
Free forever. No card. Under a minute.

Your match

How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.

Recommended for you based on this role

QA
Similar stack
Same company
New York
Remote (likely EAEU) · Full-Time · 2+ years exp
Java
Java
Gradle
AI/ML
Copilot
Cursor
ChatGPT
Mobile
Firebase
Espresso
Fastlane
Google Play Billing
StoreKit
Maestro
Bitrise
TestFlight
DevOps
GitHub Actions
CI/CD
Git
QA
Cypress
Playwright
Appium
XCUITest
Apply
QA инженер 2 hours ago
Remote (likely EAEU) · 3+ years exp · Moscow
1C
AI/ML
Machine Learning
DevOps
Rest API
Git
Management
Jira
Scrum
Kanban
QA
Swagger
Chrome DevTools
Postman
Apply
$47k – $70k per year (gross) · Remote (EU, United Kingdom, Switzerland, Iceland) · Full-Time · 3+ years exp · Tallinn
JavaScript
DevOps
Rest API
CI/CD
SOAP
Cybersecurity
Shift-Left Security
Management
Agile
QA
JMeter
Apply
$53k – $80k per year (gross) · Remote (EU, United Kingdom, Switzerland, Iceland) · Full-Time · 3+ years exp · Vilnius
JavaScript
DevOps
Rest API
CI/CD
SOAP
Cybersecurity
Shift-Left Security
Management
Agile
QA
JMeter
Apply
$42k – $63k per year (gross) · Remote (EU, United Kingdom, Switzerland, Iceland) · Full-Time · 3+ years exp · Riga
JavaScript
DevOps
Rest API
CI/CD
SOAP
Cybersecurity
Shift-Left Security
Management
Agile
QA
JMeter
Apply
In office · Singapore
AI/ML
AI Agents
Agentic Workflows
Management
Agile
Apply
In office · Singapore
AI/ML
AI Agents
Agentic Workflows
Apply
In office · 5+ years exp · Bengaluru
AI/ML
AI Agents
LLM
Design
Figma
Rive
Apply
In office · Full-Time · Dubai
AI/ML
AI Agents
Apply
In office · Bachelor's Degree · Shanghai
AI/ML
AI Agents
Apply
$60k – $120k per year · Remote (United States) · 2+ years exp · Bachelor's Degree · New York
AI/ML
AI Agents
Apply
$100k – $200k per year · In office · New York
AI/ML
RLHF
Apply
Remote (United States) · New York
Apply
≈ $100k – $204k per year (Estimated) · In office · 2+ years exp · Bachelor's Degree · New York
AI/ML
AI Agents
Apply
$400k – $600k per year · In office · Master's Degree · New York
Apply
$136k – $181k per year · In office · Full-Time · 2+ years exp · Master's Degree · Sunnyvale · New York
Python
Java
C++
AI/ML
Computer Vision
Machine Learning
Apply
$210k – $260k per year · Equity · Hybrid · PhD · Denver · New York · San Francisco
Apply
Hybrid · 7+ years exp · Bachelor's Degree · New York
Python
AI/ML
Reinforcement Learning
AI Agents
Post-training
Red Teaming
LLM Evaluation
LLM Guardrails
Apply
In office · Full-Time · 7+ years exp · Bachelor's Degree · London · New York
SQL
AI/ML
LLM
Analytics
Microsoft Excel
Management
Jira
Agile
Apply
In office · Full-Time · 7+ years exp · Bachelor's Degree · London · New York
SQL
AI/ML
LLM
Analytics
Microsoft Excel
Management
Jira
Agile
Apply
See all jobs
This is one of many
2,300,357 more open roles from verified company boards, updated every day.