Location
Remote (United States)
Seniority
Junior · 2+ years exp
Confirmed on the employer's own hiring board on Nov 16, 2026. First seen by Alion on Nov 5, 2026.
Overview
Company
Impact
Profile match
YO AI Labs is an artificial intelligence platform specializing in AI model training, evaluation, and data annotation services. Headquartered in San Francisco, California, United States, the company connects domain experts and remote specialists across the globe to review, test, and benchmark generative AI model outputs. Through its decentralized workforce solutions and structured evaluation workflows, it enables frontier technology developers to enhance the accuracy, safety, and reliability of next-generation artificial intelligence systems.
Job Title: AI Agent Tester - Real-World Tasks
Role Type: Contractor
Location: United States
Job Summary
We are seeking detail-oriented, U.S.-based AI Agent Testers to evaluate how AI assistants handle everyday tasks. You will test two AI agents using your own accounts and smartphone, assess their performance, and provide feedback to improve their real-world usability.
Key Responsibilities
- Test AI agents on tasks such as shopping, bookings, bill payments, returns, email follow-ups, travel, and online forms.
- Approve or stop agents at important decision points.
- Screen-record every task attempt.
- Score performance based on completion, quality, user control, satisfaction, and time.
- Provide clear written feedback on the overall user experience.
Required Qualifications
- Must be based in the United States.
- Bachelor's degree required.
- 2-3 years of professional experience, preferably in engineering, business management, or operations.
- Regular use of AI tools in daily life.
- Strong attention to detail, written communication, and documentation skills.
- An active personal AI assistant account (Plus tier), active Muse and Dots access, a smartphone with screen-recording capability, and a payment card for reimbursable test purchases.
No coding or AI technical background is required. Generalists are welcome, and domain experts are preferred.
Project Details
- Kickoff: Project is already active.
- Onboarding: Within 24-48 hours of selection.
- Work Schedule: Flexible, entirely from your phone.
- Task Duration: Approximately 1.5 hours per run, including evidence and scoring.
Selection Process
- Apply and complete the screening questions.
- Attend a 15-30-minute interview.
- Complete the profile review.
- Receive onboarding instructions from the Talent Operations team.
Why Join?
- Work with AI tools and evaluate real-world performance.
- Enjoy flexible, phone-based work.
- Help shape and improve next-generation AI agents.
Free account
Stop reading job ads. Get the ones that fit.
One free account turns this page into a shortlist built around your stack, your level and your pay.
Match on every job. Stack, seniority, pay and location, scored against your profile.
2,300,357 open roles. Read straight off company career pages, refreshed every day.
Unlimited applications. Every one you send is tracked in one place, on-site or on a company board.
3 tailored CVs a month. Rewritten for the exact job you are applying to. Included free.
Free forever. No card. Under a minute.
Your match
How well do you fit this role?
Two answers are enough for a real match. No account needed.
Check my fit
Answers stay in this browser until you create an account.
Recommended for you based on this role
QA
Similar stack
Same company
New York
QA Manual Engineer (iOS, Android, Web)
1 hour ago
Remote (likely EAEU) · Full-Time · 2+ years exp
Java
Java
Gradle
AI/ML
Copilot
Cursor
ChatGPT
Mobile
Firebase
Espresso
Fastlane
Google Play Billing
StoreKit
Maestro
Bitrise
TestFlight
DevOps
GitHub Actions
CI/CD
Git
QA
Cypress
Playwright
Appium
XCUITest
Apply
QA инженер
2 hours ago
Remote (likely EAEU) · 3+ years exp · Moscow
1C
AI/ML
Machine Learning
DevOps
Rest API
Git
Management
Jira
Scrum
Kanban
QA
Swagger
Chrome DevTools
Postman
Apply
Quality Assurance Engineer
2 hours ago
$47k – $70k per year (gross) · Remote (EU, United Kingdom, Switzerland, Iceland) · Full-Time · 3+ years exp · Tallinn
JavaScript
DevOps
Rest API
CI/CD
SOAP
Cybersecurity
Shift-Left Security
Management
Agile
QA
JMeter
Apply
Quality Assurance Engineer
2 hours ago
$53k – $80k per year (gross) · Remote (EU, United Kingdom, Switzerland, Iceland) · Full-Time · 3+ years exp · Vilnius
JavaScript
DevOps
Rest API
CI/CD
SOAP
Cybersecurity
Shift-Left Security
Management
Agile
QA
JMeter
Apply
Quality Assurance Engineer
2 hours ago
$42k – $63k per year (gross) · Remote (EU, United Kingdom, Switzerland, Iceland) · Full-Time · 3+ years exp · Riga
JavaScript
DevOps
Rest API
CI/CD
SOAP
Cybersecurity
Shift-Left Security
Management
Agile
QA
JMeter
Apply
In office · Singapore
AI/ML
AI Agents
Agentic Workflows
Management
Agile
Apply
In office · Singapore
AI/ML
AI Agents
Agentic Workflows
Apply
Senior AI Product Designer (UX / UI)
9 days ago
In office · 5+ years exp · Bengaluru
AI/ML
AI Agents
LLM
Design
Figma
Rive
Apply
In office · Full-Time · Dubai
AI/ML
AI Agents
Apply
In office · Bachelor's Degree · Shanghai
AI/ML
AI Agents
Apply
AI Agent Tester Real-World Tasks - Remote
11 days ago
$60k – $120k per year · Remote (United States) · 2+ years exp · Bachelor's Degree · New York
AI/ML
AI Agents
Apply
Apply
Apply
Everyday AI Agent Tester - Remote
1 month ago
≈ $100k – $204k per year (Estimated) · In office · 2+ years exp · Bachelor's Degree · New York
AI/ML
AI Agents
Apply
Senior Mechanical Engineer- Remote
10 days ago
$400k – $600k per year · In office · Master's Degree · New York
Apply
Machine Learning Engineer 2 - Freewheel
29 min ago
$136k – $181k per year · In office · Full-Time · 2+ years exp · Master's Degree · Sunnyvale · New York
Python
Java
C++
AI/ML
Computer Vision
Machine Learning
Apply
$210k – $260k per year · Equity · Hybrid · PhD · Denver · New York · San Francisco
Apply
Technical Project Manager
1 hour ago
Hybrid · 7+ years exp · Bachelor's Degree · New York
Python
AI/ML
Reinforcement Learning
AI Agents
Post-training
Red Teaming
LLM Evaluation
LLM Guardrails
Apply
Product Manager - Private Funds Research
1 hour ago
In office · Full-Time · 7+ years exp · Bachelor's Degree · London · New York
SQL
AI/ML
LLM
Analytics
Microsoft Excel
Management
Jira
Agile
Apply
Product Manager - Private Funds Analytics
1 hour ago
In office · Full-Time · 7+ years exp · Bachelor's Degree · London · New York
SQL
AI/ML
LLM
Analytics
Microsoft Excel
Management
Jira
Agile
Apply
This is one of many
2,300,357 more open roles from verified company boards, updated every day.

