{"id":1606491,"url":"https://alion.io/job/innodata-quality-lead-agentic-ai-workflow-evaluation","title":"Quality Lead, Agentic AI Workflow Evaluation","company":{"id":48089,"name":"Innodata","domain":"innodata.com","url":"https://alion.io/company/innodata","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"A","score":100,"open_postings":8,"ghost_share":0,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":59,"computed_at":"2026-10-21T05:46:15Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"lead","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["San Jose, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":75,"max":85,"currency":"USD","period":"hour","gross":null,"usd_annual":170000},"salary_estimate":null,"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Function Calling","optional":false},{"name":"Python","optional":false},{"name":"RLHF","optional":false},{"name":"SQL","optional":false}],"status":"live","first_seen_at":"2026-09-19T15:33:08Z","employer_posted_date":"2026-09-19","last_verified_at":"2026-10-21T16:54:10Z","board_verified":true,"closed_at":null,"days_open":32,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":32},"description":"Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers.\nScope of the Role: \nWe are standing up a dedicated onsite team to evaluate complex, real-world agentic AI workflows for a frontier AI customer. Reviewers work through ambiguous, multi-step scenarios inside isolated test environments, assessing whether AI agents complete tasks safely, respect user intent and consent, and hold up under close scrutiny. The Quality Lead is the person accountable for whether that output is any good. \nThis is a senior individual contributor role. You will not manage the reviewers - that sits with the Engagement Manager - but you set the standard they are held to. You own the audit sample, run calibration, keep the rubric usable as real cases stress it, and train reviewers into the work. You are also the deputy: when the Engagement Manager is out, the engagement runs on you. \nThe quality approach here is not fully defined. We expect you to build it in partnership with the customer's quality leads, or at minimum to take what they have, run it honestly, and come back with specific recommendations for where it falls short. \nWhat You’ll Own:\nOwn the quality system for the engagement: audit design, sampling strategy, scoring standards, and how quality gets measured and reported \nBuild that system with the customer's quality leads where none exists, and where one does, operate it and recommend concrete improvements based on what the data shows \nRe-score a sample of reviewer output as a second pass; identify error patterns rather than isolated mistakes \nRun calibration sessions: surface disagreement, work it to resolution, and document the reasoning so the outcome holds for future cases \nMaintain rubric health - flag criteria that are ambiguous, overlapping, or silent on cases the team keeps hitting, and drive revisions through the customer \nTrain and onboard new reviewers, including nesting plans, ramp criteria, and the judgment call on when someone is production-ready \nGive the Engagement Manager the evidence behind performance conversations: who is drifting, on what, and whether coaching is working \nReport quality trends to the Engagement Manager and, alongside them, to the customer \nDeputize for the Engagement Manager on delivery operations during absences \nMaintain information security, privacy, and facility access practices required by the customer's onsite environment \nYou’ll Thrive in This Role If You Have:\nBachelor's degree or equivalent practical experience \n4+ years in quality assurance, quality management, or senior review work within annotation, evaluation, trust and safety, or a similarly judgment-intensive domain \nDirect experience owning a quality function: you designed the audit, not just executed someone else’s \nSignificant experience with AI/ML evaluation work: annotation, red-teaming, RLHF, model or agent evaluation, or trust and safety review \nHands-on familiarity with agentic systems: tool use, multi-step task execution, sandboxed environments, and common failure modes \nDemonstrated ability to run calibration with peers - including holding a position under disagreement and changing it when the argument is better \nStrong written communication; able to document a scoring standard clearly enough that a reviewer can apply it and an auditor can check it \nComfortable in spreadsheets and in a dashboarding tool, with enough Python or SQL to pull and slice your own data (you will not be asked to build interfaces) \nExperience training or onboarding reviewers into rubric-based work \nThe expected hourly salary range for this position is $75-85 p/hour, based on experience, skills, and qualifications.\nPlease be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers. Innodata will never ask for payment, banking details, or sensitive personal information during the application process. To learn more on how to recognize job scams, please visit the Federal Trade Commission’s guide at https://consumer.ftc.gov/articles/job-scams. \nIf you believe you’ve been targeted by a recruitment scam, please report it to Innodata at  and consider reporting it to the FTC at ReportFraud.ftc.gov.","description_format":"text","description_chars":4749,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Data Entry & Processing","AI Training Data & Annotation"],"lifecycle":[{"event":"open","at":"2026-10-01T19:52:38Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 2","filings_12m":2,"filings_prev_12m":2,"green_card_filings_12m":0,"median_offered_wage_usd":150000,"route":null,"cap_exempt":false,"checked_at":"2026-10-15T11:24:14+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":0}],"liveness":{"score":77,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.856,"p_room":0.9,"age_days":31,"expected_fill_days":59,"reasons":["conf:2","velocity","win:mid"],"computed_at":"2026-10-21T05:46:15Z"},"pay":{"stated_usd_annual":170000,"is_top_pay":false},"html_url":"https://alion.io/job/innodata-quality-lead-agentic-ai-workflow-evaluation","json_url":"https://alion.io/job/innodata-quality-lead-agentic-ai-workflow-evaluation.json","meta":{"generated_at":"2026-10-21T19:55:47Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":28238,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-22T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":48089},"rest":"https://alion.io/mcp/rest/get_company?id=48089"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Finnodata-quality-lead-agentic-ai-workflow-evaluation"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Finnodata-quality-lead-agentic-ai-workflow-evaluation"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Finnodata-quality-lead-agentic-ai-workflow-evaluation"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/innodata-quality-lead-agentic-ai-workflow-evaluation\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Finnodata-quality-lead-agentic-ai-workflow-evaluation"}]}