{"id":1606522,"url":"https://alion.io/job/innodata-qa-evaluation-lead","title":"QA / Evaluation Lead","company":{"id":48089,"name":"Innodata","domain":"innodata.com","url":"https://alion.io/company/innodata","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"A","score":100,"open_postings":8,"ghost_share":0,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":59,"computed_at":"2026-10-16T05:45:00Z"}},"role":"QA","role_family":"QA","seniority":"lead","employment_type":null,"work_mode":"hybrid","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Washington, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":45,"max":50,"currency":"USD","period":"hour","gross":null,"usd_annual":100000},"salary_estimate":null,"experience_years_min":6,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Databricks","optional":false},{"name":"Python","optional":false},{"name":"Synthetic Data","optional":false},{"name":"CVAT","optional":true}],"status":"live","first_seen_at":"2026-07-10T19:46:55Z","employer_posted_date":"2026-07-10","last_verified_at":"2026-10-16T21:37:22Z","board_verified":true,"closed_at":null,"days_open":98,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":98},"description":"Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers.\nAbout the Program: \nInnodata's Federal Practice builds the trusted data layer for critical infrastructure Trust & Safety work. Partnering with a leading systems integrator, we're delivering a modern, governed data services platform in a secure federal (IL4) environment. Over an intensive 20-week phase, you'll help stand up a data services storefront, a DataCard governance framework, synthetic data integration, and Databricks write-back capabilities.\nAbout the Role: \nAs the QA/Evaluation Lead, you'll own quality and evaluation across the platform. You'll design the evaluation framework that measures whether our data services and outputs meet the bar, build repeatable test and validation processes, and give the team an objective read on readiness at each milestone. Partnering with the Delivery Owner and engineering leads, you'll turn quality from an afterthought into a measurable, demonstrable strength. It's a role for someone who thinks rigorously about evaluation and takes pride in evidence-backed quality.\nKey Responsibilities:\nDesign and own the inter-annotator agreement (IAA) methodology for the Phase 1 demonstration corpus - metric selection (Cohen's kappa, Fleiss, Krippendorff's alpha), sampling design, adjudication workflow, and agreement thresholds\nDefine evaluation framework architecture: test and evaluation plans, IAA targets, drift detection gates, and model performance metrics per SOW Section 2.9\nConfigure and operate sampling-based quality control across the self-service and white-glove annotation paths during Phase D corpus production\nDesign and implement confidence-threshold escalation routing from automated annotation to senior-annotator adjudication\nValidate quality scoring and IAA computation within the Innodata data layer\nSupport AI Solutions Engineer on evaluation design for SAM 2 and Frontier model API validation - define what 'good enough' looks like quantitatively\nProduce evaluation framework documentation for the Phase 1 NPP closeout package, including per-DataCard documentation with the SA\nMust-Have Qualifications:\nBachelor's degree in Statistics, Data Science, Computer Science, or related quantitative field required; Master's degree preferred. Equivalent experience may substitute for degree on a 2-for-1 basis.\n6+ years total professional experience, 4+ years in data quality, evaluation methodology, or QA on AI/ML programs\nIAA methodology expertise - Cohen's kappa, Fleiss' kappa, Krippendorff's alpha: hands-on, not theoretical\nEvaluation framework design for AI/ML training data programs\nQC process design: sampling methodology, escalation workflows, adjudication protocols\nPython for QC tooling, metric computation, and statistical analysis\nActive Secret clearance with TS/SCI eligibility\nNice-to-Have Qualifications:\nPrior DoD or IC data quality program experience\nCVAT or equivalent annotation platform QC workflow configuration\nDrift detection and model monitoring methodology\nExperience with FMV / video annotation quality standards\nThe expected hourly salary range for this position is $45 to $50 p/hour, based on experience, skills, and qualifications.\nNote to Candidates: \nThis role is not a project manager with QC responsibilities - it is a methodology expert who owns the intellectual framework behind data quality on a federal AI program. The right candidate can walk into a meeting with Government evaluators and explain exactly why the evaluation design produces trustworthy labels. That conversation is part of Phase 2 positioning.\nPlease be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers. Innodata will never ask for payment, banking details, or sensitive personal information during the application process. To learn more on how to recognize job scams, please visit the Federal Trade Commission’s guide at https://consumer.ftc.gov/articles/job-scams. \nIf you believe you’ve been targeted by a recruitment scam, please report it to Innodata at  and consider reporting it to the FTC at ReportFraud.ftc.gov.","description_format":"text","description_chars":4668,"description_truncated":false,"requirements":{"experience_years_min":6,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":true,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Data Entry & Processing","AI Training Data & Annotation"],"lifecycle":[{"event":"open","at":"2026-10-01T19:52:38Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 2","filings_12m":2,"filings_prev_12m":2,"green_card_filings_12m":0,"median_offered_wage_usd":150000,"route":null,"cap_exempt":false,"checked_at":"2026-10-15T11:24:14+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":0}],"liveness":{"score":22,"band":"cold","label":"Long shot","p_open":1,"p_active":0.62,"p_room":0.36,"age_days":97,"expected_fill_days":59,"reasons":["conf:2","velocity","win:tail","crowd:"],"computed_at":"2026-10-16T05:45:00Z"},"pay":{"stated_usd_annual":100000,"is_top_pay":false},"html_url":"https://alion.io/job/innodata-qa-evaluation-lead","json_url":"https://alion.io/job/innodata-qa-evaluation-lead.json","meta":{"generated_at":"2026-10-17T00:40:49Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":938,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":48089},"rest":"https://alion.io/mcp/rest/get_company?id=48089"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Finnodata-qa-evaluation-lead"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Finnodata-qa-evaluation-lead"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Finnodata-qa-evaluation-lead"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/innodata-qa-evaluation-lead\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Finnodata-qa-evaluation-lead"}]}