{"id":990400,"url":"https://alion.io/job/fetch-staff-ai-evaluation-lead","title":"Staff AI Evaluation Lead","company":{"id":672968,"name":"Fetch","domain":"fetch.com","url":"https://alion.io/company/fetch-3","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Gem","truth_index":{"grade":"A","score":88,"open_postings":15,"ghost_share":0,"stale_share":0.267,"repost_share":0,"time_to_fill_p50_days":64,"computed_at":"2026-10-07T05:47:15Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"lead","employment_type":"full_time","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"posting_text","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":["US"],"hiring_countries_total":1,"salary":{"min":129000,"max":152000,"currency":"USD","period":"year","gross":null,"usd_annual":152000},"salary_estimate":null,"experience_years_min":8,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Human-in-the-Loop","optional":false},{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"SQL","optional":false}],"status":"live","first_seen_at":"2026-08-07T19:00:50Z","employer_posted_date":"2026-08-17","last_verified_at":"2026-10-08T00:42:08Z","board_verified":true,"closed_at":null,"days_open":61,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":61},"description":"What we’re building and why we’re building it.\nFetch helps people live rewarded every day, with a vision to become the rewards destination for everyone. We turn everyday activities into meaningful rewards, whether it’s grocery shopping, grabbing a quick meal, or playing a favorite mobile game. To date, we’ve awarded more than $1 billion in Fetch Points to our users.\nEach day, more than 13 million receipts are submitted on Fetch, providing visibility into over $212 billion in gross merchandise value. This creates the largest retail-agnostic, SKU-level view of household spending, powering Fetch as an outcomes-based advertising platform that helps brands acquire and retain lifelong consumers.\nThe Fetch app is available on the App Store and Google Play, with more than 6 million five-star reviews from a highly engaged and loyal user base.\nIt’s not just our users who believe in Fetch: with investments from Softbank, ICONIQ, DST, Greycroft, and partnerships ranging from challenger brands to Fortune 500 companies, Fetch is reshaping how brands and consumers connect in the marketplace. When you work at Fetch, you play a vital role in a platform that drives brand loyalty and creates lifelong consumers with the power of Fetch points. User and partner success are at the heart of everything we do, and we extend that same commitment to our employees.\nAt Fetch, we value curiosity, adaptability, and the confidence to explore new tools, especially AI, to drive smarter, faster work. You don’t need to be an expert, but you should be ready to learn quickly and think critically. We welcome learners who move fast, challenge the status quo, and shape what’s next, with us. Ranked as one of America’s Best Startup Employers by Forbes for two years in a row, Fetch fosters a people-first culture rooted in trust, accountability, and innovation. We encourage our employees to challenge ideas, think bigger, and always bring the fun to Fetch.\nAbout the Role:\nAt Fetch, we’re building AI and automation systems that make our work smarter, faster, and more scalable. The AI Operations team ensures our models, automations, and LLM systems perform with quality, reliability, and measurable impact.\nAs a Staff AI Evaluation Lead, you’ll own automation and evaluation programs across AI Operations. You’ll translate business and functional goals into scalable systems, define how we measure quality, and ensure automation and evaluation become durable, high-impact capabilities across the organization.\nThis role is ideal for someone who combines deep technical problem-solving with systems-level thinking and strong cross-functional leadership.\nThis is a full-time role that can be held from one of our US offices or remotely in the United States.\nRole Responsibilities:\nOwn programs: Lead complex, high-impact automation and evaluation initiatives across workflows or teams\nDesign scalable solutions: Architect end-to-end workflows integrating datasets, evaluations, automations, and HITL processes; build the harness and reusable components other analysts build against; build evaluation pipelines that run against production without hands-on operation\nEstablish standards: Define dataset standards, evaluation methodology, failure taxonomy, and quality measurement across AI Operations; verify that projects you don't run are meeting them; own the process by which those standards get made and used\nOwn metric definitions: Create and maintain the metric definitions the org evaluates against, validate them against real production behavior, and revise them as models, tooling, and system architectures change\nSet the bar for production: Define the quality bar a system clears before it reaches production and where human review stays in the loop, and pull a system back when it stops meeting the bar\nAllocate evaluation depth by risk: Decide which systems get what depth of evaluation given finite capacity, name the risk accepted on the rest, and make that tradeoff visible to project stakeholders\nKeep the evaluation stack current: Define when a change to a model or platform requires re-baselining across the org and own the process for doing it; evaluate new models and AI capabilities as they ship and decide what the org adopts\nImprove systems at scale: Lead redesign of workflows, tooling, and processes to improve performance and durability across AI Operations, not only within your own programs\nDrive cross-functional alignment: Influence priorities and partner with Engineering, Product, and AI teams to deliver solutions\nMeasure and communicate impact: Define success metrics and communicate performance and recommendations to leadership\nElevate the team: Raise the bar through your work and actively mentor others by sharing approaches, guiding problem-solving, and enabling the team to build stronger automation and evaluation capabilities\nDrive innovation: Stay current on emerging tools and approaches; pilot and translate them into actionable improvements for the team\nMinimum Requirements:\n8+ years of professional experience AI, machine learning, operational automations, or a related field.\nProven ability to lead complex automation or evaluation initiatives across systems or teams\nExperience designing, building, and scaling evaluation frameworks, datasets, and quality systems at scale for LLMs or AI products\nFluency in SQL, JSON, APIs, and scripting with AI assistance, and ownership of the technical direction of the evaluation stack\nDeep working knowledge of LLM and agentic system behavior, automation platforms, and system design, demonstrated in systems you have built\nExperience with data pipelines, APIs, and production systems\nExperience in influencing cross-functional stakeholders and aligning priorities\nDemonstrated ability to define metrics and drive measurable business impact\nPreferred Requirements:\nExperience mentoring or leading technical contributors\nExperience evaluating agentic systems in production at scale\nExperience setting technical standards adopted across an organization\nCompensation:\nAt Fetch, we offer competitive compensation packages including base, equity, and benefits to the exceptional folks we hire. The base salary range for this position is $129,000 - $152,000. Discover our benefits and how our employees live rewarded at https://fetch.com/careers.\nAt Fetch, we'll give you the tools to feel healthy, happy and secure through:\nEquity: We offer full-time employees equity in Fetch, so that everyone can benefit from Fetch’s growth.\n401k Match: Dollar-for-dollar match up to 4%.\nBenefits for humans and pets: We offer comprehensive medical, dental and vision plans for everyone including your pets.\nContinuing Education: Fetch provides ten thousand per year in education reimbursement.\nEmployee Resource Groups: Take part in employee-led groups that are centered around fostering a diverse and inclusive workplace through events, dialogue and advocacy. The ERGs participate in our Inclusion Council with members of executive leadership.\nPaid Time Off: On top of our flexible PTO, Fetch observes 9 paid holidays, as well as our year-end week-long break. \nRobust Leave Policies: 20 weeks of paid parental leave for primary caregivers, 14 weeks for secondary caregivers, and a flexible return to work schedule. \nCalvin Care Cash: Employees who are welcoming new family members will also receive a one time $2,000 incentive to assist employees with covering the cost of childcare, clothing, diapers and much more!\nFlexible Work Environment: Collaborate with your team in one of our stunning offices, or you can work fully remotely from anywhere in the US. We’ll ensure you are equally equipped with the hardware and software you need to get your job done in the comfort of your home. (applicable for most roles)\nFetch is an equal opportunity employer that embraces diversity, inclusion, and respect for all individuals. We do not discriminate on the basis of race, color, religion, gender, gender identity or expression, sexual orientation, age, national origin, marital status, veteran status, disability, or any other characteristic protected by applicable law. Our commitment to inclusivity ensures that everyone is treated with dignity and has the opportunity to succeed based on their talent, skills, and potential.\nFetch also provides reasonable accommodations to qualified individuals with disabilities or those with sincerely held religious beliefs, as required by law. If you need assistance with the application process or require an accommodation, please contact us at .","description_format":"text","description_chars":8566,"description_truncated":false,"requirements":{"experience_years_min":8,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["401k plan","Equity","Flexible schedule","Parental leave"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Commerce","Gift Cards & Loyalty Programs","Coupons, Deals & Cashbacks"],"lifecycle":[{"event":"open","at":"2026-09-17T02:14:28Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 44 · green card filings: 14","filings_12m":44,"filings_prev_12m":42,"green_card_filings_12m":14,"median_offered_wage_usd":161552,"route":null,"cap_exempt":false,"checked_at":"2026-10-03T21:08:04+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":36}],"liveness":{"score":46,"band":"ok","label":"Likely open","p_open":1,"p_active":0.77,"p_room":0.6,"age_days":60,"expected_fill_days":64,"reasons":["conf:0","velocity","win:late","crowd:"],"computed_at":"2026-10-07T05:47:15Z"},"pay":{"stated_usd_annual":152000,"is_top_pay":false},"html_url":"https://alion.io/job/fetch-staff-ai-evaluation-lead","json_url":"https://alion.io/job/fetch-staff-ai-evaluation-lead.json","meta":{"generated_at":"2026-10-08T01:19:33Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler","counted_by":"address","units_charged":1,"used_today":2380,"day_limit":5000,"remaining_today":2620,"minute_limit":60,"resets_at":"2026-10-09T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":672968},"rest":"https://alion.io/mcp/rest/get_company?id=672968"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Ffetch-staff-ai-evaluation-lead"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Ffetch-staff-ai-evaluation-lead"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Ffetch-staff-ai-evaluation-lead"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/fetch-staff-ai-evaluation-lead\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Ffetch-staff-ai-evaluation-lead"}]}