Staff AI Evaluation Lead

What we’re building and why we’re building it.

Fetch helps people live rewarded every day, with a vision to become the rewards destination for everyone. We turn everyday activities into meaningful rewards, whether it’s grocery shopping, grabbing a quick meal, or playing a favorite mobile game. To date, we’ve awarded more than $1 billion in Fetch Points to our users.

Each day, more than 13 million receipts are submitted on Fetch, providing visibility into over $212 billion in gross merchandise value. This creates the largest retail-agnostic, SKU-level view of household spending, powering Fetch as an outcomes-based advertising platform that helps brands acquire and retain lifelong consumers.

The Fetch app is available on the App Store and Google Play, with more than 6 million five-star reviews from a highly engaged and loyal user base.

It’s not just our users who believe in Fetch: with investments from Softbank, ICONIQ, DST, Greycroft, and partnerships ranging from challenger brands to Fortune 500 companies, Fetch is reshaping how brands and consumers connect in the marketplace. When you work at Fetch, you play a vital role in a platform that drives brand loyalty and creates lifelong consumers with the power of Fetch points. User and partner success are at the heart of everything we do, and we extend that same commitment to our employees.

At Fetch, we value curiosity, adaptability, and the confidence to explore new tools, especially AI, to drive smarter, faster work. You don’t need to be an expert, but you should be ready to learn quickly and think critically. We welcome learners who move fast, challenge the status quo, and shape what’s next, with us. Ranked as one of America’s Best Startup Employers by Forbes for two years in a row, Fetch fosters a people-first culture rooted in trust, accountability, and innovation. We encourage our employees to challenge ideas, think bigger, and always bring the fun to Fetch.

About the Role:

At Fetch, we’re building AI and automation systems that make our work smarter, faster, and more scalable. The AI Operations team ensures our models, automations, and LLM systems perform with quality, reliability, and measurable impact.

As a Staff AI Evaluation Lead, you’ll own automation and evaluation programs across AI Operations. You’ll translate business and functional goals into scalable systems, define how we measure quality, and ensure automation and evaluation become durable, high-impact capabilities across the organization.

This role is ideal for someone who combines deep technical problem-solving with systems-level thinking and strong cross-functional leadership.

This is a full-time role that can be held from one of our US offices or remotely in the United States.

Role Responsibilities:

• Own programs: Lead complex, high-impact automation and evaluation initiatives across workflows or teams
• Design scalable solutions: Architect end-to-end workflows integrating datasets, evaluations, automations, and HITL processes; build the harness and reusable components other analysts build against; build evaluation pipelines that run against production without hands-on operation
• Establish standards: Define dataset standards, evaluation methodology, failure taxonomy, and quality measurement across AI Operations; verify that projects you don't run are meeting them; own the process by which those standards get made and used
• Own metric definitions: Create and maintain the metric definitions the org evaluates against, validate them against real production behavior, and revise them as models, tooling, and system architectures change
• Set the bar for production: Define the quality bar a system clears before it reaches production and where human review stays in the loop, and pull a system back when it stops meeting the bar
• Allocate evaluation depth by risk: Decide which systems get what depth of evaluation given finite capacity, name the risk accepted on the rest, and make that tradeoff visible to project stakeholders
• Keep the evaluation stack current: Define when a change to a model or platform requires re-baselining across the org and own the process for doing it; evaluate new models and AI capabilities as they ship and decide what the org adopts
• Improve systems at scale: Lead redesign of workflows, tooling, and processes to improve performance and durability across AI Operations, not only within your own programs
• Drive cross-functional alignment: Influence priorities and partner with Engineering, Product, and AI teams to deliver solutions
• Measure and communicate impact: Define success metrics and communicate performance and recommendations to leadership
• Elevate the team: Raise the bar through your work and actively mentor others by sharing approaches, guiding problem-solving, and enabling the team to build stronger automation and evaluation capabilities
• Drive innovation: Stay current on emerging tools and approaches; pilot and translate them into actionable improvements for the team

Minimum Requirements:

• 8+ years of professional experience AI, machine learning, operational automations, or a related field.
• Proven ability to lead complex automation or evaluation initiatives across systems or teams
• Experience designing, building, and scaling evaluation frameworks, datasets, and quality systems at scale for LLMs or AI products
• Fluency in SQL, JSON, APIs, and scripting with AI assistance, and ownership of the technical direction of the evaluation stack
• Deep working knowledge of LLM and agentic system behavior, automation platforms, and system design, demonstrated in systems you have built
• Experience with data pipelines, APIs, and production systems
• Experience in influencing cross-functional stakeholders and aligning priorities
• Demonstrated ability to define metrics and drive measurable business impact

Preferred Requirements:

• Experience mentoring or leading technical contributors
• Experience evaluating agentic systems in production at scale
• Experience setting technical standards adopted across an organization

Compensation:

At Fetch, we offer competitive compensation packages including base, equity, and benefits to the exceptional folks we hire. The base salary range for this position is $129,000 - $152,000. Discover our benefits and how our employees live rewarded at https://fetch.com/careers.

At Fetch, we'll give you the tools to feel healthy, happy and secure through:

• Equity: We offer full-time employees equity in Fetch, so that everyone can benefit from Fetch’s growth.
• 401k Match: Dollar-for-dollar match up to 4%.
• Benefits for humans and pets: We offer comprehensive medical, dental and vision plans for everyone including your pets.
• Continuing Education: Fetch provides ten thousand per year in education reimbursement.
• Employee Resource Groups: Take part in employee-led groups that are centered around fostering a diverse and inclusive workplace through events, dialogue and advocacy. The ERGs participate in our Inclusion Council with members of executive leadership.
• Paid Time Off: On top of our flexible PTO, Fetch observes 9 paid holidays, as well as our year-end week-long break.
• Robust Leave Policies: 20 weeks of paid parental leave for primary caregivers, 14 weeks for secondary caregivers, and a flexible return to work schedule.
• Calvin Care Cash: Employees who are welcoming new family members will also receive a one time $2,000 incentive to assist employees with covering the cost of childcare, clothing, diapers and much more!
• Flexible Work Environment: Collaborate with your team in one of our stunning offices, or you can work fully remotely from anywhere in the US. We’ll ensure you are equally equipped with the hardware and software you need to get your job done in the comfort of your home. (applicable for most roles)

Fetch is an equal opportunity employer that embraces diversity, inclusion, and respect for all individuals. We do not discriminate on the basis of race, color, religion, gender, gender identity or expression, sexual orientation, age, national origin, marital status, veteran status, disability, or any other characteristic protected by applicable law. Our commitment to inclusivity ensures that everyone is treated with dignity and has the opportunity to succeed based on their talent, skills, and potential.

Fetch also provides reasonable accommodations to qualified individuals with disabilities or those with sincerely held religious beliefs, as required by law. If you need assistance with the application process or require an accommodation, please contact us at [email protected].