{"id":145358,"url":"https://alion.io/job/variance-research-engineer-evals","title":"Research Engineer, Evals","company":{"id":3103,"name":"Variance","domain":"variance.com","url":"https://alion.io/company/variance","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"B","score":83,"open_postings":9,"ghost_share":0,"stale_share":0.667,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-10-17T05:46:00Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["San Francisco, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":180000,"max":300000,"currency":"USD","period":"year","gross":null,"usd_annual":300000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Post-training","optional":false}],"status":"live","first_seen_at":"2026-03-31T00:42:45Z","employer_posted_date":"2026-03-31","last_verified_at":"2026-10-17T13:40:25Z","board_verified":true,"closed_at":null,"days_open":200,"trust":{"level":"stale","repost_count":0,"flags":["stale"],"days_open":200},"description":"Role\nAt Variance, we are teaching machines to make the hardest judgment calls at scale. That means building AI agents for the high-stakes gray area of risk investigations, fraud, and identity reviews.\nWe’re a small, talent-dense team in San Francisco working on a problem at the edge of what AI systems can reliably do: making good decisions in messy, adversarial, real-world environments. We focus on building, high-consequence systems problems where the edge cases matter most.\nWe’re looking for a Research Engineer to help define how we measure and improve model quality. You’ll build the benchmarks, datasets, tooling, and evaluation loops that tell us whether our systems are actually getting better on the tasks that matter. This role sits at the center of research, product, and engineering. It is about creating rigorous, domain-specific evaluations that reflect real customer workflows, expose meaningful failure modes, and drive the next generation of model and agent improvements.\nYou’re a fit if you:\nCare deeply about craftsmanship and have strong opinions about model quality, measurement, and experimental rigor\n\nWant to work on core model and agent behavior, not just surface-level product metrics\n\nAre excited by the challenge of defining what “good” looks like in messy, high-stakes environments\n\nThink in tight loops: hypothesis, benchmark design, evaluation, failure analysis, iteration\n\nHave strong engineering fundamentals and like building robust systems around ambiguous research problems\n\nThrive in environments where success criteria are initially underspecified and need to be sharpened through work\n\nAre willing to do the work in the trenches: reviewing outputs, grading edge cases, curating datasets, and refining tasks until the evaluation actually measures what matters\n\nCare deeply about building systems that protect people from fraud, scams, and abuse\n\nWhat you’ll do\nBuild proprietary benchmarks and datasets to evaluate models and model systems on fraud, identity, and risk workflows\n\nDesign and run offline and online evals that measure model performance on real customer tasks, not just abstract benchmarks\n\nDefine quality metrics for judgment systems, including precision, calibration, consistency, abstention, and failure handling\n\nStudy where models and agents break, and turn those failures into better evals, better datasets, and better training loops\n\nBuild reusable evaluation tools and quality building blocks that can be used across different product surfaces and workflows\n\nPartner closely with research, engineering, product, and design to improve system quality through rigorous experimentation\n\nHelp create a strong culture of scientific experimentation, clear measurement, and continuous iteration\n\nPush the boundary of how AI systems are evaluated in regulated, adversarial, and high-consequence environments\n\nWhat success looks like\nWe have a clear, trusted view of how our systems perform across the workflows that matter most\n\nOur evals predict real-world quality better than generic benchmarks\n\nWe identify meaningful failure modes earlier and improve system behavior faster\n\nWe develop differentiated datasets, benchmarks, and quality loops that compound over time\n\nResearch and engineering teams use your work to make better decisions about what to train, ship, and improve next\n\nVariance becomes known for rigorous, domain-specific evaluation of judgment systems\n\nPreferred background\nExperience training, evaluating, or improving modern ML systems\n\nStrong programming skills and comfort working in research-heavy codebases\n\nExperience building benchmarks, datasets, evaluation pipelines, or quality systems\n\nFamiliarity with LLMs, agent systems, retrieval, post-training, or adjacent areas\n\nAbility to design clean experiments and draw reliable conclusions from noisy results\n\nStrong engineering judgment and a bias toward building\n\nInterest in fraud, risk, trust and safety, compliance, or other regulated and adversarial domains\n\nOur culture\nWe believe in ownership, urgency, and craft. We enjoy spirited debate, wild ideas, and building things we’re proud of. We’re fully in-person in San Francisco.\nWhat we offer\nCompetitive salary and meaningful equity\n\nPlatinum-level medical, dental, and vision insurance\n\nUnlimited PTO, sick leave, and parental leave\n\nUp to $100 per month in reimbursement for personal health and wellness expenses\n\n401(k) plan","description_format":"text","description_chars":4413,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Equity","Parental leave","Unlimited PTO","Vision insurance"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Fraud Detection","RegTech, AML & Compliance","AI Agents"],"lifecycle":[{"event":"open","at":"2026-03-31T00:42:45Z"}],"visa":[],"liveness":{"score":6,"band":"cold","label":"Long shot","p_open":1,"p_active":0.199,"p_room":0.28,"age_days":200,"expected_fill_days":33,"reasons":["conf:1","win:tail","crowd:"],"computed_at":"2026-10-17T05:46:00Z"},"pay":{"stated_usd_annual":300000,"is_top_pay":true},"html_url":"https://alion.io/job/variance-research-engineer-evals","json_url":"https://alion.io/job/variance-research-engineer-evals.json","meta":{"generated_at":"2026-10-17T15:25:39Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":20868,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-18T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3103},"rest":"https://alion.io/mcp/rest/get_company?id=3103"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fvariance-research-engineer-evals"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fvariance-research-engineer-evals"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fvariance-research-engineer-evals"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/variance-research-engineer-evals\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fvariance-research-engineer-evals"}]}