{"id":449658,"url":"https://alion.io/job/vals-ai-evaluations-engineer","title":"Evaluations Engineer","company":{"id":173609,"name":"Vals AI","domain":"vals.ai","url":"https://alion.io/company/vals-ai","size_band":"51-200","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"B","score":75,"open_postings":4,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-10-23T05:54:15Z"}},"role":"Industrial Engineering","role_family":"Industrial Engineering","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["San Francisco, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":130000,"max":185000,"currency":"USD","period":"year","gross":null,"usd_annual":185000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Git","optional":false},{"name":"LLM","optional":false},{"name":"Python","optional":false},{"name":"AWS","optional":true},{"name":"DeepSeek","optional":true},{"name":"Django","optional":true},{"name":"JavaScript","optional":true},{"name":"Machine Learning","optional":true},{"name":"NLP","optional":true},{"name":"OpenAI","optional":true},{"name":"React.js","optional":true},{"name":"Snorkel","optional":true}],"status":"live","first_seen_at":"2026-06-27T00:56:00Z","employer_posted_date":"2026-06-27","last_verified_at":"2026-10-23T09:40:29Z","board_verified":true,"closed_at":null,"days_open":118,"trust":{"level":"stale","repost_count":0,"flags":["stale"],"days_open":118},"description":"Please note that this process is designed to run extremely quickly (~1 week end-to-end) and includes a take-home component.\nAbout the Role\nWe are looking for strong engineers to join our team and own the leaderboards that appear on Vals AI.\nYou'll be responsible for testing new models against our benchmarks as they're released; covering tasks in law, tax, coding, finance, social mobility, and more. You will analyze error modes of models, evaluate their strengths and weaknesses, and work with our communications team to release results.\nOur results are used by startups, enterprises, and research labs alike. We work with all the major foundation model labs, some of the largest financial institutions, and hospital systems in the world. Our work has been featured by the Wall Street Journal, Washington Post, and Bloomberg.\nWe are building the standard for evaluating the ability of LLMs to perform real-world tasks. You will contribute directly to the leaderboards that make this possible.\nWhat You’ll Do\nEvaluate new LLM model releases across the Vals AI suite of benchmarks\n\nWork directly with both open-source and closed-source foundation model labs in evaluating model performance\n\nUse tools like Docent to analyze common failure modes and patterns in model performance\n\nWork directly with our social media team to post interesting findings and results\n\nAdd new models and maintain integrations in our model library\n\nHelp improve and maintain the infrastructure we use to run benchmarks (agentic and non-agentic).\n\nThis role follows the rhythm of model releases. Expect intense sprints in the days following a major launch, and calmer stretches in between releases.\nRequirements\nFamiliarity with the LLMs: You should already be familiar with the space - the current leading models, relative performance across them, how to use large language models in practice.\n\nStrong engineering fundamentals: You can build and ship quickly with high quality. You should have a track record of building things of significant scope (at jobs, side projects, open source, etc.)\n\nPython expertise: Significant experience in Python, especially in a professional setting.\n\nTeam collaboration: Experience working in development sprints, Git workflows, and pull request reviews.\n\nStrong work ethic: Willingness to work long hours during model releases and get high-quality results out under tight deadlines.\n\nLocation: We are an in-person team based in San Francisco. We will support your relocation or transportation as needed.\n\nNice-to-Haves\nPrevious experience with benchmarking large language models, or creating benchmarks\n\nPrevious experience working at a startup or starting your own company\n\nTechnical writing experience and ability\n\nMachine learning research experience\n\nWhat We Offer\nHighly competitive salary and meaningful ownership. Excellence is well rewarded.\n\nRelocation and transportation support\n\nHealth/dental insurance coverage\n\nLunch and dinner provided, free snacks/coffee/drinks\n\n401K plan\n\nUnlimited PTO\n\n$1,500 housing stipend (within one-mile radius)\n\nAbout Us\nFounding team: The core methodology behind this platform comes from NLP evaluation research we conducted at Stanford. We raised a $5M seed from some of the top institutional and angel investors in the valley. Our team has prior work experience at NVIDIA, Meta, Microsoft, Palantir and HRT. Collectively, we have over 300 citations in our published work. Our early team includes Stanford PhDs, ex-Jane Street quants, and the first designer at Snorkel.\nWe recently announced our $40M Series A at a $400M valuation, led by Andreessen Horowitz, with participation from existing investors 8VC, Pear VC, and Bloomberg and new investors Hudson River Trading and NextLadder Ventures.\nTech stack: We use Python for most things at Vals AI. Our platform is built on Django, with a React frontend. All of the infra is on AWS using CDK for IaC.\nWhat We're Looking For\nLearning velocity: The role encompasses a wide variety of tasks. Rather than expecting you to be an expert on Day 1, we are looking for someone who can learn new skills and technologies quickly.\n\nOwnership: Working in a small, talent-dense team, we expect everyone to show initiative to build where it's needed, not where it's asked. We strive for autonomy over consensus.\n\nIntensity: The LLM landscape is constantly changing. Foundation model labs are continuously pushing the frontier. The unicorn companies that will emerge from this technology shift are being built now. Those that win will have an incredibly high speed of execution.\n\nSolution-oriented mindset: We're looking for people who see opportunities to craft solutions at each juncture, not those who pass hard problems to others or admit defeat.\n\nVals in the Media:\nAI tools mostly fumble basic financial tasks, study finds\n\nThe Winners (and Losers) of This New Vibe-Coding Benchmark Will Surprise You\n\nOpenAI’s Less-Flashy Rival Might Have a Better Business Model\n\nMeta Announces New AI Model in Major Test of Company’s Ambitions\n\nDeepSeek’s Sequel Set to Extend China’s Reach in Open-Source A.I.","description_format":"text","description_chars":5092,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["401k plan","Dental insurance","Insurance coverage","Unlimited PTO"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":true,"industries":["AI Evaluation & Observability"],"lifecycle":[{"event":"open","at":"2026-09-06T08:17:29Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 2","filings_12m":2,"filings_prev_12m":0,"green_card_filings_12m":0,"median_offered_wage_usd":162500,"route":null,"cap_exempt":false,"checked_at":"2026-10-15T11:24:14+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":0}],"liveness":{"score":8,"band":"cold","label":"Long shot","p_open":1,"p_active":0.273,"p_room":0.28,"age_days":118,"expected_fill_days":31,"reasons":["conf:1","win:tail","crowd:"],"computed_at":"2026-10-23T05:54:15Z"},"pay":{"stated_usd_annual":185000,"is_top_pay":true},"html_url":"https://alion.io/job/vals-ai-evaluations-engineer","json_url":"https://alion.io/job/vals-ai-evaluations-engineer.json","meta":{"generated_at":"2026-10-23T12:12:52Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":18609,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-24T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":173609},"rest":"https://alion.io/mcp/rest/get_company?id=173609"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fvals-ai-evaluations-engineer"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fvals-ai-evaluations-engineer"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fvals-ai-evaluations-engineer"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/vals-ai-evaluations-engineer\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fvals-ai-evaluations-engineer"}]}