{"id":127297,"url":"https://alion.io/job/thinking-machines-lab-research-post-training-evals","title":"Research, Post-Training Evals","company":{"id":14382,"name":"Thinking Machines Lab","domain":"thinkingmachines.ai","url":"https://alion.io/company/thinking-machines-lab","size_band":"501-1000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":{"grade":"B","score":75,"open_postings":20,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":29,"computed_at":"2026-11-04T06:34:25Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":null,"employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["San Francisco, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":350000,"max":475000,"currency":"USD","period":"year","gross":null,"usd_annual":475000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":true,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Machine Learning","optional":false},{"name":"Post-training","optional":false},{"name":"AI Agents","optional":true},{"name":"JAX","optional":true},{"name":"LLM","optional":true},{"name":"Python","optional":true},{"name":"PyTorch","optional":true},{"name":"Reinforcement Learning","optional":true},{"name":"TensorFlow","optional":true}],"status":"live","first_seen_at":"2026-08-27T02:29:43Z","employer_posted_date":"2026-08-27","last_verified_at":"2026-11-04T04:45:30Z","board_verified":true,"closed_at":null,"days_open":69,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":69},"description":"About Thinking Machines\nThe mission of Thinking Machines is to build AI that extends human will and judgment. We are training frontier models with Inkling, developing Tinker to let people make models their own, and crafting interfaces that broaden human-AI communication. We believe the future worth building is human, and we're hiring people who want to build it.\nAbout the Role\nWe’re looking for a researcher to help develop reliable model evaluations for research signals. This role spans evaluation creation, usability, auditing, and efficiency.\nYou’ll work closely with researchers and engineers across post-training and the broader research organization. Depending on your interests and experience, you may focus on one area or work across several of these problems.\nWhat You’ll Do\nCreate internal evaluations and research signals for capabilities and behaviors important to model research and post-training.\n\nDevelop usability evaluations that measure whether models are genuinely useful in real research and product workflows, and partner with the data flywheel to turn evaluation insights into better data and training signals.\n\nImprove evaluation robustness, including grader reliability, ambiguous ground truth, evaluator disagreement, false positives and negatives, and gaps between measured and intended behavior.\n\nBuild benchmark auditing methodologies that help researchers understand, trust, and appropriately use evaluation signals.\n\nDevelop specialized agentic evaluation environments and user simulators, including supporting harness development and studying cross-user, cross-harness and cross-environment generalization.\n\nDevelop evaluations for personalized preferences, biases, values, and other nuanced dimensions of model behavior in collaboration with post-training crafting.\n\nSkills and Qualifications\nMinimum qualifications:\nBachelor’s degree or equivalent experience in Computer Science, Machine Learning, Physics, Mathematics, or a related discipline with strong theoretical and empirical grounding.\n\nExperience designing, building, or analyzing evaluations, benchmarks, datasets, graders, or other measurement systems.\n\nStrong written and verbal communication skills, with the ability to collaborate effectively across research and engineering teams.\n\nPreferred qualifications - we encourage you to apply if you meet some but not all of these:\nExperience with LLMs, post-training, reinforcement learning, or agentic systems.\n\nExperience with evaluation auditing, human evaluations, LLM-judges, or open-ended task evaluation.\n\nExperience with agentic evaluation, harnesses, long-horizon tasks, or RL environments.\n\nExperience evaluating preferences, personalization, biases, values, or other nuanced model behaviors.\n\nTrack record of developing new evaluation methodologies or research signals that meaningfully influenced model development.\n\nProficiency in Python and familiarity with at least one deep learning framework (e.g., PyTorch, TensorFlow, or JAX). Comfortable with debugging distributed training and writing code that scales.\n\nStrong research judgment: clean ablations, honest baselines, and clear technical writing.\n\nPhD in Computer Science, Machine Learning, Physics, Mathematics, or a related discipline with strong theoretical and empirical grounding; or, equivalent industry research experience.\n\nLogistics\nLocation: This role is based in San Francisco, California.\n\nCompensation: Depending on background, skills and experience, the expected annual salary range for this position is $350,000 - $475,000 USD.\n\nVisa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.\n\nBenefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.","description_format":"text","description_chars":3892,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"phd","optional":true},"security_clearance":false,"languages":[]},"benefits":["Parental leave","Unlimited PTO"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":true,"industries":["Multimodal AI","Foundation Models","AI Research Labs","AI Development Tools"],"lifecycle":[{"event":"open","at":"2026-08-27T02:29:43Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 30","filings_12m":30,"filings_prev_12m":8,"green_card_filings_12m":0,"median_offered_wage_usd":192500,"route":null,"cap_exempt":false,"checked_at":"2026-10-29T16:27:21+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":28}],"liveness":{"score":8,"band":"cold","label":"Long shot","p_open":1,"p_active":0.278,"p_room":0.28,"age_days":69,"expected_fill_days":29,"reasons":["conf:1","stale_co","velocity","win:tail","crowd:"],"computed_at":"2026-11-04T06:34:25Z"},"pay":{"stated_usd_annual":475000,"is_top_pay":true},"html_url":"https://alion.io/job/thinking-machines-lab-research-post-training-evals","json_url":"https://alion.io/job/thinking-machines-lab-research-post-training-evals.json","meta":{"generated_at":"2026-11-04T08:42:53Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":13519,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-05T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":14382},"rest":"https://alion.io/mcp/rest/get_company?id=14382"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fthinking-machines-lab-research-post-training-evals"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fthinking-machines-lab-research-post-training-evals"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fthinking-machines-lab-research-post-training-evals"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/thinking-machines-lab-research-post-training-evals\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fthinking-machines-lab-research-post-training-evals"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":127297}},{"tool":"search_jobs","arguments":{"company":"thinkingmachines.ai","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":127297},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}