{"id":91475,"url":"https://alion.io/job/nous-research-machine-learning-engineer-evals","title":"Machine Learning Engineer, Evals","company":{"id":35054,"name":"Nous Research","domain":"nousresearch.com","url":"https://alion.io/company/nous-research","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":null},"role":"AI/ML","role_family":"AI/ML","seniority":"middle","employment_type":"full_time","work_mode":"remote","remote_scope":"stated_regions","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"inferred","locations":[],"countries":[],"hiring_countries":["US","AR","BR","CA","MX","AG","AW","BS","BB","BZ","BM","BO","VG","BQ","KY","CL","CR","CW","DM","DO","EC","SV","FK","GF","GL","GD","GP","GT","GY","HT","HN","JM","MQ","MS","NI","PA","PY","PE","PR","KN"],"hiring_countries_total":48,"salary":null,"salary_estimate":null,"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"CI/CD","optional":false},{"name":"Docker","optional":false},{"name":"Fine-tuning","optional":false},{"name":"Git","optional":false},{"name":"Harbor","optional":false},{"name":"Linux","optional":false},{"name":"LLM","optional":false},{"name":"NVIDIA NeMo","optional":false},{"name":"Python","optional":false},{"name":"RAG","optional":false},{"name":"Red Teaming","optional":false},{"name":"RLHF","optional":false}],"status":"live","first_seen_at":"2026-08-13T16:28:01Z","employer_posted_date":"2026-08-13","last_verified_at":"2026-10-17T22:24:54Z","board_verified":true,"closed_at":null,"days_open":65,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":65},"description":"The Role\nYou'll work across the lab on agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together. This is a high-growth, high-ownership role on a small team, and you'll ship evaluation infrastructure that researchers depend on from day one.\nResponsibilities\nRun the full eval pipeline end to end and reproduce known results during onboarding, pairing with a senior engineer on your first task\n\nBuild a judge calibration protocol: sample human-labeled decisions, measure agreement (κ, per-class P/R), identify drift zones, and document it so anyone can re-run it\n\nExtend an existing benchmark (GAIA, τ-Bench, SWE-bench slice, etc.) with new tasks targeting known capability gaps, including the prompt, environment, rubric, automated grader, and QA\n\nRun failure analysis on model outputs: categorize failure modes, quantify prevalence, and write up findings with recommendations for training data, judge prompts, or benchmark changes\n\nOwn a recurring eval workflow (weekly regression suite, judge drift dashboard, red-team evaluation for a new capability) and ship tooling researchers actually use\n\nQualifications\n3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job\n\nExperience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short\n\nHands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents\n\nSolid Python. You write clean, tested, version-controlled code that a colleague could run without you babysitting it\n\nComfort with Git, CI/CD basics, Docker, and the Linux command line (SSH, tmux, debugging a remote job)\n\nUnderstanding of basic eval statistics: why accuracy misleads on imbalanced judges, what Cohen's κ measures, how to think about confidence intervals on a metric\n\nAt least 3 of the following: you can explain why LLM-as-judge needs calibration; you've done failure analysis and can tell model bugs apart from prompt, grader, or retrieval issues; you know at least two agent benchmarks (GAIA, AgentBench, τ-Bench, MINT, SWE-bench, WebShop, ALFWorld) and a limitation of each; you've designed or extended an eval dataset with happy paths, edge cases, and adversarial examples; you've thought about non-determinism in eval, how you sample, how many runs, how you report variance\n\nYou communicate clearly to both researchers and engineers, in the right language for each\n\nYou're comfortable with ambiguity, can turn a half-formed request into a plan, and know when to ask for help\n\nPreferred\nRLVR / RLHF pipeline experience\n\nTraining data curation experience\n\nDistributed eval orchestration experience\n\nBenchmark design from scratch\n\nRed teaming and adversarial eval experience\n\nFamiliarity with psychometrics or measurement theory","description_format":"text","description_chars":3006,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"},{"name":"Argentina","iso":"AR","kind":"region"},{"name":"Brazil","iso":"BR","kind":"region"},{"name":"Canada","iso":"CA","kind":"region"},{"name":"Colombia","iso":null,"kind":"region"},{"name":"Mexico","iso":"MX","kind":"region"},{"name":"Anguilla","iso":null,"kind":"region"},{"name":"Antigua and Barbuda","iso":"AG","kind":"region"},{"name":"Aruba","iso":"AW","kind":"region"},{"name":"Bahamas","iso":"BS","kind":"region"},{"name":"Barbados","iso":"BB","kind":"region"},{"name":"Belize","iso":"BZ","kind":"region"},{"name":"Bermuda","iso":"BM","kind":"region"},{"name":"Bolivia","iso":"BO","kind":"region"},{"name":"British Virgin Islands","iso":"VG","kind":"region"},{"name":"Bonaire, Sint Eustatius and Saba","iso":"BQ","kind":"region"},{"name":"Cayman Islands","iso":"KY","kind":"region"},{"name":"Chile","iso":"CL","kind":"region"},{"name":"Costa Rica","iso":"CR","kind":"region"},{"name":"Curaçao","iso":"CW","kind":"region"},{"name":"Dominica","iso":"DM","kind":"region"},{"name":"Dominican Republic","iso":"DO","kind":"region"},{"name":"Ecuador","iso":"EC","kind":"region"},{"name":"El Salvador","iso":"SV","kind":"region"},{"name":"Falkland Islands","iso":"FK","kind":"region"},{"name":"French Guiana","iso":"GF","kind":"region"},{"name":"Greenland","iso":"GL","kind":"region"},{"name":"Grenada","iso":"GD","kind":"region"},{"name":"Guadeloupe","iso":"GP","kind":"region"},{"name":"Guatemala","iso":"GT","kind":"region"},{"name":"Guyana","iso":"GY","kind":"region"},{"name":"Haiti","iso":"HT","kind":"region"},{"name":"Honduras","iso":"HN","kind":"region"},{"name":"Jamaica","iso":"JM","kind":"region"},{"name":"Martinique","iso":"MQ","kind":"region"},{"name":"Montserrat","iso":"MS","kind":"region"},{"name":"Nicaragua","iso":"NI","kind":"region"},{"name":"Panama","iso":"PA","kind":"region"},{"name":"Paraguay","iso":"PY","kind":"region"},{"name":"Peru","iso":"PE","kind":"region"},{"name":"Puerto Rico","iso":"PR","kind":"region"},{"name":"Saint Kitts and Nevis","iso":"KN","kind":"region"},{"name":"Saint Lucia","iso":"LC","kind":"region"},{"name":"Saint Vincent and the Grenadines","iso":null,"kind":"region"},{"name":"Saint-Pierre and Miquelon","iso":"PM","kind":"region"},{"name":"Sint Maarten","iso":"SX","kind":"region"},{"name":"Suriname","iso":"SR","kind":"region"},{"name":"Trinidad and Tobago","iso":"TT","kind":"region"},{"name":"Turks and Caicos Islands","iso":"TC","kind":"region"},{"name":"Uruguay","iso":"UY","kind":"region"},{"name":"United States Virgin Islands","iso":"VI","kind":"region"}],"hiring_excludes":[],"relocation_offered":false,"industries":["LLM & Generative AI","Foundation Models","AI Research Labs"],"lifecycle":[{"event":"open","at":"2026-08-13T16:28:01Z"}],"visa":[],"liveness":{"score":23,"band":"cold","label":"Long shot","p_open":1,"p_active":0.52,"p_room":0.45,"age_days":64,"expected_fill_days":35,"reasons":["conf:2","velocity","win:tail"],"computed_at":"2026-10-17T05:46:00Z"},"pay":null,"html_url":"https://alion.io/job/nous-research-machine-learning-engineer-evals","json_url":"https://alion.io/job/nous-research-machine-learning-engineer-evals.json","meta":{"generated_at":"2026-10-18T00:45:04Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":1292,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-19T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":35054},"rest":"https://alion.io/mcp/rest/get_company?id=35054"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fnous-research-machine-learning-engineer-evals"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fnous-research-machine-learning-engineer-evals"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fnous-research-machine-learning-engineer-evals"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/nous-research-machine-learning-engineer-evals\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fnous-research-machine-learning-engineer-evals"}]}