{"id":2026384,"url":"https://alion.io/job/servicenow-senior-machine-learning-engineer-agent-evaluation-platform","title":"Senior Machine Learning Engineer (Agent Evaluation Platform)","company":{"id":97,"name":"ServiceNow","domain":"servicenow.com","url":"https://alion.io/company/servicenow","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"SmartRecruiters","truth_index":{"grade":"B","score":80,"open_postings":34,"ghost_share":0,"stale_share":1,"repost_share":0,"time_to_fill_p50_days":17,"computed_at":"2026-10-12T06:55:09Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Santa Clara, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":190000,"max_usd":361000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":1129},"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Fine-tuning","optional":false},{"name":"LLM","optional":false},{"name":"Machine Learning","optional":false},{"name":"Model Distillation","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Python","optional":false},{"name":"Reward Modeling","optional":false},{"name":"RLAIF","optional":false},{"name":"RLHF","optional":false},{"name":"SFT","optional":false}],"status":"closed","first_seen_at":"2026-09-06T00:17:41Z","employer_posted_date":"2026-10-07","last_verified_at":"2026-10-12T00:01:14Z","board_verified":false,"closed_at":"2026-10-12T00:01:14Z","days_open":35,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":35},"description":"Join our team as a Senior Machine Learning Engineer, where you'll be responsible for building the judgement layer of our agent evaluation platform. This role involves developing rubrics, judges, and calibration methodologies, as well as fine-tuning small models and evaluating their performance. You'll work on applied machine learning in a dynamic and challenging environment, with opportunities for growth and impact.\nMissions\nConstruire la couche de jugement de notre plateforme d'évaluation des agents, y compris les rubriques, les juges, la calibration par rapport aux étiquettes humaines.\nDévelopper un juge calibré qui peut évaluer une trajectoire et servir de signal de récompense pour l'optimisation de l'agent.\nCollaborer avec l'équipe d'annotation pour établir une boucle de calibration continue contre les trajectoires étiquetées par des humains.\nProfil recherché\n - Fine-tuning and evaluating small models: SFT, preference tuning, distillation- 5+ years in applied ML, data science, or ML-adjacent engineering, with a track record of work that shipped and got used\n- Strong Python, and the discipline to ship production-grade code rather than notebooks\n- Experience in at least 3 of these:\n- Search ranking, recsys, or online experimentation evaluation - golden-set staleness, offline/online divergence, side-by-side rater agreement. This is the closest existing analog to agentic eval, and it transfers directly\n- Ability to think and communicate clearly about complex problems - a large part of this job is convincing engineers that a number means what you say it means, and being right\n- Reward modeling, RLHF/RLAIF, or process reward models\n- A high degree of ownership and a bias toward shipping at startup pace\n- Prompt engineering as an engineering discipline - versioned, tested, and measured, not tuned by vibes\n- Strong applied ML fundamentals, and comfort treating LLMs as a component you evaluate, prompt, and fine-tune rather than one you pretrain\n- Experience turning subjective human judgement into a measurement that holds up - one that other people, and ideally other models, can act on. This is the core of the job\n- Human annotation programs: rubric authoring, label quality, and annotator throughput as a real constraint\n- Agent trajectory analysis and step-level fault attribution\n- LLM-as-judge or automated evaluation design, and calibrating it against human judgement\n- Comfort with ambiguity, and the judgement to know when a measurement is good enough to act on","description_format":"text","description_chars":2500,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-10-07T13:28:08Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 664 · green card filings: 278","filings_12m":664,"filings_prev_12m":942,"green_card_filings_12m":278,"median_offered_wage_usd":163402,"route":null,"cap_exempt":false,"checked_at":"2026-10-03T21:08:04+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":300}],"liveness":null,"pay":null,"html_url":"https://alion.io/job/servicenow-senior-machine-learning-engineer-agent-evaluation-platform","json_url":"https://alion.io/job/servicenow-senior-machine-learning-engineer-agent-evaluation-platform.json","meta":{"generated_at":"2026-10-12T17:14:59Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":27602,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-13T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":97},"rest":"https://alion.io/mcp/rest/get_company?id=97"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fservicenow-senior-machine-learning-engineer-agent-evaluation-platform"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fservicenow-senior-machine-learning-engineer-agent-evaluation-platform"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fservicenow-senior-machine-learning-engineer-agent-evaluation-platform"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/servicenow-senior-machine-learning-engineer-agent-evaluation-platform\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fservicenow-senior-machine-learning-engineer-agent-evaluation-platform"}]}