{"id":3547955,"url":"https://alion.io/job/apple-senior-agent-evaluation-engineer-ai-evaluation-platform","title":"Senior Agent Evaluation Engineer, AI Evaluation Platform","company":{"id":12,"name":"Apple","domain":"apple.com","url":"https://alion.io/company/apple","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Apple Jobs","truth_index":{"grade":"A","score":90,"open_postings":860,"ghost_share":0,"stale_share":0.179,"repost_share":0.002,"time_to_fill_p50_days":95,"computed_at":"2026-11-02T06:13:15Z"}},"role":"Industrial Engineering","role_family":"Industrial Engineering","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Seattle, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":160000,"max_usd":302000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":6333},"experience_years_min":4,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"AI Agents","optional":false},{"name":"Braintrust","optional":false},{"name":"Claude Code","optional":false},{"name":"FastAPI","optional":false},{"name":"Function Calling","optional":false},{"name":"LangSmith","optional":false},{"name":"LLM","optional":false},{"name":"Model Context Protocol","optional":false},{"name":"Promptfoo","optional":false},{"name":"Pydantic","optional":false},{"name":"Python","optional":false},{"name":"Dask","optional":true},{"name":"Go","optional":true},{"name":"Kubernetes","optional":true},{"name":"MLFlow","optional":true},{"name":"Ray","optional":true},{"name":"Temporal","optional":true},{"name":"Weights & Biases","optional":true}],"status":"live","first_seen_at":"2026-05-19T18:39:42Z","employer_posted_date":"2026-10-30","last_verified_at":"2026-11-03T00:23:58Z","board_verified":true,"closed_at":null,"days_open":167,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":167},"description":"Join Apple Services Engineering to define how we measure whether AI agents actually work. We are seeking a senior engineer who has built agents and knows how to evaluate them rigorously. You will design evaluation suites, build and validate judges, and ship agents on our evaluation platform, so the platform is shaped by real use and not guesswork. You will partner with researchers, product teams, and platform engineers to turn evaluation methodology into tools other teams can adopt. We are looking for builders who move quickly, iterate in the open, and can tell a trustworthy evaluation from a theatrical one.\nDescription\nWe're building the evaluation platform that will serve all of Apple's generative AI and agent systems. Building agents got faster. Knowing whether they got better did not. This role closes that gap.\nYou'll build real agents and evaluate them: multi-turn trajectories, tool-use chains, non-deterministic outputs, and judge reliability. You'll write evaluation suites, validate judges against human labels, and measure variance across trials. Some agents you build will evaluate other systems (test-case generation, judging, failure triage). Others will be reference agents or built with partner teams. When the platform gets in your way, you'll file the gap, and often fix it yourself.\nThis is a hands-on role with a lot of autonomy. You'll work in Python with a team of 7-12 engineers.\nMinimum Qualifications\n* 4-8 years of software engineering experience shipping production software.\n* Hands-on agent builder. You've built LLM agents beyond demos: tool use, MCP, multi-step workflows, context management, error recovery.\n* Evaluation depth. You've designed evaluations for agents, not just models. You understand non-deterministic outputs, multi-trial measurement (pass@k, variance), trajectory scoring, and scoring drift. You've worked with tools like Inspect, Braintrust, LangSmith, or Promptfoo, or built your own.\nJudge rigor. You know how to build an LLM judge, validate it against human agreement, and know when not to trust it.\nFast from idea to prototype. You can point to things you built quickly, then iterated on or shipped. You can also tell us about a prototype you killed, and why.\nFluent with AI coding tools. You use Claude Code or equivalents daily, including skills, slash commands, and agent-style workflows.\nPython as primary language. Strong with FastAPI, Pydantic, and the ecosystem. Your code is clean, tested, and easy for the next person to pick up.\nJudgment on speed vs. discipline. You know when a quick prototype is enough and when you need a design doc, tests, and a careful rollout.\nCross-team collaboration. You've worked closely with researchers, product teams, or platform engineers. You write clearly and flag blockers early.\nPreferred Qualifications\nExperience working on developer platforms, internal tools, or SDKs\nProduction experience with LLM/agent systems - building, evaluating, or operating them\nFamiliarity with job orchestration frameworks (Temporal.io, Airflow, or similar)\nDistributed compute experience (Ray, Dask, or Kubernetes-based job systems)\nExperience with experiment tracking or ML lifecycle tooling (Weights & Biases, MLflow, etc.)\nStartup or early-stage experience where you wore multiple hats and shipped under constraint","description_format":"text","description_chars":3310,"description_truncated":false,"requirements":{"experience_years_min":4,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Operating Systems","Laptops","Smartwatches & Fitness Trackers","Tablets"],"lifecycle":[{"event":"open","at":"2026-10-30T21:27:41Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 6,350 · green card filings: 37","filings_12m":6350,"filings_prev_12m":6527,"green_card_filings_12m":37,"median_offered_wage_usd":177355,"route":null,"cap_exempt":false,"checked_at":"2026-10-29T16:27:21+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":0}],"liveness":{"score":32,"band":"fade","label":"Fading","p_open":1,"p_active":0.877,"p_room":0.36,"age_days":166,"expected_fill_days":95,"reasons":["conf:2","urgency","velocity","win:tail","crowd:brand"],"computed_at":"2026-11-02T06:13:15Z"},"pay":null,"html_url":"https://alion.io/job/apple-senior-agent-evaluation-engineer-ai-evaluation-platform","json_url":"https://alion.io/job/apple-senior-agent-evaluation-engineer-ai-evaluation-platform.json","meta":{"generated_at":"2026-11-03T03:14:25Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":4318,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-04T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":12},"rest":"https://alion.io/mcp/rest/get_company?id=12"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fapple-senior-agent-evaluation-engineer-ai-evaluation-platform"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fapple-senior-agent-evaluation-engineer-ai-evaluation-platform"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fapple-senior-agent-evaluation-engineer-ai-evaluation-platform"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/apple-senior-agent-evaluation-engineer-ai-evaluation-platform\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fapple-senior-agent-evaluation-engineer-ai-evaluation-platform"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3547955}},{"tool":"search_jobs","arguments":{"company":"apple.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3547955},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}