{"id":3898243,"url":"https://alion.io/job/apple-ai-software-evaluation-engineer-siri-ai-quality","title":"AI Software Evaluation Engineer, Siri AI Quality","company":{"id":12,"name":"Apple","domain":"apple.com","url":"https://alion.io/company/apple","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Apple Jobs","truth_index":{"grade":"A","score":89,"open_postings":835,"ghost_share":0,"stale_share":0.217,"repost_share":0.006,"time_to_fill_p50_days":90,"computed_at":"2026-11-09T06:03:00Z"}},"role":"Backend","role_family":"Backend","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Cupertino, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"Agentic Workflows","optional":false},{"name":"AI Agents","optional":false},{"name":"CI/CD","optional":false},{"name":"Function Calling","optional":false},{"name":"LLM","optional":false},{"name":"Prompt Engineering","optional":false},{"name":"Python","optional":false},{"name":"SQL","optional":false},{"name":"Model Context Protocol","optional":true},{"name":"Snowflake","optional":true},{"name":"Tableau","optional":true}],"status":"live","first_seen_at":"2026-11-06T19:52:58Z","employer_posted_date":"2026-11-06","last_verified_at":"2026-11-09T18:49:54Z","board_verified":true,"closed_at":null,"days_open":2,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":2},"description":"Play a part in the next revolution in human-computer interaction and redefine mobile computing through voice interaction!\nApple's Siri AI Quality team is looking for a passionate and skilled AI Software Evaluation Engineer to join our mission of delivering the highest-quality AI experiences to millions of users worldwide. We own the end-to-end Siri pre-ship evaluation dataset and the synthetic personas that power it. In this role, you will combine agentic engineering, data analytics and AI evaluation experience to help us catch quality issues early, automate repetitive eval operations, and expand coverage as Siri's features and platforms grow.\nDescription\nIn this role, you will build the systems that evaluate Siri's AI experiences and turn the results into action. Your work will span multiple areas of focus:\n* Evaluation data insights. Investigate why test scenarios regress and which issues recur. Turn failure and bug data into root-cause findings and recommendations that product, engineering and partner teams act on.\n* Coverage analysis. Find gaps across app intents, features and product areas, and partner with domain owners to close them. Build and report on dataset quality, issue trends and coverage.\n* Agentic scenario generation, triage and auto-fix. Design agentic workflows that propose, ground, validate, triage and nominate new evaluation scenarios with minimal manual effort. Extend automated screening, triage and fix tooling to new domains and device platforms.\nYou will work with engineering, product, evaluator, persona and data-platform teams to define quality standards and deliver evaluation systems that guide Siri's development.\nIf you are a self-motivated engineer with a strong analytical mind, a bias toward automation and a desire to improve a product used by millions, we encourage you to apply.\nMinimum Qualifications\n5+ years of industry experience in software engineering, quality engineering, data engineering or ML engineering.\n2+ years of hands-on experience building LLM-based applications or agents, including tool use, multi-step workflows and prompt engineering.\nExperience designing and running AI or LLM evaluation: building evaluation datasets, defining metrics and rubrics, LLM-as-judge or automated evaluators, and analyzing failures.\nStrong data analytics skills, including SQL and Python, and experience turning large datasets into clear insights and dashboards.\nStrong programming skills in Python and experience with automation and CI/CD.\nStrong communication and collaboration skills, with the ability to influence cross-functional partners and work across teams.\nPreferred Qualifications\nExperience with agent frameworks and developer tooling, such as MCP servers, skills or plugins, and agentic testing.\nExperience with voice assistants, conversational AI or multi-turn dialogue evaluation.\nExperience with data schemas, validation frameworks, and structured data formats (e.g. JSON).\nExperience with data visualization and BI tools (e.g. Tableau) and with data warehouses (e.g. Snowflake).\nFamiliarity with coverage analysis, traffic or taxonomy analysis, and root-cause analysis of flaky or regressing tests.\nComfortable working in a fast-paced, collaborative environment with shifting priorities.","description_format":"text","description_chars":3264,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Operating Systems","Laptops","Smartwatches & Fitness Trackers","Tablets"],"lifecycle":[{"event":"open","at":"2026-11-06T23:40:21Z"}],"visa":[{"country":"US","licensed_sponsor":true,"adverts_offering_sponsorship_24m":0,"sanctioned_now":false,"evidence":"H-1B filings in 12 months: 6,350 · green card filings: 39","filings_12m":6350,"filings_prev_12m":6528,"green_card_filings_12m":39,"median_offered_wage_usd":177313,"route":null,"cap_exempt":false,"checked_at":"2026-11-09T07:32:21+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":2996}],"liveness":{"score":99,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.993,"p_room":1,"age_days":2,"expected_fill_days":90,"reasons":["conf:1","urgency","velocity","win:early","comp:brand"],"computed_at":"2026-11-09T06:03:00Z"},"pay":null,"html_url":"https://alion.io/job/apple-ai-software-evaluation-engineer-siri-ai-quality","json_url":"https://alion.io/job/apple-ai-software-evaluation-engineer-siri-ai-quality.json","meta":{"generated_at":"2026-11-09T19:38:30Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":5766,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-10T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":12},"rest":"https://alion.io/mcp/rest/get_company?id=12"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fapple-ai-software-evaluation-engineer-siri-ai-quality"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fapple-ai-software-evaluation-engineer-siri-ai-quality"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fapple-ai-software-evaluation-engineer-siri-ai-quality"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/apple-ai-software-evaluation-engineer-siri-ai-quality\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fapple-ai-software-evaluation-engineer-siri-ai-quality"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3898243}},{"tool":"search_jobs","arguments":{"company":"apple.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3898243},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}