{"id":2133044,"url":"https://alion.io/job/apple-senior-software-development-engineer-in-test-llm-evaluation-automation-t3e","title":"Senior Software Development Engineer in Test — LLM Evaluation & Automation, T3E","company":{"id":12,"name":"Apple","domain":"apple.com","url":"https://alion.io/company/apple","size_band":"5000+","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Apple Jobs","truth_index":{"grade":"B","score":73,"open_postings":1666,"ghost_share":0.212,"stale_share":0.002,"repost_share":0.615,"time_to_fill_p50_days":153,"computed_at":"2026-11-19T09:08:57Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["San Diego, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":229000,"max_usd":441000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":1332},"experience_years_min":3,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"LLM","optional":false},{"name":"CI/CD","optional":true},{"name":"NLP","optional":true},{"name":"Python","optional":true},{"name":"Rest API","optional":true},{"name":"SQL","optional":true}],"status":"live","first_seen_at":"2026-08-03T21:34:43Z","employer_posted_date":"2026-11-18","last_verified_at":"2026-11-19T08:25:13Z","board_verified":true,"closed_at":null,"days_open":107,"trust":{"level":"ok","repost_count":1,"flags":[],"days_open":107},"description":"The Apple Intelligence Platform Experience Validation team builds the tooling and automation that keeps Apple Intelligence features high-quality before they ship. We are looking for a Senior SDET to lead the design and implementation of automated model evaluation: standing up LLM-as-a-judge in existing and new pipelines, and building the infrastructure that catches model regressions before they reach human evaluation or the live on population.\nThis is a hands-on, senior individual-contributor role. You will own eval automation as a discipline across the team, partnering with modeling, framework, and infrastructure teams to make model quality a first-class, continuously measured signal.\nDescription\nYou will build and maintain model level, component or end-to-end evaluation coverage for the generative features our team validates. Your job is to leverage LLM judge scoring output quality in automation, ensuring reliable, repeatable eval jobs that run that produce actionable signal.\nThe kinds of problems you will work on include:\n* Image / visual generation: validating model output and its associated classification metadata, and detecting quality or behavior regressions across model updates.\n* Natural-language generation: evaluating whether generated artifacts and responses match user intent, moving at-desk LLM judges into a scalable and repeatable automation environment.\n* Correctness beyond string matching: replacing exact-match checks for open-ended or factual responses with an LLM-as-judge stage integrated into the pipeline.\n* Generated insights and summaries: assessing whether model-generated content is sensible and good enough to surface to users.\nYou will decide when a component-level check (an API or CLI that exercises the model against its framework) is sufficient and when a full end-to-end user flow is required, and you will build the tooling for both.\nMinimum Qualifications\nBS in Computer Science, Mathematics, or a related field (or equivalent practical experience)\nThree years of relevant industry experience in test automation, software development, or related areas.\nPreferred Qualifications\nStrong practical knowledge of Python, including data-pipeline fluency (JSON/YAML, REST APIs).\nHands-on experience with LLM-as-a-judge evaluation and rubric design, or a strong demonstrated ability to ramp into it quickly.\nStrong software engineering fundamentals - able to define atomic, composable components and build maintainable pipelines and tooling, not just scripts.\nStrong debugging and triage skills; able to separate genuine regressions from infrastructure or rubric noise.\nStrong knowledge of the software development lifecycle, testing methodologies, and QA processes.\nExcellent written and verbal communication; able to document clearly and describe quality signal to modeling and leadership audiences.\nAbility to lead work across varying priorities and partner multi-functionally with modeling, framework, and infrastructure teams.\nExperience building on-device tooling and device/model eval infrastructure.\nExperience integrating with CI/CD and job orchestration systems, and comfort deploying tooling as reusable libraries.\nFamiliarity with generative model behavior - image generation, NLP, or LLM output evaluation.\nExperience curating and reasoning about large datasets; comfort manually inspecting data (Jupyter or similar) to build intuition and drive next steps.\nAwareness of dataset bias and fairness considerations in evaluation.\nExperience with database/query tooling (e.g., SQL) and dashboards/visualization for reporting quality trends.\nExperience with Xcode is a bonus.","description_format":"text","description_chars":3632,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":true},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Operating Systems","Laptops","Smartwatches & Fitness Trackers","Tablets"],"lifecycle":[{"event":"open","at":"2026-10-09T01:16:33Z"},{"event":"close","at":"2026-10-20T19:26:25Z"},{"event":"reopen","at":"2026-10-21T03:16:03Z"},{"event":"close","at":"2026-10-21T16:58:38Z"},{"event":"reopen","at":"2026-10-21T21:38:17Z"},{"event":"close","at":"2026-10-21T21:49:40Z"},{"event":"reopen","at":"2026-10-22T01:51:31Z"},{"event":"close","at":"2026-11-04T20:23:34Z"},{"event":"reopen","at":"2026-11-05T06:21:11Z"},{"event":"close","at":"2026-11-06T18:41:53Z"},{"event":"reopen","at":"2026-11-06T23:40:21Z"},{"event":"close","at":"2026-11-10T00:24:00Z"},{"event":"reopen","at":"2026-11-10T05:56:30Z"},{"event":"close","at":"2026-11-11T22:29:36Z"},{"event":"reopen","at":"2026-11-12T04:23:18Z"},{"event":"close","at":"2026-11-12T16:09:35Z"},{"event":"reopen","at":"2026-11-13T04:11:20Z"},{"event":"close","at":"2026-11-13T10:47:28Z"},{"event":"reopen","at":"2026-11-13T23:46:56Z"},{"event":"close","at":"2026-11-17T04:54:28Z"},{"event":"reopen","at":"2026-11-18T12:15:40Z"}],"visa":[{"country":"US","licensed_sponsor":true,"adverts_offering_sponsorship_24m":0,"sanctioned_now":false,"evidence":"H-1B filings in 12 months: 6,545 · green card filings: 214","filings_12m":6545,"filings_prev_12m":7060,"green_card_filings_12m":214,"median_offered_wage_usd":176796,"route":null,"cap_exempt":false,"checked_at":"2026-11-19T07:44:56+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)","US Department of Labor: PERM disclosure data (green cards)"],"filings_for_role_12m":2993}],"liveness":{"score":51,"band":"ok","label":"Likely open","p_open":1,"p_active":0.847,"p_room":0.6,"age_days":107,"expected_fill_days":153,"reasons":["conf:0","win:late","crowd:brand"],"computed_at":"2026-11-19T09:08:57Z"},"pay":null,"html_url":"https://alion.io/job/apple-senior-software-development-engineer-in-test-llm-evaluation-automation-t3e","json_url":"https://alion.io/job/apple-senior-software-development-engineer-in-test-llm-evaluation-automation-t3e.json","meta":{"generated_at":"2026-11-19T10:45:46Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":18108,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-20T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":12},"rest":"https://alion.io/mcp/rest/get_company?id=12"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fapple-senior-software-development-engineer-in-test-llm-evaluation-automation-t3e"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"free","price":{"credits":0,"usd":0},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fapple-senior-software-development-engineer-in-test-llm-evaluation-automation-t3e"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fapple-senior-software-development-engineer-in-test-llm-evaluation-automation-t3e"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/apple-senior-software-development-engineer-in-test-llm-evaluation-automation-t3e\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fapple-senior-software-development-engineer-in-test-llm-evaluation-automation-t3e"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":2133044}},{"tool":"search_jobs","arguments":{"company":"apple.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":2133044},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}