{"id":2628308,"url":"https://alion.io/job/xai-site-reliability-engineer-2","title":"Site Reliability Engineer","company":{"id":28,"name":"xAI","domain":"x.ai","url":"https://alion.io/company/xai","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"A","score":88,"open_postings":100,"ghost_share":0,"stale_share":0.47,"repost_share":0,"time_to_fill_p50_days":59,"computed_at":"2026-10-16T05:45:00Z"}},"role":"DevOps","role_family":"DevOps","seniority":"senior","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Memphis, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":{"min_usd":198000,"max_usd":384000,"period":"year","method":"role_seniority_country_remote_cell","sample_n":1602},"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"C++","optional":false},{"name":"Grafana","optional":false},{"name":"InfiniBand","optional":false},{"name":"Java","optional":false},{"name":"NVLink","optional":false},{"name":"PagerDuty","optional":false},{"name":"Prometheus","optional":false},{"name":"Python","optional":false},{"name":"Rust","optional":false},{"name":"Kubernetes","optional":true},{"name":"NCCL","optional":true},{"name":"SLURM","optional":true}],"status":"live","first_seen_at":"2026-10-15T14:05:31Z","employer_posted_date":"2026-10-15","last_verified_at":"2026-10-16T11:20:56Z","board_verified":true,"closed_at":null,"days_open":1,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":1},"description":"SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today, SpaceX is actively developing the technologies to make this possible, with the ultimate goal of making life multiplanetary and expanding consciousness to the stars.\nSpaceXSI is building Super Intelligence to help humanity understand the universe. Our teams are building the full Super Intelligence stack: from our Colossus supercomputers, to Grok models, to our suite of applications, including Grok Bot, Grok Build, Grok Imagine, Grok Voice, and X. We're creating a powerful flywheel where more compute trains better models, better models power more useful products, and what we learn from people using those products shapes what we build next.\nABOUT THE ROLE:\nAs a Site Reliability Engineer focused on campus reliability, you will design what the campus watches and trusts, technically command cross-discipline SEVs, and build the guardrails that make the next incident smaller. You are the connective tissue across compute, network, storage, power, and cooling on one of the largest AI training campuses in the world. This role demands calm incident leadership, fleet-scale observability judgment, and the ability to drive reliability work across software and facility boundaries. We are looking for candidates from power plants, nuclear power plants, data centers, or people who currently work or have worked in facilities like SpaceXSI with power, compute, cooling, and related plant systems.\nRESPONSIBILITIES:\nOwn monitoring architecture and signal quality across GPU and host telemetry (DCGM, Prometheus/Grafana) and facility telemetry (BMS, EPMS, DCIM, SCADA, CDU / direct liquid cooling): what we alert on, suppress, and trust. Consume NOC noise-disposition feedback to drive suppression and redesign. Treat alert noise as a design failure, not an operator failure.\nProvide SEV command support: technical incident leadership, bridge coordination with the NOC, paging and escalation (e.g., PagerDuty), and timeline and severity hygiene.\nRun blameless postmortems and drive corrective actions to closed, not filed.\nGate high-risk change: fleet-scale firmware/BIOS/BMC rollouts (Redfish, canaries, rollback) and SOP/MOP/EOP-governed work on power and cooling systems through a disciplined change management process.\nLead cross-functional reliability projects spanning compute, network (InfiniBand/RoCE), storage, and facility signal boundaries, including grey failures that cross those boundaries (e.g., NVLink flapping, thermal throttling, power sequencing).\nPartner with Hardware Failure Analysis on systemic failure classes: turn fleet data (MTBF/AFR, repeat-RMA trends) into monitoring, stop-rules, and vendor RCA inputs.\nBuild and maintain playbooks, run game days, and keep cross-discipline dependency maps current. Own runbook quality jointly with the NOC (SRE designs; NOC operates and corrects).\nDefine error budgets and availability objectives at campus and service boundaries as adopted by the business.\nParticipate in on-call rotations and incident response for SEV-class events in the Memphis / Southaven data center campus.\nPROFILE SIGNALS:\nExperience in power plants, nuclear power plants, data centers, or facilities like SpaceXSI - current or prior - spanning power, compute, cooling, or related plant systems.\nProven large-scale incident command experience and calm technical leadership on a bridge. Nuclear control-room, shift supervisor / Senior Reactor Operator (SRO), Shift Technical Advisor, or Navy nuclear experience counts as incident command.\nDemonstrated monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality. Treats alert noise as a design failure, not an operator failure.\nCross-team facilitation and systems engineering depth across software and facility boundaries.\nSUCCESS MEASURED BY:\nMTTD / MTTR trend for SEV-class events\n% of SEVs with a blameless postmortem and closed action items\nAlert actionable ratio\nRecurrence rate of incident classes\nMonitoring coverage of critical dependencies across compute, network, storage, power, and cooling\nGPU fleet availability lost to infrastructure events\nBASIC QUALIFICATIONS:\nBachelor's degree in Systems Engineering, Computer Science, Electrical Engineering, or a related field (or equivalent experience).\n5+ years of experience in site reliability, systems engineering, plant operations, or large-scale production operations, preferably in power plants, nuclear power plants, high-performance computing, or data center environments.\nProven large-scale incident command experience and calm technical leadership on a bridge, whether from an SRE/NOC bridge or a plant control room.\nDemonstrated monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality.\nHands-on experience with at least one of: Prometheus/Grafana, NVIDIA DCGM, or industrial telemetry and controls (SCADA, BMS/BAS, EPMS, DCIM, PLCs, Modbus/BACnet).\nExperience working across at least two of: compute, network, storage, power, and cooling / facilities telemetry.\nExperience writing and operating playbooks, runbooks, or SOP/MOP/EOPs with a 24/7 operations, control room, or NOC partner.\nProficiency in scripting (Python, Bash) for automation and analysis, plus general experience in at least one systems language (C, C++, Java, Go, Rust, or similar). Not required to be expert in all of them.\nExcellent problem-solving skills with a data-driven approach to reliability engineering.\nAbility to work collaboratively with cross-functional teams, including NOC, data center operations, hardware failure analysis, and infrastructure engineering.\nPREFERRED SKILLS AND EXPERIENCE:\nExperience in power plants, nuclear power plants, or other high-consequence industrial control environments (ICS/OT).\nCurrent or prior work in data centers or facilities like SpaceXSI spanning power, compute, cooling, and related plant systems, including CDU / direct liquid cooling.\nExperience in AI/ML infrastructure or supercomputing environments: GB200/NVL72 or H100/H200-class GPU fleets, NCCL, InfiniBand/RoCE, Slurm or Kubernetes.\nFleet firmware and BMC management (Redfish, IPMI), including firmware release validation and staged rollouts.\nReliability engineering methods: MTBF/AFR analysis, reliability-centered maintenance, grey-failure diagnosis, and vendor RCA/RMA partnership.\nExperience with CMMS (e.g., Maximo) and commissioning / integrated systems testing (IST) and operational turnover.\nHands-on definition and use of SLOs, SLIs, and error budgets at service or campus boundaries.\nExperience running game days, dependency mapping, and closed-loop corrective action programs.\nFamiliarity with data center hardware and plant signals (servers, GPUs, networking, power, cooling) in addition to software telemetry.\nPrior work in a fast-paced startup or tech company like SpaceXSI.\nSpaceXSI is an equal opportunity employer. For details on data processing, view our Recruitment Privacy Notice.","description_format":"text","description_chars":7110,"description_truncated":false,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["LLM & Generative AI","Foundation Models","AI Research Labs"],"lifecycle":[{"event":"open","at":"2026-10-15T15:36:58Z"}],"visa":[],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":0,"expected_fill_days":59,"reasons":["conf:3","velocity","win:early","comp:brand"],"computed_at":"2026-10-16T05:45:00Z"},"pay":null,"html_url":"https://alion.io/job/xai-site-reliability-engineer-2","json_url":"https://alion.io/job/xai-site-reliability-engineer-2.json","meta":{"generated_at":"2026-10-16T15:35:13Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":15457,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-17T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":28},"rest":"https://alion.io/mcp/rest/get_company?id=28"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fxai-site-reliability-engineer-2"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fxai-site-reliability-engineer-2"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fxai-site-reliability-engineer-2"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/xai-site-reliability-engineer-2\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fxai-site-reliability-engineer-2"}]}