{"id":2251405,"url":"https://alion.io/job/rivian-staff-software-engineer-ml-infra-autonomy-2","title":"Staff Software Engineer, ML Infra, Autonomy","company":{"id":37817,"name":"Rivian","domain":"rivian.com","url":"https://alion.io/company/rivian","size_band":"1001-5000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Jibe","truth_index":{"grade":"A","score":88,"open_postings":98,"ghost_share":0,"stale_share":0.5,"repost_share":0,"time_to_fill_p50_days":54,"computed_at":"2026-10-21T05:46:15Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"staff","employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Palo Alto, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":206500,"max":258100,"currency":"USD","period":"year","gross":null,"usd_annual":258100},"salary_estimate":null,"experience_years_min":5,"visa_sponsorship":false,"relocation_package":false,"has_equity":true,"technologies":[{"name":"AWS","optional":false},{"name":"AWS Trainium","optional":false},{"name":"C++","optional":false},{"name":"Go","optional":false},{"name":"JAX","optional":false},{"name":"Kubernetes","optional":false},{"name":"Python","optional":false},{"name":"PyTorch","optional":false},{"name":"PyTorch C++","optional":false},{"name":"Ray","optional":false},{"name":"Rust","optional":false},{"name":"SLURM","optional":false},{"name":"TPU","optional":false},{"name":"LLM","optional":true},{"name":"MLFlow","optional":true},{"name":"NCCL","optional":true}],"status":"live","first_seen_at":"2026-09-22T17:04:00Z","employer_posted_date":"2026-10-11","last_verified_at":"2026-10-21T04:16:00Z","board_verified":true,"closed_at":null,"days_open":28,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":28},"description":"About RivianRivian is on a mission to keep the world adventurous forever. This goes for the emissions-free Electric Adventure Vehicles we build, and the curious, courageous souls we seek to attract.\nAs a company, we constantly challenge what’s possible, never simply accepting what has always been done. We reframe old problems, seek new solutions and operate comfortably in areas that are unknown. Our backgrounds are diverse, but our team shares a love of the outdoors and a desire to protect it for future generations.\nRole Summary\nRivian Autonomy is building an ML Infrastructure team to give hundreds of ML engineers a training platform they can trust at fleet scale. We are seeking a Staff Software Engineer to help design and build the platform that trains and evaluates our autonomous driving models: the control plane that schedules jobs across accelerator clusters, the storage and I/O layer that feeds them, the observability that tells us where every GPU-hour goes, and the training framework layer that lets ML engineers write model code once and run it on any silicon we operate.\nAutonomy at Rivian runs a training fleet of thousands of GPUs on Kubernetes over petabyte-scale sensor data, with additional accelerator types arriving in the coming months. The fleet is fully allocated, so the next step change comes from goodput - how much useful training each GPU-hour delivers. Raising it, through scheduling that keeps large gang jobs fed, a storage and I/O layer that keeps up with the accelerators, and per-workload baselines that make every optimization measurable, is the heart of this role.\nThe control plane and training fleet already exist and serve every ML engineer in Rivian Autonomy; the scheduling, storage/I/O, observability and framework layers on top of them are largely still to be built. This is a role for someone who wants to set technical direction rather than maintain it.\nThe platform spans four areas - job scheduling and multi-tenant cluster management, training data storage and I/O, observability and workload optimization, and the training framework layer. You will lead one or two of these areas end-to-end and contribute across the rest; we do not expect one person to be an expert in all four. You will partner closely with the model training teams who are the platform’s customers, with the Cloud Infrastructure team that owns the clusters underneath, and with the Data Infrastructure team that produces the datasets the platform serves.\nAs an early member of the team, you will help define its technical direction, operating model and future hiring.\nResponsibilities\nJob scheduling and multi-tenant cluster management\nEvolve our control plane into the single entry point for every accelerator cluster we operate, across cloud providers and silicon types: a user asks for N accelerators of a given type, not for a specific cluster.\nDesign scheduling that maximizes fleet goodput rather than queue order: gang scheduling for jobs of hundreds of nodes, multi-factor priority and fair sharing across teams, quota borrowing with enforceable reclaim, and execution-time-aware backfill.\nTreat availability as an engineered system: continuous node health checking with automatic cordon, drain and replace, and automatic classification of every failed job (user, out-of-memory, communication, hardware, platform).\nTraining data storage and I/O\nDesign the storage tiering between object storage, shared or node-local caches and memory, and the request patterns that keep thousands of concurrent readers from overwhelming the object store.\nKill the small-file problem for good: shard formats and indexing that serve both fleet-scale shuffled random access and sequential scans, source data stored once and referenced everywhere, and the evaluation and adoption of a training-native storage format.\nMake “GPU-hours lost to input wait” a first-class metric and drive it down on real jobs.\nObservability and workload optimization\nBuild the monitoring and profiling toolchain at every layer a job touches - storage, node, GPU and interconnect, scheduler, and per-job metrics surfaced to the job owner - so that platform and users see the same picture.\nEstablish a workload taxonomy and per-type execution baselines (step time, utilization, communication fraction, input wait, checkpoint cost) that make regressions detectable and every optimization quantifiable.\nLay the groundwork for AI-assisted triage and optimization: metrics, logs and scheduler state accessible enough that an agent can be the first responder for failed jobs and propose improvements measured against the baselines.\nTraining framework and stack currency\nHelp build a thin, opinionated training framework layer over open-source distributed training libraries: one API for users, per-accelerator backends underneath, with golden images, validated launch recipes and a model-zoo CI that gates every release.\nRun coordinated upgrade programs across the ML stack (distributed compute framework, Kubernetes operators, queueing, experiment tracking) on a cluster that is never idle.\nTechnical leadership\nDefine the platform roadmap with the team lead - build-versus-buy decisions, boundaries with the cluster, data and model teams, and prioritization by measured impact on goodput and cost.\nWork directly with ML engineers to find where the platform slows them down or fails, and turn that into improvements to the scheduler, the data path, the tooling and the documentation.\nLead architecture across organizational boundaries, communicate recommendations to engineering leadership, and mentor the engineers building and operating the platform.\nQualifications\nRequired\n5+ years of software engineering experience, or equivalent demonstrated impact, with substantial distributed-systems work in production.\nStaff-level technical leadership: you identify the problems worth solving, shape strategy across teams, make pragmatic trade-offs, and drive ambiguous initiatives from evidence to production.\nHands-on experience building or operating large-scale compute or ML infrastructure - and the failure modes that only show up at scale: hung collectives, retry storms, stranded capacity, silent input-bound jobs.\nDeep expertise in at least one of the following areas, with working familiarity with the others:\nCluster scheduling and multi-tenancy - Kubernetes-based scheduling and resource management (Kueue, Volcano, Slurm, YARN or equivalent), including quota, fairness and preemption design.\nTraining data storage and I/O - object-store request behavior, caching tiers, shard formats, shuffle-versus-locality trade-offs, and measuring whether a job is input-bound.\nDistributed training frameworks and accelerators - PyTorch distributed, Ray Train, JAX or equivalent, on GPU, TPU or Trainium: how process groups, collectives, sharding and checkpointing actually behave.\nObservability and performance engineering - profiling and monitoring distributed workloads from the kernel and GPU up to the scheduler, and turning measurements into optimizations.\nStrong programming skills in Python and experience with at least one additional relevant language such as Go, Rust or C++.\nExperience operating on a major cloud provider (AWS preferred) and on Kubernetes.\nStrong communication and developer empathy, with a track record of building platforms that engineers adopt and trust; self-directed in ambiguous problem spaces.\nBonus Points\nDeep understanding of operating systems - memory management, I/O and network stack, scheduling, kernel-level debugging - as applied to debugging and optimizing distributed workloads.\nExperience with large-model training and cross-GPU communication: collective communication, tensor/pipeline parallelism, NCCL performance at scale.\nExperience with Ray and KubeRay internals, Kueue or Kubernetes scheduler extensions, or maintaining a patch set on top of an upstream project.\nExperience with GPU profiling and performance tooling (torch profiler, Nsight Systems and Compute, DCGM, NCCL telemetry) or their counterparts on other accelerators.\nFamiliarity with training-native or columnar data formats (Lance, WebDataset, Parquet row groups) and with GPU-side video decode in the input path.\nExperience with experiment tracking and model registry platforms (MLflow or equivalent) at scale.\nBackground in applying LLM agents to infrastructure operations, triage or performance tuning.\nPay Disclosure\nThe salary range for this role is $206,500-$258,100 for San Francisco Bay Area based applicants. This is the lowest to highest salary we in good faith believe we would pay for this role at the time of this posting. An employee’s position within the salary range will be based on several factors including, but not limited to, specific competencies, relevant education, qualifications, certifications, experience, skills, geographic location, shift, and organizational needs.\nWe offer a comprehensive package of benefits for full-time and part-time employees, their spouse or domestic partner, and children up to age 26, including but not limited to paid vacation, paid sick leave, and a competitive portfolio of insurance benefits including life, medical, dental, vision, short-term disability insurance, and long-term disability insurance to eligible employees. You may also have the opportunity to participate in Rivian’s 401(k) Plan and Employee Stock Purchase Program if you meet certain eligibility requirements. Full-time employee coverage is effective on their first day of employment. Part-time employee coverage is effective the first of the month following 90 days of employment. More information about benefits is available at rivianbenefits.com.\nEqual Opportunity\nRivian is an equal opportunity employer and complies with all applicable federal, state, and local fair employment practices laws. All qualified applicants will receive consideration for employment without regard to race, color, religion, national origin, ancestry, sex, sexual orientation, gender, gender expression, gender identity, genetic information or characteristics, physical or mental disability, marital/domestic partner status, age, military/veteran status, medical condition, or any other characteristic protected by law.\nRivian is committed to ensuring that our hiring process is accessible for persons with disabilities. If you have a disability or limitation, such as those covered by the Americans with Disabilities Act, that requires accommodations to assist you in the search and application process, please email us at .\nCandidate Data Privacy and Technology\nRivian may collect, use and disclose your personal information or personal data (within the meaning of the applicable data protection laws) when you apply for employment and/or participate in our recruitment processes (“Candidate Personal Data”). This data includes contact, demographic, communications, educational, professional, employment, social media/website, network/device, recruiting system usage/interaction, security and preference information. Rivian may use your Candidate Personal Data for the purposes of (i) tracking interactions with our recruiting system; (ii) carrying out, analyzing and improving our application and recruitment process, including assessing you and your application and conducting employment, background and reference checks; (iii) establishing an employment relationship or entering into an employment contract with you; (iv) complying with our legal, regulatory and corporate governance obligations; (v) recordkeeping; (vi) ensuring network and information security and preventing fraud; and (vii) as otherwise required or permitted by applicable law.\nRivian may share your Candidate Personal Data with (i) internal personnel who have a need to know such information in order to perform their duties, including individuals on our People Team, Finance, Legal, and the team(s) with the position(s) for which you are applying; (ii) Rivian affiliates; and (...","description_format":"text","description_chars":13551,"description_truncated":true,"requirements":{"experience_years_min":5,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":["Automotive Manufacturing","Automotive Components"],"lifecycle":[{"event":"open","at":"2026-10-11T02:24:43Z"}],"visa":[],"liveness":{"score":56,"band":"ok","label":"Likely open","p_open":1,"p_active":0.617,"p_room":0.9,"age_days":28,"expected_fill_days":54,"reasons":["conf:1","stale_co","velocity","win:mid","comp:brand"],"computed_at":"2026-10-21T05:46:15Z"},"pay":{"stated_usd_annual":258100,"is_top_pay":false},"html_url":"https://alion.io/job/rivian-staff-software-engineer-ml-infra-autonomy-2","json_url":"https://alion.io/job/rivian-staff-software-engineer-ml-infra-autonomy-2.json","meta":{"generated_at":"2026-10-21T06:53:40Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":9743,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-22T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":37817},"rest":"https://alion.io/mcp/rest/get_company?id=37817"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Frivian-staff-software-engineer-ml-infra-autonomy-2"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Frivian-staff-software-engineer-ml-infra-autonomy-2"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Frivian-staff-software-engineer-ml-infra-autonomy-2"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/rivian-staff-software-engineer-ml-infra-autonomy-2\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Frivian-staff-software-engineer-ml-infra-autonomy-2"}]}