Sr. Software Development Engineer, Trainium Systems

Have you ever wondered what it takes to build the software systems that turn a rack of AI accelerators from inert silicon into a system a customer can run workloads on? Do you want deep ownership of a service that every one of those systems depends on to boot, run, and stay healthy?

Who are we?

Annapurna Labs is the part of Amazon that designs the custom silicon behind the cloud, including Trainium for AI training, Graviton CPUs, and the AWS Nitro system. Nearly every server in AWS runs hardware we designed. We build the chips, the systems, and the software that power some of the largest AI training and inference workloads in the world.

What do we do?

We build the software that provisions, operates, updates, and monitors the servers behind AWS's largest AI training workloads, built on Trainium, Graviton, and the AWS Nitro system. We give the people who run this hardware a complete way to stand it up, watch its health, deliver new software to it, and diagnose it, all through open, standard interfaces. We own that software from design through the running system.

Why it's high-impact?

The hardware is just silicon until our software makes it provisionable, observable, and serviceable. Every system runs through what we build. The service you own is on the path for all of them, and the people who run this hardware depend on it every day.

What's the challenge?

There are many ambiguous and difficult challenges in our fast-moving space. Our services require deep system and software expertise. You need to have the ability to work within a fast moving and startup-like environment in a large company. You will identify solutions, trying ideas, given space to fail and iterate to produce products that your customers love.

What you will do?

You will be part of a team that builds services end to end, from a blank page to an operator running it in production. You work backward from what the customer needs, build on the interfaces the hardware teams expose, and set the requirements those teams meet for you.

Who would succeed in this role?

A deeply technical engineer who stays close to both the customer and the system design. You dive deep to understand how the hardware and the software actually behave, you decide when a small change is enough and when a problem needs a new design, and you own your service without waiting to be told. You raise the bar in code reviews and you lift the engineers around you.

Key job responsibilities

- Own building provisioning services end to end that bring a server from power-on to ready-to-run.

- Build the services or interfaces for monitoring, health, and anomaly detection that surface a server's state and flag a failing component before it takes a workload down.

- Develop the software-delivery service that pushes an update to a running server, verifies it, and recovers the server cleanly when an update fails.

- Develop the diagnostics and troubleshooting tools that let an operator isolate a fault without pulling in the team that built the service.

- Define the standard interfaces an operator's tools drive, and set the behavior the hardware teams deliver against them, including plugins, libraries, tools, or APIs.

- Build the integrations with standard orchestration stacks.

- Develop and build the services that let an operator run a fleet of these servers at scale.

A day in the life

You design and build the provisioning and management surface over standard interfaces like Redfish, and the boot flow operator tools drive. You build the offline software-update service and prove the signed bundle applies and recovers cleanly. You design each system against the interfaces the platform exposes and turn architecture decisions into APIs and tools a customer runs without you in the loop.

About the team

Our team builds the software suite for provisioning, monitoring, software delivery, and the diagnostics of Trainium systems, and we build all of it on open, standard interfaces. We serve the engineers and operators who run large-scale AI training and need to provision, monitor, update, and troubleshoot it on their own terms. We own it end to end.

We're a small team with a large charter. You own real systems on day one. There are no layers between you and the problem. You build it, you ship it, you operate it, you improve it.

Basic qualifications

- 5+ years of non-internship professional software development experience

- 5+ years of programming with at least one software programming language experience

- 5+ years of leading design or architecture (design patterns, reliability and scaling) of new and existing systems experience

- Experience as a mentor, tech lead or leading an engineering team

Preferred qualifications

- 5+ years of full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations experience

- Bachelor's degree in computer science or equivalent

Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.

Los Angeles County applicants: Job duties for this position include: work safely and cooperatively with other employees, supervisors, and staff; adhere to standards of excellence despite stressful conditions; communicate effectively and respectfully with employees, supervisors, and staff to ensure exceptional customer service; and follow all federal, state, and local laws and Company policies. Criminal history may have a direct, adverse, and negative relationship with some of the material job duties of this position. These include the duties and responsibilities listed above, as well as the abilities to adhere to company policies, exercise sound judgment, effectively manage stress and work safely and respectfully with others, exhibit trustworthiness and professionalism, and safeguard business operations and the Company’s reputation. Pursuant to the Los Angeles County Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.

Our inclusive culture empowers Amazonians to deliver the best results for our customers. If you have a disability and need a workplace accommodation or adjustment during the application and hiring process, including support for the interview or onboarding process, please visit https://amazon.jobs/content/en/how-we-hire/accommodations for more information. If the country/region you’re applying in isn’t listed, please contact your Recruiting Partner.

The base salary range for this position is listed below. Your Amazon package will include sign-on payments and restricted stock units (RSUs). Final compensation will be determined based on factors including experience, qualifications, and location. Amazon also offers comprehensive benefits including health insurance (medical, dental, vision, prescription, Basic Life & AD&D insurance and option for Supplemental life plans, EAP, Mental Health Support, Medical Advice Line, Flexible Spending Accounts, Adoption and Surrogacy Reimbursement coverage), 401(k) matching, paid time off, and parental leave. Learn more about our benefits at https://amazon.jobs/en/benefits.

USA, CA, Cupertino - 193,300.00 - 261,500.00 USD annually

USA, CA, San Diego - 168,100.00 - 227,400.00 USD annually

USA, TX, Austin - 168,100.00 - 227,400.00 USD annually

USA, WA, Seattle - 168,100.00 - 227,400.00 USD annually