Principal Software Engineer - AI Runtime & Distributed Inference Systems

About the Role

We are building the next generation of AI infrastructure: a large-scale inference platform powered

by custom AI accelerators capable of serving frontier dense and Mixture-of-Experts (MoE)

foundation models at datacenter scale. As a Principal Software Engineer, you will define and lead

the architecture of the software stack that transforms hundreds to thousands of accelerator ASICs

into a unified inference system. You will drive the design of distributed runtimes, communication

libraries, scheduling systems, memory management, and model-serving infrastructure that deliver

industry-leading throughput, latency, and efficiency. This is a highly cross-functional leadership

role working across silicon architecture, networking, compiler, runtime, infrastructure, and AI

research teams.

What Makes This Role Unique

This role sits at the intersection of distributed systems, AI infrastructure, networking, compilers,

and computer architecture. You will help shape the software foundation for a new class of AI

supercomputers and define how future foundation models are served efficiently at scale. The ideal

candidate is equally comfortable discussing transformer architectures, network topology, runtime

scheduling, memory systems, and accelerator microarchitecture, and can translate those insights

into production-grade software that powers large-scale AI services.

Responsibilities

Technical Leadership

• Define the software architecture for large-scale AI inference systems.
• Lead the evolution of the distributed runtime stack for dense and MoE foundation models.
• Establish long-term technical strategy for performance, scalability, reliability, and

programmability.

• Influence hardware-software co-design decisions across future accelerator generations.

Distributed Inference Runtime

• Architect runtime systems supporting:

o Tensor parallelism

o Pipeline parallelism

o Expert parallelism

o Data parallelism

o Multi-rack inference execution

• Design systems for efficient prefill and decode execution.
• Drive continuous batching and request scheduling strategies.
• Optimize TTFT, TPOT, throughput, utilization, and QoS.• Experience in AI accelerator heterogenous programming models like CUDA/HIP/Triton

implementing device management, memory management, Kernel launching, Error handling

and diagnostics, Kernel optimization techniques with tiling and fused kernels.

MoE Infrastructure

• Lead design of token routing and expert execution systems.
• Define expert placement and replication strategies.
• Develop mechanisms for expert load balancing and congestion control.
• Optimize communication and compute overlap across distributed expert clusters.

Communication and Fabric Software

• Architect high-performance collective communication libraries.
• Design software interfaces for custom accelerator fabrics.
• Develop scalable implementations of:

o AllReduce

o AllGather

o ReduceScatter

o Broadcast

o Expert dispatch and gather

• Drive latency and bandwidth optimization across accelerator clusters.

Memory Systems

• Define architecture for distributed KV-cache management.
• Drive memory placement, eviction, and tiering policies.
• Optimize utilization across local memory, pooled memory, and storage tiers.
• Create runtime services for dynamic memory allocation and workload balancing.

Performance Engineering

• Establish performance methodology, benchmarks, and KPIs.
• Analyze bottlenecks across:

o Compute

o Memory bandwidth

o Communication fabric

o Expert routing

o Scheduling layers

• Lead root-cause analysis of production performance issues.Infrastructure and Production Systems
• Drive reliability, observability, telemetry, and debugging architecture.
• Define health monitoring and fleet management capabilities.
• Partner with infrastructure teams to operate inference systems at scale.
• Establish production readiness standards and operational excellence practices.
• System and Infrastructure experience - Experience implementing Virtualization

software, Rack controllers, Pcie/Memory controllers, device life cycle management

software, Experience with NCCL/RCCL/MPI collective communications, Scale Up

networking with UALoE/ESUN/NVLink, scale out networking with RoCEv2, Network

congestion control techniques like PFC/DCQCN.

Qualifications

Required

related field.

• BS, MS, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or
• 12+ years of software engineering experience.
• 5+ years building distributed systems at scale.
• Strong expertise in C++, Rust, or similar systems programming languages.
• Deep understanding of large-scale AI inference systems.
• Experience with networking and distributed communication systems.
• Proven track record leading architecture across multiple engineering teams.
• Experience driving projects from concept through production deployment.

Preferred

• Experience with large language model serving systems.
• Experience with vLLM, SGLang, TensorRT-LLM, Triton, PyTorch Distributed, NCCL, or

equivalent technologies.

• Experience designing runtime systems for custom accelerators, GPUs, TPUs, or AI ASICs.
• Knowledge of transformer architectures, sparse models, and MoE systems.
• Experience with RDMA, InfiniBand, RoCE, PCIe, Ethernet fabrics, or custom interconnects.
• Experience in compiler technologies, graph optimization, or kernel scheduling.
• Familiarity with HBM, memory hierarchy design, and cache management strategies.

Success Metrics

In the first 12-24 months, this leader will be expected to:

• Deliver industry-leading TTFT and TPOT performance.
• Achieve high accelerator utilization across production clusters.• Enable scalable execution of frontier dense and MoE models.
• Improve throughput per rack and throughput per watt.
• Reduce communication overhead across distributed inference deployments.
• Establish a robust and production-ready AI runtime platform.
• Influence future hardware generations through software-driven architectural insights.