{"id":3927345,"url":"https://alion.io/job/internationalrecruit-principal-software-engineer-ai-runtime-distributed-inference-systems","title":"Principal Software Engineer - AI Runtime & Distributed Inference Systems","company":{"id":3827146,"name":"Internationalrecruit","domain":"internationalrecruit.net","url":"https://alion.io/company/internationalrecruit","size_band":null,"is_staffing_agency":true,"employer_type":"agency","is_intermediary":false,"listed_via":null,"ats_vendor":null,"truth_index":null},"role":"Backend","role_family":"Backend","seniority":"lead","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["San Jose, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":null,"salary_estimate":null,"experience_years_min":12,"visa_sponsorship":false,"relocation_package":false,"has_equity":false,"technologies":[{"name":"C++","optional":false},{"name":"CUDA","optional":false},{"name":"InfiniBand","optional":false},{"name":"KV Cache","optional":false},{"name":"LLM","optional":false},{"name":"Mixture of Experts","optional":false},{"name":"NCCL","optional":false},{"name":"NVLink","optional":false},{"name":"PyTorch","optional":false},{"name":"Rust","optional":false},{"name":"SGLang","optional":false},{"name":"TensorRT","optional":false},{"name":"TensorRT-LLM","optional":false},{"name":"TPOT","optional":false},{"name":"TPU","optional":false},{"name":"Triton","optional":false},{"name":"vLLM","optional":false}],"status":"live","first_seen_at":"2026-11-07T02:09:38Z","employer_posted_date":null,"last_verified_at":"2026-11-07T02:09:38Z","board_verified":false,"closed_at":null,"days_open":2,"trust":{"level":"not_scored","repost_count":null,"flags":[],"days_open":2},"description":"About the Role\nWe are building the next generation of AI infrastructure: a large-scale inference platform powered\nby custom AI accelerators capable of serving frontier dense and Mixture-of-Experts (MoE)\nfoundation models at datacenter scale. As a Principal Software Engineer, you will define and lead\nthe architecture of the software stack that transforms hundreds to thousands of accelerator ASICs\ninto a unified inference system. You will drive the design of distributed runtimes, communication\nlibraries, scheduling systems, memory management, and model-serving infrastructure that deliver\nindustry-leading throughput, latency, and efficiency. This is a highly cross-functional leadership\nrole working across silicon architecture, networking, compiler, runtime, infrastructure, and AI\nresearch teams.\nWhat Makes This Role Unique\nThis role sits at the intersection of distributed systems, AI infrastructure, networking, compilers,\nand computer architecture. You will help shape the software foundation for a new class of AI\nsupercomputers and define how future foundation models are served efficiently at scale. The ideal\ncandidate is equally comfortable discussing transformer architectures, network topology, runtime\nscheduling, memory systems, and accelerator microarchitecture, and can translate those insights\ninto production-grade software that powers large-scale AI services.\nResponsibilities\nTechnical Leadership\nDefine the software architecture for large-scale AI inference systems.\nLead the evolution of the distributed runtime stack for dense and MoE foundation models.\nEstablish long-term technical strategy for performance, scalability, reliability, and\nprogrammability.\nInfluence hardware-software co-design decisions across future accelerator generations.\nDistributed Inference Runtime\nArchitect runtime systems supporting:\no Tensor parallelism\no Pipeline parallelism\no Expert parallelism\no Data parallelism\no Multi-rack inference execution\nDesign systems for efficient prefill and decode execution.\nDrive continuous batching and request scheduling strategies.\nOptimize TTFT, TPOT, throughput, utilization, and QoS.• Experience in AI accelerator heterogenous programming models like CUDA/HIP/Triton\nimplementing device management, memory management, Kernel launching, Error handling\nand diagnostics, Kernel optimization techniques with tiling and fused kernels.\nMoE Infrastructure\nLead design of token routing and expert execution systems.\nDefine expert placement and replication strategies.\nDevelop mechanisms for expert load balancing and congestion control.\nOptimize communication and compute overlap across distributed expert clusters.\nCommunication and Fabric Software\nArchitect high-performance collective communication libraries.\nDesign software interfaces for custom accelerator fabrics.\nDevelop scalable implementations of:\no AllReduce\no AllGather\no ReduceScatter\no Broadcast\no Expert dispatch and gather\nDrive latency and bandwidth optimization across accelerator clusters.\nMemory Systems\nDefine architecture for distributed KV-cache management.\nDrive memory placement, eviction, and tiering policies.\nOptimize utilization across local memory, pooled memory, and storage tiers.\nCreate runtime services for dynamic memory allocation and workload balancing.\nPerformance Engineering\nEstablish performance methodology, benchmarks, and KPIs.\nAnalyze bottlenecks across:\no Compute\no Memory bandwidth\no Communication fabric\no Expert routing\no Scheduling layers\nLead root-cause analysis of production performance issues.Infrastructure and Production Systems\nDrive reliability, observability, telemetry, and debugging architecture.\nDefine health monitoring and fleet management capabilities.\nPartner with infrastructure teams to operate inference systems at scale.\nEstablish production readiness standards and operational excellence practices.\nSystem and Infrastructure experience - Experience implementing Virtualization\nsoftware, Rack controllers, Pcie/Memory controllers, device life cycle management\nsoftware, Experience with NCCL/RCCL/MPI collective communications, Scale Up\nnetworking with UALoE/ESUN/NVLink, scale out networking with RoCEv2, Network\ncongestion control techniques like PFC/DCQCN.\nQualifications\nRequired\nrelated field.\nBS, MS, or PhD in Computer Science, Computer Engineering, Electrical Engineering, or\n12+ years of software engineering experience.\n5+ years building distributed systems at scale.\nStrong expertise in C++, Rust, or similar systems programming languages.\nDeep understanding of large-scale AI inference systems.\nExperience with networking and distributed communication systems.\nProven track record leading architecture across multiple engineering teams.\nExperience driving projects from concept through production deployment.\nPreferred\nExperience with large language model serving systems.\nExperience with vLLM, SGLang, TensorRT-LLM, Triton, PyTorch Distributed, NCCL, or\nequivalent technologies.\nExperience designing runtime systems for custom accelerators, GPUs, TPUs, or AI ASICs.\nKnowledge of transformer architectures, sparse models, and MoE systems.\nExperience with RDMA, InfiniBand, RoCE, PCIe, Ethernet fabrics, or custom interconnects.\nExperience in compiler technologies, graph optimization, or kernel scheduling.\nFamiliarity with HBM, memory hierarchy design, and cache management strategies.\nSuccess Metrics\nIn the first 12-24 months, this leader will be expected to:\nDeliver industry-leading TTFT and TPOT performance.\nAchieve high accelerator utilization across production clusters.• Enable scalable execution of frontier dense and MoE models.\nImprove throughput per rack and throughput per watt.\nReduce communication overhead across distributed inference deployments.\nEstablish a robust and production-ready AI runtime platform.\nInfluence future hardware generations through software-driven architectural insights.","description_format":"text","description_chars":5885,"description_truncated":false,"requirements":{"experience_years_min":12,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":[],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":false,"industries":[],"lifecycle":[{"event":"open","at":"2026-11-07T20:40:15Z"}],"visa":[],"liveness":{"score":52,"band":"ok","label":"Likely open","p_open":1,"p_active":0.516,"p_room":1,"age_days":2,"expected_fill_days":28,"reasons":["seen:2","agency","win:early"],"computed_at":"2026-11-09T06:03:00Z"},"pay":null,"html_url":"https://alion.io/job/internationalrecruit-principal-software-engineer-ai-runtime-distributed-inference-systems","json_url":"https://alion.io/job/internationalrecruit-principal-software-engineer-ai-runtime-distributed-inference-systems.json","meta":{"generated_at":"2026-11-09T15:25:47Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":21848,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-10T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":3827146},"rest":"https://alion.io/mcp/rest/get_company?id=3827146"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Finternationalrecruit-principal-software-engineer-ai-runtime-distributed-inference-systems"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Finternationalrecruit-principal-software-engineer-ai-runtime-distributed-inference-systems"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Finternationalrecruit-principal-software-engineer-ai-runtime-distributed-inference-systems"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/internationalrecruit-principal-software-engineer-ai-runtime-distributed-inference-systems\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Finternationalrecruit-principal-software-engineer-ai-runtime-distributed-inference-systems"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":3927345}},{"tool":"search_jobs","arguments":{"company":"internationalrecruit.net","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":3927345},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}