{"id":2208369,"url":"https://alion.io/job/thinking-machines-lab-software-engineer-production-inference-distributed-inference","title":"Software Engineer, Production Inference (Distributed Inference)","company":{"id":14382,"name":"Thinking Machines Lab","domain":"thinkingmachines.ai","url":"https://alion.io/company/thinking-machines-lab","size_band":"501-1000","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Ashby","truth_index":null},"role":"Backend","role_family":"Backend","seniority":"middle","employment_type":"full_time","work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["San Francisco, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":350000,"max":500000,"currency":"USD","period":"year","gross":null,"usd_annual":500000},"salary_estimate":null,"experience_years_min":3,"visa_sponsorship":true,"relocation_package":false,"has_equity":false,"technologies":[{"name":"C++","optional":false},{"name":"KV Cache","optional":false},{"name":"Python","optional":false},{"name":"Quantization","optional":false},{"name":"Rust","optional":false},{"name":"Speculative Decoding","optional":false},{"name":"CUDA","optional":true},{"name":"CUDA Toolkit","optional":true},{"name":"Kubernetes","optional":true},{"name":"LLM","optional":true},{"name":"SGLang","optional":true},{"name":"TensorRT","optional":true},{"name":"TensorRT-LLM","optional":true},{"name":"vLLM","optional":true}],"status":"live","first_seen_at":"2026-10-10T00:10:32Z","employer_posted_date":"2026-10-10","last_verified_at":"2026-10-13T10:54:43Z","board_verified":true,"closed_at":null,"days_open":3,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":3},"description":"About Thinking Machines\nThe mission of Thinking Machines is to build AI that extends human will and judgment. We are training frontier models with Inkling, developing Tinker to let people make models their own, and crafting interfaces that broaden human-AI communication. We believe the future worth building is human, and we're hiring people who want to build it.\nAbout the Role\nWe're hiring a Software Engineer to build and scale the distributed production inference systems that serve Inkling, Inkling-Small, and Tinker in production. You'll own the systems that turn trained models into fast, reliable, cost-efficient services - from request routing and batching to multi-node serving and GPU utilization at scale.\nThis is a systems-heavy, production-first role. You'll work closely with research and infrastructure teams to translate rapidly evolving model architectures into serving systems that meet real-world latency, throughput, and reliability requirements, and you'll be on the front line when production inference systems need to scale, recover, or improve.\nWhat You'll Do\nDesign, build, and operate distributed infrastructure for large-scale model serving, including request routing, load balancing, batching, and multi-node coordination\n\nOptimize inference latency and throughput in production, including work on KV cache management, continuous batching, speculative decoding, and quantization\n\nBuild and maintain high-concurrency serving systems with strong uptime, low tail latency, and deep observability\n\nBenchmark, tune, and extend inference engines to support new model architectures as they move from research into production\n\nPartner with research and infrastructure teams to translate emerging model designs into production-ready serving systems\n\nBuild tooling for tracing, debugging, and resolving issues across the serving stack, from orchestration down to GPU kernels\n\nParticipate in on-call rotation to support production inference systems\n\nSkills & Qualifications\n3+ years of experience building and operating distributed systems in production\n\nStrong systems programming skills in Python, C++, Rust, or similar languages\n\nExperience with production infrastructure at scale: reliability, observability, and performance under real-world load\n\nSolid understanding of networking, concurrency, and distributed systems fundamentals\n\nPreferred Qualifications\nExperience with LLM inference engines such as vLLM, SGLang, or TensorRT-LLM\n\nFamiliarity with GPU programming (CUDA) or low-level performance optimization\n\nExperience with model parallelism, tensor/pipeline parallelism, or other distributed inference techniques\n\nTrack record of operating large-scale production systems with strict latency and uptime requirements\n\nExperience with Kubernetes or similar orchestration systems for GPU workloads\n\nContributions to open-source ML systems or inference infrastructure projects\n\nLogistics\nLocation: This role is based in San Francisco, CA.\n\nCompensation: Depending on background, skills and experience, the expected annual salary range for this position is $350,000 - $500,000 USD (placeholder - verify against current internal bands before publishing).\n\nVisa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.\n\nBenefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.","description_format":"text","description_chars":3516,"description_truncated":false,"requirements":{"experience_years_min":3,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Parental leave","Unlimited PTO"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":true,"industries":["Multimodal AI","Foundation Models","AI Research Labs","AI Development Tools"],"lifecycle":[{"event":"open","at":"2026-10-10T00:39:28Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 30","filings_12m":30,"filings_prev_12m":8,"green_card_filings_12m":0,"median_offered_wage_usd":192500,"route":null,"cap_exempt":false,"checked_at":"2026-10-03T21:08:04+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":7}],"liveness":{"score":90,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.903,"p_room":1,"age_days":3,"expected_fill_days":29,"reasons":["conf:1","velocity","win:early"],"computed_at":"2026-10-13T05:45:00Z"},"pay":{"stated_usd_annual":500000,"is_top_pay":true},"html_url":"https://alion.io/job/thinking-machines-lab-software-engineer-production-inference-distributed-inference","json_url":"https://alion.io/job/thinking-machines-lab-software-engineer-production-inference-distributed-inference.json","meta":{"generated_at":"2026-10-13T12:26:23Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":19338,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-14T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":14382},"rest":"https://alion.io/mcp/rest/get_company?id=14382"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fthinking-machines-lab-software-engineer-production-inference-distributed-inference"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fthinking-machines-lab-software-engineer-production-inference-distributed-inference"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fthinking-machines-lab-software-engineer-production-inference-distributed-inference"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/thinking-machines-lab-software-engineer-production-inference-distributed-inference\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fthinking-machines-lab-software-engineer-production-inference-distributed-inference"}]}