{"id":829155,"url":"https://alion.io/job/river-ai-software-engineer-inference-systems","title":"Software Engineer, Inference Systems","company":{"id":25477,"name":"River AI","domain":"river.ai","url":"https://alion.io/company/river-ai","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"C","score":68,"open_postings":19,"ghost_share":0.526,"stale_share":0,"repost_share":0,"time_to_fill_p50_days":null,"computed_at":"2026-10-28T05:59:10Z"}},"role":"Backend","role_family":"Backend","seniority":null,"employment_type":null,"work_mode":"on_site","remote_scope":null,"remote_scope_basis":null,"remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Palo Alto, United States"],"countries":["US"],"hiring_countries":[],"hiring_countries_total":0,"salary":{"min":200000,"max":420000,"currency":"USD","period":"year","gross":null,"usd_annual":420000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":true,"relocation_package":false,"has_equity":false,"technologies":[{"name":"C++","optional":false},{"name":"KV Cache","optional":false},{"name":"Mixture of Experts","optional":false},{"name":"Python","optional":false},{"name":"Reinforcement Learning","optional":false},{"name":"Rust","optional":false},{"name":"CUDA","optional":true},{"name":"Pre-training","optional":true},{"name":"SGLang","optional":true},{"name":"Speculative Decoding","optional":true},{"name":"TensorRT","optional":true},{"name":"TensorRT-LLM","optional":true},{"name":"vLLM","optional":true}],"status":"live","first_seen_at":"2026-09-12T02:02:31Z","employer_posted_date":"2026-09-20","last_verified_at":"2026-10-28T19:16:30Z","board_verified":true,"closed_at":null,"days_open":46,"trust":{"level":"ok","repost_count":0,"flags":["company_stale"],"days_open":46},"description":"At River AI, our mission is to create personal AI owned and shaped by each individual. To achieve this, we are rewriting the entire stack from scratch: personal hardware for local inference, bespoke training infrastructure, next-generation UIs, and frontier deep learning research.\nWho we are\nWe are scientists, engineers, and builders from the industry's top tech companies and AI labs. We bring a proven track record of scaling consumer systems for hundreds of millions of users and architecting the pre-training infrastructure behind today's frontier models.\nAbout the Role\nWe are looking for exceptional inference systems engineers to build the engines that serve large models through the River API. Your goal is to deliver fast, reliable inference while making efficient use of GPU compute and memory.\nYou will take ownership of the serving runtime, from request scheduling and continuous batching to KV-cache management, distributed model execution, and checkpoint loading. Your work will support both customer-facing inference and the sampling workloads that power reinforcement learning.\nWorking closely with GPU kernel engineers, researchers, and infrastructure engineers, you will bring new models into production and improve their performance across realistic workloads. You will measure success through latency, throughput, reliability, and cost, with careful attention to numerical correctness and model behavior.\nWhat You’ll Do\nOptimize inference for dense and mixture-of-experts models, including fine-tuned models and adapters.\nImprove batching, caching, and admission control to balance throughput, latency, memory use, and fairness.\nAccelerate multi-GPU execution, communication, and model loading while preserving model-version consistency.\nImprove RL sampling throughput while keeping samples and log probabilities tied to the correct model version.\nBuild reliable streaming, cancellation, and recovery under failures and overload.\nProfile bottlenecks and validate improvements through reproducible performance and correctness tests.\nSkills & Qualifications\nMinimum Qualifications:\nBachelor’s degree in Computer Science, Computer Engineering, or equivalent practical experience.\nExperience building inference engines or performance-sensitive distributed services.\nStrong understanding of transformer inference, GPU memory, concurrency, and networking.\nProficiency in Python and C++ or Rust.\nStrong debugging and profiling skills across models, runtimes, and services.\nA collaborative mindset and strong ownership of engineering outcomes.\nPreferred Qualifications: (We encourage you to apply even if you don't meet all of these)\nExperience extending SGLang, vLLM, TensorRT-LLM, or similar frameworks.\nWork on advanced serving techniques, such as speculative decoding or disaggregated prefill and decode.\nFamiliarity with expert parallelism, GPU collectives, and quantized inference.\nExperience with multi-adapter serving, dynamic checkpoint loading, or RL sampling.\nFamiliarity with CUDA graphs, custom kernels, and NVIDIA profiling tools.\nExperience operating model-serving systems under production traffic.\nLogistics & Benefits\nLocation: Palo Alto, California.\nCompensation: $200,000-$420,000 USD annual base pay, depending on experience and skills, plus equity.\nBenefits: Comprehensive health, dental, and vision insurance; unlimited PTO; and relocation assistance as needed.\n Visa Sponsorship: We sponsor visas and support the process for the right candidate.","description_format":"text","description_chars":3482,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":{"level":"bachelor","optional":false},"security_clearance":false,"languages":[]},"benefits":["Equity","Relocation assistance","Unlimited PTO","Vision insurance"],"hiring_locations":[],"hiring_excludes":[],"relocation_offered":true,"industries":["Foundation Models","AI Chips & Accelerators","AI Development Tools"],"lifecycle":[{"event":"open","at":"2026-09-12T15:48:52Z"}],"visa":[{"country":"US","licensed_sponsor":true,"evidence":"H-1B filings in 12 months: 1","filings_12m":1,"filings_prev_12m":0,"green_card_filings_12m":0,"median_offered_wage_usd":497500,"route":null,"cap_exempt":false,"checked_at":"2026-10-15T11:24:14+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":1}],"liveness":{"score":14,"band":"cold","label":"Long shot","p_open":1,"p_active":0.303,"p_room":0.45,"age_days":46,"expected_fill_days":27,"reasons":["conf:0","stale_co","win:tail"],"computed_at":"2026-10-28T05:59:10Z"},"pay":{"stated_usd_annual":420000,"is_top_pay":true},"html_url":"https://alion.io/job/river-ai-software-engineer-inference-systems","json_url":"https://alion.io/job/river-ai-software-engineer-inference-systems.json","meta":{"generated_at":"2026-10-28T21:19:44Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":29447,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-29T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":25477},"rest":"https://alion.io/mcp/rest/get_company?id=25477"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Friver-ai-software-engineer-inference-systems"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Friver-ai-software-engineer-inference-systems"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Friver-ai-software-engineer-inference-systems"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/river-ai-software-engineer-inference-systems\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Friver-ai-software-engineer-inference-systems"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":829155}},{"tool":"search_jobs","arguments":{"company":"river.ai","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":829155},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}