{"id":1039886,"url":"https://alion.io/job/salad-technologies-staff-software-engineer-ai-inference-gateway","title":"Staff Software Engineer, AI Inference Gateway","company":{"id":63295,"name":"Salad Technologies","domain":"salad.com","url":"https://alion.io/company/salad-technologies","size_band":null,"is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Rippling","truth_index":null},"role":"Backend","role_family":"Backend","seniority":"staff","employment_type":"full_time","work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":[],"countries":[],"hiring_countries":["US"],"hiring_countries_total":1,"salary":{"min":180000,"max":220000,"currency":"USD","period":"year","gross":null,"usd_annual":220000},"salary_estimate":null,"experience_years_min":null,"visa_sponsorship":false,"relocation_package":false,"has_equity":true,"technologies":[{"name":"Embeddings","optional":false},{"name":"KV Cache","optional":false},{"name":"llama.cpp","optional":false},{"name":"LLM","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"Quantization","optional":false},{"name":"Rust","optional":false},{"name":"TensorRT-LLM","optional":false},{"name":"TGI","optional":false},{"name":"vLLM","optional":false},{"name":"AWQ","optional":true},{"name":"CUDA","optional":true},{"name":"GGUF","optional":true},{"name":"GPTQ","optional":true},{"name":"ROCm","optional":true},{"name":"TensorRT","optional":true}],"status":"live","first_seen_at":"2026-09-18T17:20:19Z","employer_posted_date":"2026-09-18","last_verified_at":"2026-11-09T19:33:18Z","board_verified":true,"closed_at":null,"days_open":52,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":52},"description":"Our mission is to democratize cloud computing\nby utilizing latent consumer resources to power a sustainable, affordable, environmentally friendly cloud for everyone.\nsalad.com/about\nStaff Software Engineer, AI Inference Gateway\nSalad Technologies · Remote · Full-time\nThe role\nSaladCloud runs inference on thousands of geodistributed workstation and consumer GPUs - NVIDIA and AMD - that nobody else can use. Sitting in front of that fleet is a unique AI Gateway written in Rust, built on Pingora and WireGuard, that serves real-time and batch inference and embeddings to customers as a single, reliable API. You'll own it.\nThis is an infrastructure role with real autonomy. You'll design and ship the systems that decide how requests are fulfilled, when to add or drop capacity, and how to bill for every token - and you'll be the person who knows whether a new model, quantization, or GPU is actually worth running.\nWhat you'll do\nOwn the AI Gateway system end to end: request routing, streaming, batch/async job handling, and horizontal scaling across multiple gateway servers\nBuild and tune the fleet-efficiency algorithms - autoscaling nodes, scoring performance, and evicting underperformers so the network converges on peak throughput per dollar\nOperate and scale the underlying inference nodes running vLLM, llama.cpp, and similar servers\nDesign and maintain OpenTelemetry-based observability across the gateway and the fleet; pay-per-token and subscription billing already rides on this telemetry, so you'll keep that integration accurate\nSpend roughly 25% of your time benchmarking new models, quantizations, and GPU hardware, and work with the product and marketing teams to make the call on what goes into production\nCollaborate on cross-system design and integrations with the rest of Salad engineering; explain trade-offs clearly to both technical and non-technical teammates\nThe team\nYou'll join a small team - five engineers plus the CTO, who still writes code, and two deeply technical product managers who ship their own fixes rather than queue them for you. You'll report directly to the CTO. Everyone on the engineering team has 10+ years of experience and has built systems that already run at massive scale - with minimal incidents in a typical year, because we ship things that are production-worthy the first time.\nYou'll be the first hire with inference-gateway expertise and you'll be asked to own the whole surface. You won't do it alone: your teammates are eager to learn the domain, will pair on design and integrations, and share after-hours support. The bar is high; so is the support.\nWe've listed a lot below. If you're strong in two of the first three - Rust, LLM inference servers, distributed systems - apply, and expect to learn the rest fast with people who'll help.\nWhat we're looking for\nStrong production Rust experience, ideally on high-throughput networked services\nHands-on experience running LLM inference servers (vLLM, llama.cpp, TGI, TensorRT-LLM, or similar) - you know what a KV cache is and why it fills up\nDistributed systems fundamentals: load balancing, backpressure, failure handling, unreliable nodes\nComfort operating what you build: metrics, tracing, on-call instincts\nWilling to be on-call for a system you helped make quiet - we treat after-hours pages as a design bug to fix, not a lifestyle\nClear written and verbal communication; you can run a design without hand-holding and bring others along\nNice to have\nPingora, Tokio, or proxy/gateway internals\nExperience with heterogeneous or consumer-grade GPU fleets, CUDA, or ROCm\nQuantization formats (GGUF, AWQ, GPTQ, FP8) and their performance characteristics\nWhy Salad\nYou'll work on a hard, unusual problem - coaxing datacenter-grade reliability out of hardware in people's homes - with the freedom to make major decisions and see them hit production quickly.\nWe're scaling rapidly: demand for inference on SaladCloud currently exceeds the capacity we can bring online, and this role exists to keep up with it. There is no shortage of work and no prospect of it ramping down.\nThe platform handles workload security and isolation, so you can focus on performance. We don't log or train on our customers' prompts or data.\nBenefits\nUnlimited PTO\n75% of health insurance premiums covered for you and your dependents\nDental and vision coverage\n401(k) plan\nStock options\nCompany-provided computer\n$500 WFH budget\nFully remote, with flexible hours\nCompensation\n$180,000-$220,000/year","description_format":"text","description_chars":4488,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Health insurance","Stock options","Unlimited PTO"],"hiring_locations":[{"name":"United States","iso":"US","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Artificial Intelligence","Cybersecurity","LLM & Generative AI","Cloud Platforms (IaaS & PaaS)"],"lifecycle":[{"event":"open","at":"2026-09-18T18:28:20Z"}],"visa":[{"country":"US","licensed_sponsor":true,"adverts_offering_sponsorship_24m":0,"sanctioned_now":false,"evidence":"H-1B filings in 12 months: 1","filings_12m":1,"filings_prev_12m":0,"green_card_filings_12m":0,"median_offered_wage_usd":192500,"route":null,"cap_exempt":false,"checked_at":"2026-11-09T07:32:21+00:00","sources":["US Department of Labor: LCA disclosure data (H-1B, H-1B1, E-3)"],"filings_for_role_12m":1}],"liveness":{"score":28,"band":"fade","label":"Fading","p_open":1,"p_active":0.632,"p_room":0.45,"age_days":51,"expected_fill_days":28,"reasons":["conf:1","win:tail"],"computed_at":"2026-11-09T06:03:00Z"},"pay":{"stated_usd_annual":220000,"is_top_pay":false},"html_url":"https://alion.io/job/salad-technologies-staff-software-engineer-ai-inference-gateway","json_url":"https://alion.io/job/salad-technologies-staff-software-engineer-ai-inference-gateway.json","meta":{"generated_at":"2026-11-09T20:32:18Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":7799,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-11-10T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.1,"usd":0.01,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":63295},"rest":"https://alion.io/mcp/rest/get_company?id=63295"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fsalad-technologies-staff-software-engineer-ai-inference-gateway"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":2,"usd":0.2},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fsalad-technologies-staff-software-engineer-ai-inference-gateway"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.1,"usd":0.01},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fsalad-technologies-staff-software-engineer-ai-inference-gateway"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/salad-technologies-staff-software-engineer-ai-inference-gateway\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fsalad-technologies-staff-software-engineer-ai-inference-gateway"}],"agent":{"mcp":"https://alion.io/mcp/jobs","mcp_open":"https://alion.io/mcp/open","next":[{"tool":"get_job","arguments":{"id":1039886}},{"tool":"search_jobs","arguments":{"company":"salad.com","detail":"compact"}},{"tool":"match_people","arguments":{"job_id":1039886},"note":"people open to offers who fit this vacancy"}],"note":"The same data through MCP tools: https://alion.io/mcp/open needs no sign-in (compact searches, postings, people); https://alion.io/mcp with an Alion account has everything. Docs: https://alion.io/developers"}}