{"id":2612497,"url":"https://alion.io/job/cast-ai-senior-ml-engineer-kimchi-inference-stack-optimization","title":"Senior ML Engineer | Kimchi (Inference Stack Optimization)","company":{"id":54,"name":"CAST AI","domain":"cast.ai","url":"https://alion.io/company/cast-ai","size_band":"201-500","is_staffing_agency":false,"employer_type":"direct","is_intermediary":false,"listed_via":null,"ats_vendor":"Greenhouse","truth_index":{"grade":"B","score":82,"open_postings":7,"ghost_share":0,"stale_share":0.714,"repost_share":0,"time_to_fill_p50_days":49,"computed_at":"2026-10-15T05:46:00Z"}},"role":"AI/ML","role_family":"AI/ML","seniority":"senior","employment_type":null,"work_mode":"remote","remote_scope":"stated_countries","remote_scope_basis":"board_field","remote_working_hours":null,"hiring_geo_confidence":"structured","locations":["Austria","France","Germany","Italy","Netherlands","Poland","Spain","United Kingdom"],"countries":["AT","FR","DE","IT","NL","PL","ES","GB"],"hiring_countries":["AT"],"hiring_countries_total":1,"salary":null,"salary_estimate":{"min_usd":65000,"max_usd":168000,"period":"year","method":"global_role_cell_scaled_by_country","sample_n":2102},"experience_years_min":null,"visa_sponsorship":true,"relocation_package":false,"has_equity":true,"technologies":[{"name":"Anthropic","optional":false},{"name":"ClickHouse","optional":false},{"name":"CUDA","optional":false},{"name":"CUDA Toolkit","optional":false},{"name":"DeepSeek","optional":false},{"name":"GitLab CI","optional":false},{"name":"Grafana","optional":false},{"name":"Kubernetes","optional":false},{"name":"LLM","optional":false},{"name":"MiniMax","optional":false},{"name":"OpenAI","optional":false},{"name":"OpenTelemetry","optional":false},{"name":"Prometheus","optional":false},{"name":"Python","optional":false},{"name":"PyTorch","optional":false},{"name":"Quantization","optional":false},{"name":"SGLang","optional":false},{"name":"TensorRT-LLM","optional":false},{"name":"vLLM","optional":false},{"name":"KV Cache","optional":true},{"name":"Perplexity","optional":true},{"name":"Speculative Decoding","optional":true},{"name":"TensorRT","optional":true}],"status":"live","first_seen_at":"2026-10-15T08:07:38Z","employer_posted_date":"2026-10-15","last_verified_at":"2026-10-15T14:48:30Z","board_verified":true,"closed_at":null,"days_open":0,"trust":{"level":"ok","repost_count":null,"flags":[],"days_open":0},"description":"Why Kimchi?\nGreat models are raw ingredients. Kimchi is what happens when you ferment them with good engineering. We started inside CAST AI helping companies run LLMs on their own Kubernetes clusters; today we run our own coding harness on any model you pick, and we run the models underneath it ourselves.\nKimchi Inference\nOne API line and a customer's existing client just works: DeepSeek, GLM, Kimi, MiniMax, Nemotron served serverless from our own GPU fleet (OpenAI- and Anthropic-compatible, pay per token, spend visible per user, team, and key). When control matters, the same models run on the customer's own cluster instead. Behind that one line sits the machinery you'll own: SGLang and vLLM engines, B300 nodes, a shared cluster filesystem that holds every checkpoint exactly once, KV caches riding coding traffic with heavy prefix reuse, and the benchmark harness that gates every change we ship.\nMade by Cast AI - a Miami-headquartered, globally distributed team across 34 countries that hit unicorn status in January 2026 with a $50B+ backed investment pushing its valuation past $1 billion.\nAbout the role\nThroughput. Latency. Cold starts. Cost per token.\nMove those four numbers in the right direction and two things happen: customers get faster, cheaper inference, and our margins improve. That's the entire thesis of this role. Every loader you tune, every scheduler decision you land, every kernel you stare at shows up in a customer's p99 and on our P&L. Your mandate is the inference stack's optimization, whatever it takes, and the technical direction of how we optimize it is yours to argue with data.\nThe problem: the checkpoints run 0.3-1.5 TB, and the \"right\" serving configuration depends on traffic shape, sequence-length distribution, batch dynamics, GPU SKU, memory bandwidth, and quantization tolerance, all variables that shift week to week. Most teams pick a configuration once, over-provision GPUs, and absorb the cost. We don't: we took Kimi-K3 (1.5 TB) from over an hour cold-to-serving to under twelve minutes with a shared filesystem, O_DIRECT weight loaders, and node-local JIT caches, and we're nowhere near done. Prefill/decode disaggregation and tiered KV caches are next.\nStack: Python; SGLang; vLLM; TensorRT-LLM; PyTorch; fastsafetensors; CUDA-adjacent tooling; Kubernetes; virtiofs shared filesystems; ClickHouse; OpenTelemetry; Prometheus; Grafana; GitLab CI.\nRequirements\nYou've built real ML systems for 5+ years, and your portfolio shows depth in inference or training infrastructure, not just model training notebooks.\nYour Python is production-grade: services, not scripts.\nYou've run vLLM, SGLang, or TensorRT-LLM for real, and you have a working mental model of why an engine performs the way it does on a given GPU.\nYou've measured quantization regressions, not just compression ratios.\nMulti-GPU and multi-node setups hold no fear: tensor and expert parallelism, collective communication, and their practical failure modes.\nYou know where the bytes live (page caches, O_DIRECT, network filesystems under terabyte-scale reads) and how fast they move.\nYou instrument before you optimize, and you can tell a real win from a benchmark artifact.\nA wide mandate excites you more than it unsettles you.\nNice to haves\nYou've worked on LLM routing specifically: RouterBench-class evals, cost/quality/latency Pareto fronts.\nCalibration, abstention, or conformal methods in production decision systems.\nYou've run online A/B loops with drift monitoring on a model that touches real money.\nResponsibilities\nPush throughput. Continuous batching, chunked prefill, speculative decoding with draft models, CUDA-graph capture, and scheduler tuning across the engine fleet. Find the ceiling on each GPU SKU, then raise it.\n\nCut latency. Attack TTFT and inter-token latency separately; profile to find the actual bottleneck (compute, memory bandwidth, scheduling, storage, or network) and fix it, not the one you assumed.\n\nGet more out of the KV cache. Hierarchical and paged attention, prefix caching on heavy-reuse coding traffic, eviction policies, quantized KV. A lot of the unrealized throughput lives here, and you'll own it.\n\nKill cold starts. Weight loaders (O_DIRECT, fastsafetensors-class), JIT and compile caches, shared-filesystem read paths. A 1.5 TB checkpoint has no business taking an hour to reach serving.\n\nQuantize without regressing quality. INT8/INT4, FP8/FP4 across weights, activations, and KV, measured on real workloads, not just perplexity benchmarks.\nScale across nodes. Distributed inference topologies, network-aware placement, prefill/decode disaggregation, checkpointing strategies that don't bottleneck on storage or interconnect.\n\nSet the technical direction. Decide what we benchmark, what we adopt, and what we build ourselves. Bring the team along with strong writeups and reproducible experiments.\n\nWhat’s in it for you?\nCompetitive salary (depending on the level of experience).\nEnjoy a flexible, remote-first global environment.\nCollaborate with a global team of cloud experts and innovators, passionate about pushing the boundaries of Kubernetes technology\nEquity options.\nGet quick feedback with a fast-paced workflow. Most feature projects are completed in 1 to 4 weeks.\nSpend 10% of your work time on personal projects or self-improvement. \nLearning budget for professional and personal development - including access to international conferences and courses that elevate your skills.\nAnnual hackathon to spark new ideas and strengthen team bonds.\nTeam-building budget and company events to connect with your colleagues.\nEquipment budget to ensure you have everything you need.\nExtra days off to help maintain a healthy work-life balance.\nHiring process\nScreening call with Recruiter\nHiring Manager interview\nTechnical interview (system design)\nLive coding\nCulture Check interview with an executive\nAs part of our standard hiring process, we would like to inform you that a background check may be conducted at the final stage of recruitment through our third-party provider, Checkr.\nPlease note that Cast AI does not provide any form of visa sponsorship/work permit.","description_format":"text","description_chars":6146,"description_truncated":false,"requirements":{"experience_years_min":null,"management_years_min":null,"team_size_min":null,"manages_managers":false,"education":null,"security_clearance":false,"languages":[]},"benefits":["Company events","Equity"],"hiring_locations":[{"name":"Austria","iso":"AT","kind":"country"}],"hiring_excludes":[],"relocation_offered":false,"industries":["Virtualization & Containers","Cloud Cost & Multi-Cloud Management"],"lifecycle":[{"event":"open","at":"2026-10-15T11:25:10Z"}],"visa":[],"liveness":{"score":86,"band":"hot","label":"Hiring now","p_open":1,"p_active":0.86,"p_room":1,"age_days":0,"expected_fill_days":49,"reasons":["conf:2","win:early"],"computed_at":"2026-10-15T17:33:55Z"},"pay":null,"html_url":"https://alion.io/job/cast-ai-senior-ml-engineer-kimchi-inference-stack-optimization","json_url":"https://alion.io/job/cast-ai-senior-ml-engineer-kimchi-inference-stack-optimization.json","meta":{"generated_at":"2026-10-15T17:33:55Z","cache_seconds":300,"methodology":"https://alion.io/methodology","terms":"https://alion.io/terms","contact":"https://alion.io/contact","api":"https://alion.io/developers","about":"Alion is a live layer of people, companies and AI agents: who they are, whether they are real and active right now, what they do and how to work with them, readable by people and by agents and paid per call.","catalog":"https://alion.io/catalog.json","usage":{"tier":"crawler_verified","counted_by":"address","units_charged":1,"used_today":26589,"day_limit":null,"remaining_today":null,"minute_limit":300,"resets_at":"2026-10-16T00:00:00Z"}},"offers":[{"id":"company.slices","title":"One company in depth, by slice","status":"live","price":{"credits":0.02,"usd":0.002,"plus_per_slice":{"credits":0.05,"usd":0.005}},"unit":"per company, plus each slice with data","note":"the employer in depth","call":{"mcp_tool":"get_company","arguments":{"id":54},"rest":"https://alion.io/mcp/rest/get_company?id=54"},"human":"https://alion.io/catalog?offer=company.slices&for=job%2Fcast-ai-senior-ml-engineer-kimchi-inference-stack-optimization"},{"id":"market.stats","title":"A market slice: pay, demand and time to fill","status":"live","price":{"credits":1,"usd":0.1},"unit":"per slice","note":"pay, demand and time to fill for this role and place","call":{"mcp_tool":"market_stats"},"human":"https://alion.io/catalog?offer=market.stats&for=job%2Fcast-ai-senior-ml-engineer-kimchi-inference-stack-optimization"},{"id":"job.search","title":"Open jobs by role, technology, place, pay and visa","status":"live","price":{"credits":0.02,"usd":0.002},"unit":"per posting in a list","note":"similar open postings","call":{"mcp_tool":"search_jobs"},"human":"https://alion.io/catalog?offer=job.search&for=job%2Fcast-ai-senior-ml-engineer-kimchi-inference-stack-optimization"},{"id":"company.verify","title":"Is this company real and active right now","status":"pilot","price":null,"unit":"per company","request":{"url":"https://alion.io/catalog/request","method":"POST","body":"{\"offer\": \"company.verify\", \"for\": \"job/cast-ai-senior-ml-engineer-kimchi-inference-stack-optimization\", \"note\": \"what you need it for\"}"},"human":"https://alion.io/catalog?offer=company.verify&for=job%2Fcast-ai-senior-ml-engineer-kimchi-inference-stack-optimization"}]}