Overview
News
Technologies
Salaries
Products
People
Growth
Financials
Overview
Kubernetes-native, high-performance distributed LLM inference.
News
Sticky Until Saturated: Token-Aware Routing in llm-d
How the llm-d router balances KV-cache affinity against token load The llm-d router's default configuration has changed to something an operator can reason about, built on a single methodology: token-aware routing. The scheduler prioritizes KV-cache affin
Read more
Report
Pull, Don't Recompute: Peer-to-Peer KV Cache Sharing in llm-d
A request arrives and its 48K-token prefix is already sitting in a KV cache, but the endpoint holding it - one schedulable serving instance: either a complete engine instance or a rank within a multi-rank deployment - is busy. Route the request to the cac
Read more
Report
Scaling Vision-Heavy Kimi-VL with Heterogeneous E/PD on llm-d and SGLang
Vision-language model (VLM) inference involves several stages: vision encoding, language model prefill, and language model decode. Each stage has different compute and memory characteristics, making heterogeneous disaggregation a natural fit. This gives u
Read more
Report
Serving GLM-5.2 for Agentic Workloads on llm-d
llm-d's well-lit paths are tested, benchmarked recipes for serving large language models in production. The agentic-serving path starts with the workload and combines the capabilities it needs into a deployment for each accelerator. This post applies that
Read more
Report
RL Post-Training: Co-Operative Time-Slicing with llm-d
In Reinforcement Learning (RL) post-training for Large Language Models using algorithms like GRPO, optimizing the ratio of generator (sampler) to trainer throughput is the single largest driver of infrastructure Total Cost of Ownership (TCO). Because typi
Read more
Report
Pro access
Upgrade to see all 27 mentions
Upgrade to a paid plan to read every media mention of this company - funding news, awards, product launches and press releases from all the outlets writing about it.
Every media mention and press release
Funding news, awards and product launches
Fresh coverage from every outlet writing about the company
Upgrade now
Cancel anytime. Secure checkout. Instant activation.