inference.academy

alphaXiv, Yifan Zhang: A decoder that carries its final hidden state across every token, prompt and response alike, so depth grows with sequence length instead of being fixed by layer count. Worth reading here for what it costs rather than what it saves:

DeepSeek, DeepSeek-AI: Long-horizon agents make the workload input-heavy, so the bytes of cache per token are the bill. V4.1-Flash attacks that in the architecture: the lower 20 of 40 layers run as an encoder and the decoder's global KV is projected from

arXiv, Taekyung Heo, Rasoul Shafipour, Ritchie Zhao and others: Every model swap in a cascade or a router throws the prefill away, because the bigger sibling cannot read the smaller one's cache. This paper fits a per-head ridge regression from the small m

Cohere, Xiaochun Tong, Conway Zhu and Donglu Wang: A decode step on one H100 moves 6.6 GB of weights and 0.5 GB of cache, so its speed of light is about 470 tokens a second; vLLM gets 185 of them and the gaps between kernels are where the rest go. One per

arXiv, Yaniv Leviathan, Matan Kalman and Yossi Matias: The paper that made a small model guessing ahead respectable: the target verifies the guesses in one pass and a rejection-sampling step keeps its output distribution exactly. Two to three times faster

arXiv, Charlie Chen, Sebastian Borgeaud, Geoffrey Irving and others: The same idea arrived at independently at DeepMind, on Chinchilla 70B with a 4B draft across a pod: two to two and a half times faster in a distributed setting where the draft has to be

arXiv, Amey Agrawal, Nitin Kedia, Ashish Panwar and others: Names the generation stall and removes it: prefill in chunks that ride inside decode steps under a per-step token budget, so a long prompt never freezes the batch. 2.6x the serving capacity of vL

Compound, Batuhan Aktas: Which provider fits your workload? 12,795 calls across 14 serving routes show how cost, first-token latency, generation speed and reliability change with prompt size and caching. Repeated prompts can change the cost ranking, inclu

SemiAnalysis, Cam Quilici, Bryan Shan, Alec Ibarra and others: An agent turn is 88K tokens in and 413 out, so the benchmark is really a KV cache hit-rate benchmark: 91% on B300 at 384 sessions, 73% on B200 with the rest spilling to DRAM. Nvidia, AMD, five

Perplexity, Perplexity Engineering: The roofline on one laptop. Decode hits 90% of the chip's sustained weight-read rate and removing the arithmetic changes nothing, so speculative decoding comes out 18% slower at batch 1.