doublewordai

Accelerating snapshot-based model serving with on-the-fly LZ4 memory compression in CRIU.

Three checkable conditions under which disaggregated prefill is never worse than aggregated serving, and the reasons it's usually better. Plus an inventory-theory model of traffic drift: cold starts, safety stock, and a map of the shocks a deployment can

Guide to the DeltaNet Family of linear attention mechanisms.

Why scale-up links are fast and short, how NVSwitch grew the domain from a board to a rack, and what TPU, UALink, and scale-up Ethernet do differently.

Freezing a live CUDA process to host memory and thawing it again, what the driver does - and doesn't - do to make that work, and how understanding that lets us restore CUDA processes up to 4x faster.

Some thinking about how to trade off batching and speculative decoding in a running inference engine.

Using artificial analysis benchmarks we try to predict when an open source LLM will be released that matches frontier LLMs.

Doubleword's batch inference offering keeps costs down by keeping throughput high, something which isn't easily done given the architecture of popular Mixture-o

Speculative decoding speeds up LLM generation by letting a system propose several