idlemachines

How does a neural network understand text? It's a question so basic, you can be forgiven for forgetting about it. Words and letters are human-readable, but are discrete (even if infinite in combination) and not compatible with the continuous representatio

What goes wrong when naively putting weight decay in the Adam optimiser, and how AdamW fixes it.

Thinking Machines' first model encodes position with a learnt attention bias rather than RoPE, but it's not just a replacement, it's a fundamental shift in how position is thought of.

What every line of a PyTorch training loop does, why it belongs where it is, and what breaks if you move it.

Mean squared error and mean absolute error are usually compared by how they handle outliers. But how often do you think that MSE predicts the conditional mean and MAE the conditional median? And if you do, how often does Gauss or Laplace come to mind?

Perplexity is the geometric mean of inverse probabilities, which makes it the effective branching factor the model is choosing between. The number on its own tells you almost nothing across papers, but it's a useful sanity check at initialisation.

There is no single 'more efficient'. A design can be cheaper in FLOPs and more expensive in time at the same loss, and the two can rank models in opposite orders. MAI's Efficiency Gain is the metric that makes that disagreement precise.

Every model that turns a variable-length collection of vectors into one vector is making a choice about what to throw away. Pooling methods are a spectrum, and frontier embedding models live at specific points on it.

A technical reading of DeepSeek V4: Compressed Sparse Attention, Heavily Compressed Attention, manifold-constrained hyper-connections, Muon, and MXFP4, derived from the paper and the inference code.

Gemma 4 makes five quiet departures from the standard transformer recipe. QK-norm instead of 1/d, partial RoPE on global layers, per-layer input gating, KV sharing across layers, and an MoE that sits alongside the MLP rather than replacing it.