Sheng Zha

MOSAIC jointly selects a sparse-MoE architecture, token budget, and parallel layout under a fixed cluster and training window.

A practitioner's account of what pretraining research can predict, where current methods break, and which questions remain open.

AI raised individual throughput but coordination overhead stayed fixed. For many product-engineering orgs, the bottleneck flipped from compute-bound to communication-bound.

A practical framework for comparing research projects by probability of success, effect size, and weighted reach.

Deriving model size and training tokens from a fitted scaling law, then extending the calculation to repeated data, inference demand, and cluster efficiency.

Why we built GluonNLP to make NLP experiments easier to reproduce, maintain, and reuse.