Outrageously Small Neural Networks: Emergent Basic Reasoning at 6,616 tok/SEC [pdf] (huggingface.co)

blog: https://gregdiamos.com/2026/09/07/outrageously-small-neural-...

X discussion: https://x.com/GregoryDiamos/status/2096873745420075020?s=20

[flagged]

The loss does not saturate. Across a 4.91B-token run, smoothed training loss falls monotonically within each curriculum phase and is still descending at the end

Data is doing more of the work than it used to. Every source in our mixture is a curated artifact built with large models

Training a model this small on them is distillation

When models of this size were last studied seriously such corpora did not exist

[dead]