Most "train GPT-2 on a budget" projects ship one clever recipe and call it done. This one turned the recipe into a leaderboard. By freezing the hardware (8 NVIDIA H100s), the dataset (FineWeb), and the finish line (3.28 cross-entropy validation loss), it leaves wall-clock time as the only variable — so every contributor's trick is directly comparable, and the record only moves when something is genuinely faster.
What Sets It Apart
- It is where the Muon optimizer first proved itself: roughly 1.5x more sample-efficient than Adam, lower memory than Adam, and under 2% wallclock overhead — gains that showed up as record drops, not benchmark claims.
- The numbers are stark: the baseline (Karpathy's llm.c, May 2024) needed 45 minutes and ~10B tokens; the current record reaches the same loss in about 1.3 minutes on under 400M tokens.
- Records are open to anyone, by any method, and every entry ships runnable code — so the leaderboard doubles as a curated archive of training optimizations that actually work.
- A second track trains GPT-2 Medium to 2.92 loss, extending the same discipline to a larger model.
Who It's For
Great fit if you want a battle-tested, end-to-end reference for modern LLM training tricks — Muon, architectural tweaks, and data/throughput tuning — or a fair playground to pit your own optimization against a real record. Look elsewhere if you need a general-purpose, multi-node training framework: this is deliberately a single-node, 124M-parameter sprint, and its code is tuned for the leaderboard's exact constraints rather than flexibility or scale.
