Most "learn how GPT works" repos make a choice: either readable but useless toy code, or production code you can't fit in your head. nanoGPT refuses the trade-off — it trains a real GPT-2 124M to a 2.85 validation loss yet keeps the whole thing in two files of roughly 300 lines each. The insight is that a competitive language-model trainer doesn't need ten thousand lines; it needs DDP, mixed precision, and a clean attention block, and not much else.
What Sets It Apart
- Teeth over education: unlike its predecessor minGPT, this version optimizes for actually reproducing GPT-2 results, not just pedagogy — you get real numbers (124M at 3.11 train / 3.12 val, scaling up to GPT-2-xl 1558M at 2.56) instead of a notebook demo.
- Hackable by design: train.py and model.py are short enough to read in one sitting and edit without fear, which means you can drop in a new attention variant or dataset and see the effect, rather than spelunking a framework.
- Two-tier on-ramp: a char-level Shakespeare model (6 layers, 6 heads, 384 channels) hits 1.47 val loss in ~3 minutes on an A100, and a shrunk config runs on a laptop CPU in the same time — so you can prototype before committing the ~4-day, 8xA100 full run.
Who It's For
Great fit if you want to understand transformer training by modifying working code, or need a clean baseline to fork for experiments. Look elsewhere if you want a polished, abstracted framework with plugins and configs for every scenario, or if you only need inference — and note the author now calls this repo old and deprecated in favor of its successor, nanochat.
Where It Fits
It sits between Hugging Face Transformers (full-featured, heavy) and pure-pedagogy notebooks (clear, non-scalable). nanoGPT is the rare middle: small enough to fully grasp, capable enough to reproduce a published model.
