Most LLM education starts top-down: import a library, call a pretrained model, tweak a few hyperparameters, and the machinery underneath stays a black box. This repository inverts that path — you hand-write the tokenizer, the attention mechanism, and the training loop until a working GPT emerges with nothing but PyTorch beneath it. The payoff isn't a model checkpoint; it's that "attention," "context window," and "pretraining" stop being jargon and become code you can read.
What Sets It Apart
- Every layer is yours. There's no
transformersimport to hide behind — tokenization, multi-head attention, and the GPT block are spelled out, so when a paper mentions KV caching or rotary embeddings you already know exactly where it would plug in. - Runs on a laptop. The base model is kept small enough to pretrain and finetune on a single consumer GPU, so you can execute every chapter rather than just read along.
- Bonus tracks beyond the book. Extra material reimplements modern architectures (Llama, Qwen, Gemma) and techniques like LoRA finetuning, turning the repo into a reference for how today's open models diverge from vanilla GPT-2.
Who It's For
Great fit if you can already train a basic neural net in PyTorch and want the transformer to stop feeling like magic — engineers moving into LLM work, or researchers who want a clean, hackable baseline. Look elsewhere if you need a production training stack: it optimizes for clarity over throughput, skips distributed multi-GPU scaling, and is a teaching codebase rather than a framework to deploy.
