Demonstrated that language model loss falls as a smooth power law in model size, data, and compute across more than seven orders of magnitude — turning "make it bigger" from a hunch into a budget you can plan, and justifying the GPT-3 scale-up.
A line-by-line PyTorch reimplementation of the Transformer paper as a runnable notebook, where each part of the paper sits next to the code that implements it — turning a dense architecture into something you can read and run end to end.
Builds a single rigorous theory from one question: why some bit strings look random. Defines plain and prefix complexity, the incompressibility method, and Martin-Löf randomness, tying information content to whether a short program can reproduce a string.