Treats a network's weights as a noisy channel and penalizes the bits needed to describe them, formalizing the "bits-back" coding trick — an early variational argument later recognized as a conceptual ancestor of the VAE.
Reframes model selection as data compression: the best hypothesis is the one that lets you describe the data in the fewest bits. Walks through MDL twice — once conceptually, once with full math — turning Occam's razor into a usable inference principle.
Gives intelligence a falsifiable mathematical definition — an agent's expected reward across all computable environments, weighted by simplicity — turning a fuzzy word into the Universal Intelligence Measure built on AIXI and Solomonoff induction.
Argues that "interesting" complexity is low in both ordered and fully random states but peaks in between, and proposes "complextropy" — a resource-bounded Kolmogorov-complexity measure — to capture the rise-then-fall pattern entropy can't explain.
The result that kicked off the deep learning era: in 2012 a deep CNN cut ImageNet top-5 error from 26% to 15%, showing that GPU-trained networks with ReLU and dropout could beat decades of hand-engineered computer vision features.
Measures why complexity in closed systems rises then falls while entropy only climbs, using a coffee-and-cream cellular automaton. The key result: only interacting particles produce a transient complexity peak; non-interacting ones never do.
First model to make a decoder dynamically focus on different source words instead of cramming a whole sentence into one fixed vector — the soft-alignment idea that became "attention" and, three years later, powered the Transformer.
A one-line fix that finally made dropout work with LSTMs: apply it only to the non-recurrent connections, leaving the memory path untouched. This let researchers train much larger RNNs without the overfitting that had capped their size.
Bolts a differentiable, addressable memory bank onto a neural network and trains the whole thing with gradient descent, letting it learn algorithms like copying, sorting, and recall from examples — a learned computer rather than a fixed circuit.
Stanford's course teaches deep learning by making you build vision models from scratch — k-NN and linear classifiers up through CNNs, detection, segmentation, and Transformers — with three PyTorch assignments and a self-chosen final project.
Karpathy's 2015 walkthrough of character-level RNNs trained to predict the next character, showing how a tiny model learns to generate convincing Shakespeare, C code, and LaTeX — and what its neurons actually track.
Repurposes attention as a pointer that selects positions in the input rather than blending them into a context vector, so the output vocabulary can grow with input length — handling sorting, convex hulls, and TSP that fixed seq2seq cannot.