Frames generative modeling as a two-player game: a generator forges data while a discriminator learns to spot fakes, training both by backpropagation alone — no Markov chains, no inference networks. The adversarial pressure yields sharp samples.
A one-line fix that finally made dropout work with LSTMs: apply it only to the non-recurrent connections, leaving the memory path untouched. This let researchers train much larger RNNs without the overfitting that had capped their size.
Bolts a differentiable, addressable memory bank onto a neural network and trains the whole thing with gradient descent, letting it learn algorithms like copying, sorting, and recall from examples — a learned computer rather than a fixed circuit.
Stanford's course teaches deep learning by making you build vision models from scratch — k-NN and linear classifiers up through CNNs, detection, segmentation, and Transformers — with three PyTorch assignments and a self-chosen final project.
Karpathy's 2015 walkthrough of character-level RNNs trained to predict the next character, showing how a tiny model learns to generate convincing Shakespeare, C code, and LaTeX — and what its neurons actually track.
Repurposes attention as a pointer that selects positions in the input rather than blending them into a context vector, so the output vocabulary can grow with input length — handling sorting, convex hulls, and TSP that fixed seq2seq cannot.
Walks through the LSTM gating mechanism step by step, showing how the cell state and forget/input/output gates let the network carry information across long sequences where plain RNNs lose it to vanishing gradients.
Demonstrates that the order you feed inputs and outputs into a seq2seq model changes what it learns — even for sets that have no inherent order — and adds an attention-based set encoder plus a training loss that searches over output orderings.
Before residual connections, stacking more layers made networks worse, not better — this 2015 paper fixed that by having layers learn a residual F(x)=H(x)-x via shortcut connections, enabling 152-layer nets that won ILSVRC 2015.
Re-examines residual blocks and shows that pure identity skip connections plus pre-activation (BN-ReLU before each conv) let gradients flow cleanly enough to train a 1001-layer ResNet, hitting 4.62% error on CIFAR-10.
AI Infra2016
Meta (Facebook AI Research), PyTorch Foundation (Linux Foundation)PyTorch Foundation (Linux Foundation)
Lets researchers and engineers build neural networks as regular Python programs, with GPU-backed tensors, autograd, distributed training, and production paths through TorchScript and related tooling.
Builds deep learning from the ground up, first teaching the linear algebra, probability, and numerical methods most ML texts assume you know. Three parts run from math foundations to practical networks to research topics, favoring reasoning over recipes.