A one-line fix that finally made dropout work with LSTMs: apply it only to the non-recurrent connections, leaving the memory path untouched. This let researchers train much larger RNNs without the overfitting that had capped their size.
End-to-end encoder–decoder using deep LSTMs to map variable-length input sequences to output sequences; demonstrated competitive English→French translation (BLEU 34.8) and improved optimization by reversing source sentences, showing strong handling of long sentences.
Bolts a differentiable, addressable memory bank onto a neural network and trains the whole thing with gradient descent, letting it learn algorithms like copying, sorting, and recall from examples — a learned computer rather than a fixed circuit.
Repurposes attention as a pointer that selects positions in the input rather than blending them into a context vector, so the output vocabulary can grow with input length — handling sorting, convex hulls, and TSP that fixed seq2seq cannot.
Demonstrates that the order you feed inputs and outputs into a seq2seq model changes what it learns — even for sets that have no inherent order — and adds an attention-based set encoder plus a training loss that searches over output orderings.
Introduced dilated (atrous) convolutions, which expand a filter's receptive field exponentially with no loss of resolution and no extra parameters — the trick that let dense-prediction networks see wide context while keeping per-pixel detail.
Bet that one neural net, scaled with HPC, could transcribe both English and Mandarin without hand-built pipelines — reaching human-competitive accuracy by training fast enough to iterate on architecture in days, not weeks.
Before residual connections, stacking more layers made networks worse, not better — this 2015 paper fixed that by having layers learn a residual F(x)=H(x)-x via shortcut connections, enabling 152-layer nets that won ILSVRC 2015.
Combines a policy network (to narrow move choices) and a value network (to score board positions) with Monte Carlo tree search, cutting Go's vast search space enough to beat top programs 99.8% of the time and the European champion 5-0.
Re-examines residual blocks and shows that pure identity skip connections plus pre-activation (BN-ReLU before each conv) let gradients flow cleanly enough to train a 1001-layer ResNet, hitting 4.62% error on CIFAR-10.
Reframes the VAE's tendency to ignore its latent code as a controllable design choice: by limiting a PixelCNN decoder's receptive field and using autoregressive flow priors, the code is forced to keep only global structure and discard local texture.
Recasts a scatter of competing graph-network designs as one message-passing recipe — propagate, aggregate, read out — then proves it on QM9, hitting chemical accuracy on most molecular property targets without hand-built descriptors.