Tag

Walks through the LSTM gating mechanism step by step, showing how the cell state and forget/input/output gates let the network carry information across long sequences where plain RNNs lose it to vanishing gradients.

Demonstrates that the order you feed inputs and outputs into a seq2seq model changes what it learns — even for sets that have no inherent order — and adds an attention-based set encoder plus a training loss that searches over output orderings.

Introduced dilated (atrous) convolutions, which expand a filter's receptive field exponentially with no loss of resolution and no extra parameters — the trick that let dense-prediction networks see wide context while keeping per-pixel detail.

Bet that one neural net, scaled with HPC, could transcribe both English and Mandarin without hand-built pipelines — reaching human-competitive accuracy by training fast enough to iterate on architecture in days, not weeks.

Before residual connections, stacking more layers made networks worse, not better — this 2015 paper fixed that by having layers learn a residual F(x)=H(x)-x via shortcut connections, enabling 152-layer nets that won ILSVRC 2015.

Re-examines residual blocks and shows that pure identity skip connections plus pre-activation (BN-ReLU before each conv) let gradients flow cleanly enough to train a 1001-layer ResNet, hitting 4.62% error on CIFAR-10.

Reframes the VAE's tendency to ignore its latent code as a controllable design choice: by limiting a PixelCNN decoder's receptive field and using autoregressive flow priors, the code is forced to keep only global structure and discard local texture.

Recasts a scatter of competing graph-network designs as one message-passing recipe — propagate, aggregate, read out — then proves it on QM9, hitting chemical accuracy on most molecular property targets without hand-built descriptors.

Isolates relational reasoning into a tiny plug-in module that scores pairwise object relations, bolting onto CNN/LSTM encoders to hit super-human 95.5% on CLEVR — and proving plain convnets lack this capacity on their own.

The 2017 paper that replaced recurrence with pure self-attention, making sequence models fully parallelizable — and, almost as a side effect, laying the architectural foundation for nearly every large language model that followed, from BERT to GPT.

Embeds multi-head self-attention inside an LSTM-style memory, so stored memories can attend to one another instead of just sitting in separate slots — sharpening relational reasoning and topping WikiText-103, Project Gutenberg, and GigaWord.

Chops any layer-sequence model across accelerators and splits each mini-batch into micro-batches to keep the pipeline busy, hitting near-linear speedup without architecture-specific tricks or fast interconnects.