Isolates relational reasoning into a tiny plug-in module that scores pairwise object relations, bolting onto CNN/LSTM encoders to hit super-human 95.5% on CLEVR — and proving plain convnets lack this capacity on their own.
The 2017 paper that replaced recurrence with pure self-attention, making sequence models fully parallelizable — and, almost as a side effect, laying the architectural foundation for nearly every large language model that followed, from BERT to GPT.
Embeds multi-head self-attention inside an LSTM-style memory, so stored memories can attend to one another instead of just sitting in separate slots — sharpening relational reasoning and topping WikiText-103, Project Gutenberg, and GigaWord.
Introduced the two-stage recipe behind the GPT lineage: unsupervised generative pre-training on unlabeled text, then supervised fine-tuning per task. A single 12-layer Transformer decoder beat bespoke architectures on 9 of 12 NLP benchmarks.
Pre-trains a deep bidirectional Transformer encoder with masked-language-modeling and next-sentence prediction, then fine-tunes one model on 11 NLP tasks, reaching state-of-the-art on GLUE, SQuAD, and MultiNLI with little task-specific tuning.
Chops any layer-sequence model across accelerators and splits each mini-batch into micro-batches to keep the pipeline busy, hitting near-linear speedup without architecture-specific tricks or fast interconnects.
A 1.5B-parameter model trained only to predict the next token on diverse web text does translation, summarization, and QA zero-shot, with no fine-tuning. It recast NLP tasks as conditional language modeling and sparked the staged-release misuse debate.
Demonstrated that language model loss falls as a smooth power law in model size, data, and compute across more than seven orders of magnitude — turning "make it bigger" from a hunch into a budget you can plan, and justifying the GPT-3 scale-up.
At 175 billion parameters, this autoregressive model becomes a strong few-shot learner: it handles translation, QA, and reasoning from a few prompt examples with no gradient updates, establishing in-context learning as an alternative to fine-tuning.
Collects 60+ PyTorch implementations of neural network papers — transformers, diffusion, GANs, RL, optimizers — each annotated line-by-line and rendered beside the code at nn.labml.ai, so you study the math and a runnable implementation together.
Showed that fine-tuning a GPT model on public GitHub code yields a capable program synthesizer, and introduced HumanEval — the docstring-to-function benchmark that still anchors code-generation evaluation. A production variant powers GitHub Copilot.
AI Dataset2022
A 57-subject multiple-choice benchmark for measuring broad language understanding in LLMs; provides per-subject configs and test/dev/auxiliary_train splits for few-/zero-shot evaluation, widely used for model comparison and academic reporting.