The headline in 2019 was 1.5 billion parameters, but the real claim was stranger: GPT-2 was never told what tasks to perform. Trained only to predict the next token on 40GB of web text, it picked up translation, summarization, and question answering as byproducts — because those behaviors already appear, implicitly, in natural text. The quiet thesis: a good enough language model is already a multitask learner.
What Sets It Apart
- Tasks as conditioning, not architecture. Everything is framed as predicting output given input and a task description supplied in the prompt — no task-specific heads, labeled datasets, or gradient updates. This is the direct ancestor of prompting and in-context learning.
- Scale as the lever. GPT-2 set zero-shot state of the art on 7 of 8 language-modeling benchmarks, and quality climbed smoothly with model size — an early signal of what later became scaling laws.
- Data curation matters. WebText was built from outbound Reddit links with at least 3 karma, deliberately trading raw crawl volume for higher-signal text.
Great Fit If
Read it to understand where prompting, in-context learning, and the "just scale the language model" strategy came from. Look elsewhere if you need competitive benchmark numbers — GPT-2 is a 2019 baseline long surpassed — or training and sampling code, which live in OpenAI's release rather than in the paper.
A Note on the Staged Release
GPT-2 is also remembered for how it shipped: OpenAI initially withheld the full 1.5B model citing misuse risk, then released it in stages. That choice opened a still-running debate about responsible disclosure norms for capable generative models.
