Language Models are Unsupervised Multitask Learners

You played one round and we told you the writer's word. GPT-2 plays billions of rounds with no one to tell it anything — but it doesn't need anyone: the next word of the text is the answer key.

Scoring works like this. At each position, look up the probability the model gave to the token that really came next. High probability, small penalty; low probability, big penalty. Let's score three positions of our sentence by hand, with made-up probabilities (the real ones aren't published, so ours are for teaching):

What you just computed has a name — the cross-entropy loss, the average penalty per token — and pushing it down is the entire training signal. Written compactly, the probability of a text x made of tokens s₁ … sₙ is:

So the game is “guess the next token” and the grade is the average loss. Two questions remain before the interesting part: what plays the game, and on what text? Neither answer is where the novelty lives — which is itself worth knowing.

The player is a Transformer decoder — the same architecture we walk through block by block in our Attention Is All You Need explainer, so we're keeping its lid closed here. If you haven't seen inside one, all you need today is: it reads tokens left to right and is very good at using long context. The paper changes almost nothing about it (the small tweaks are in the fine print below). GPT-2 comes in four sizes; the name usually means the biggest, at 1.5 billion parameters.

The diet is the real design decision. The team wanted text that is big, diverse, and written by people who cared — diverse, because the core idea's bet needs the text to contain demonstrations of many tasks. A raw crawl of everything was rejected as mostly unintelligible. Instead they scraped every link posted on Reddit that earned at least 3 karma, using upvotes as a free, crowd-sourced “a human found this worth reading” filter:

The machine is standard and the grade is one number — the novelty is supposed to be in those 40 GB. So let's do what the paper does in Table 1: actually read some WebText, and notice what's hiding in it.

Here are three snippets. All three are real quotes from the training set, found by the authors — nobody wrote them to train a model. In each, we've marked the cue (the part that announces “a translation is coming”) in amber, and the answer (the translation the writer then helpfully provides) in coral:

And it isn't just translation. Forum posts hold questions followed by answers. News sites publish articles followed by “TL;DR:” and a summary. Interviews contain “in other words…” rephrasings. The paper's term for all of this is naturally occurring demonstrations: the web is full of people showing their work, cue and answer side by side.

You've now seen, in real training text, little translation lessons lying around. The question is why the model can't just ignore them and coast on easy English — and the answer is the averaging rule from step 2.

Recall: the loss is the average penalty over every position. Our sentence contains the position right after “Je ne suis pas un”. A model that has learned English grammar but no French can do nothing there — the next token is French, and its meaning is pinned by the English half of the sentence. Watch what one such position does to the whole sentence's grade. The slider is the probability the model manages to give the true word “imbecile”:

If your reaction is “hold on — getting one French word right is a party trick, not translation” — good. Slow down here, because this is the exact spot where we got stuck reading the paper too. The resolution: WebText contains millions of positions like this one, in both directions, at every length (step 4's exhibits were three of them). No lookup table covers them all. The cheapest machinery that predicts them all well is machinery that actually maps meaning between English and French — so that is what gradient descent, grinding down the average loss, slowly builds.

And nothing about this is special to French. Wherever the text has a question followed by its answer, the loss quietly demands question answering; wherever an article is followed by its summary, it demands summarizing. One objective, many skills — each one paid for, position by position.

You dragged the slider and watched one French word hold a whole sentence's average hostage. The paper compresses that observation into two dense sentences — worth unpacking slowly, because they're its theoretical heart.

First: a supervised task means “predict the output given the input” — p(output | input). A general system must also be told which task, so really it needs p(output | input, task). The usual approach wires the task into the architecture. The paper's move, following McCann et al.'s decaNLP: write the task in plain language, inside the text itself. A translation example can simply be the sequence (translate to french, english text, french text) — which is exactly the shape of our exhibit sentences, with “like they say in French:” playing the role of the task name.

Second — the dense one. The paper notes that the supervised objective and the unsupervised one are the same objective, evaluated on different tokens. Look at our sentence one more time:

That containment is the whole argument: the global minimum of the unsupervised objective is also a global minimum of the supervised one. Perfect next-word prediction contains perfect translation as a special case. The honest caveat, which the paper raises itself: nothing ever reaches the global minimum, so the real question is whether a merely large model, optimized the ordinary way, gets usefully far. Their preliminary experiments said yes — though much more slowly than direct supervision — and that is why Part C is all about measurement.

Training baked many skills into one next-word predictor. But at test time all the model can do is continue text — so how do you ask it for one specific skill, with no fine-tuning and no task flag?

You disguise your request as WebText. Hand the model a context shaped like the demonstrations it grew up on, and the most probable continuation — the thing it was trained to produce — is the answer you wanted. Every result in this paper is collected this way. The three formats below are the paper's own; notice that each one is just a step-4 exhibit with the coral part left blank:

Does the cue actually matter, or would the model summarize anyway? The paper measured it: remove the “TL;DR:” hint and summarization drops by 6.4 ROUGE points — the cue is a real switch. This way of working is what the paper calls zero-shot: no gradient updates, no task-specific training. One honest footnote: the translation prompt does place a few example pairs in the context, so “zero” refers to the weights — they never move — not to the context.

A prompt can only wake a skill that training actually built. Whether it gets built turns out to hinge on one variable above all: how big the model is. This is why the paper trained four sizes of the same design.

The cleanest test is LAMBADA — sentences whose final word needs at least 50 tokens of context to guess, so it's pure long-range next-word prediction, the model's home game. Drag across the four sizes and watch the numbers move (all real, from Table 3):

Across the paper's eight language-modeling benchmarks, the full-size GPT-2 sets a new state of the art on seven — zero-shot, having trained on none of their training data. And capacity keeps mattering all the way up: on open-domain question answering the smallest model can't clear 1% accuracy while the largest reaches 4.1% — a 5.3× jump from scale alone.

Language-modeling records are the model's home game, though. The paper's actual claim is about tasks — so here is the zero-shot scorecard, wins and embarrassments side by side, because the paper honestly reports both.

One more exhibit belongs here, because it made GPT-2 famous far beyond any scorecard: prompted with a fake news lede about unicorns discovered in the Andes, the model wrote a fluent, structured news article — inventing a scientist, quotes, a back-story. Read one detail closely, though: it names the herd “Ovid's Unicorn”, then calls them “four-horned, silver-white unicorns” one sentence later. Fluency arrived before coherence. (The paper labels the sample honestly: it's a cherry-pick, the best of 10 tries.)

Some of those scores look suspicious for a model nobody supervised. The paper raises the objection itself: with 40 GB of web text, maybe the test sets were simply in the training data, and “zero-shot” is quiet plagiarism.

To check, the authors built Bloom filters — a compact structure for asking “have I seen this exact phrase before?” — over every 8-word sequence in WebText, then measured how much of each benchmark's test set also appears in training. Overlap exists. But look at it next to the right baseline:

Read the two bars together: the standard benchmarks overlap their own training data (grey, 5.9%) almost twice as much as they overlap WebText (cinnabar, 3.2%). Whatever advantage GPT-2 gets from contamination, everyone else gets more of it. And when the authors sampled from GPT-2, most generations shared under 1% of their 8-grams with the training set. It can recite famous passages — feed it the opening of the Gettysburg Address, which appears ~40 times in WebText, and it recites for a while before drifting — but recitation is the exception, not the engine.

The closing observation flips the question around: on WebText's own held-out split, training and test loss fall together as model size grows, all the way to 1.5B parameters. The biggest model is still underfitting — it hasn't finished extracting what its own diet contains. Whatever ceiling this recipe has, GPT-2 wasn't at it.

Learning, mostly — and still hungry. That's the argument end to end: one game, a diet full of hidden lessons, a loss that can't skip them, prompts that wake them, scale as the fuel. Before the bigger story, check that it stuck.

Try each question in your head before opening it — the fold-out answer is a couple of sentences, and each one names the step to revisit if it doesn't come easily.