Improving Language Understanding by Generative Pre-Training

Twelve tasks, twelve bespoke models, and a label famine. The paper's answer is a division of labor: let unlabeled text do the heavy lifting, and ask the labels to do only the last mile.

The whole method is the two lanes below. In stage 1, a Transformer plays a single game across the BooksCorpus — over 7,000 unpublished books — with no labels anywhere: given the words so far, guess the next one. In stage 2, those learned weights are carried over unchanged; our premise and hypothesis are flattened into one token sequence, a small output layer is bolted on top, and the whole thing is fine-tuned briefly on the labeled pairs.

One choice in this drawing matters more than it looks: which raw text. The team picked BooksCorpus over the better-known 1B Word Benchmark, which is a similar size but shuffled at the sentence level. Books keep long stretches of contiguous story, so guessing the next word sometimes requires remembering something from paragraphs ago — and that pressure is exactly what stage 2 will cash in on. Notice also the asymmetry between the lanes: 100 epochs of reading, then just 3 epochs of fine-tuning. Stage 1 does almost all of the work.

The recipe calls for one network in both lanes. Before we watch it train, let's look inside those ink-colored boxes.

If you've met the original Transformer, this is its decoder — the tower that writes — kept, and the encoder discarded. GPT-1 stacks 12 decoder blocks, each holding a masked multi-head self-attention layer and a feed-forward layer, with 768-dimensional states and 12 attention heads. Text is first chopped into sub-word pieces by byte-pair encoding (BPE) — a 40,000-entry vocabulary of fragments, so no word is ever out-of-vocabulary — and each piece gets an embedding plus a learned position vector. In the paper's notation the input matrix is built as h₀ = U·W_e + W_p, then flows up through the twelve blocks.

The word doing the real work here is masked. In the figure, look at the arcs under the token row: when the model computes the representation of a, attention may reach any word to its left, but the arc toward sport is cut. Every position is blind to its own future. That one constraint is what makes the model a generator — if a position could peek ahead, "guess the next word" would be a solved game with nothing to learn — and it's the design choice that separates this paper from BERT, which arrived months later reading in both directions but unable to generate.

We won't re-derive attention itself on this page — how those arcs get their strengths is a six-step calculation that deserves its own lesson, and the 2017 paper's explainer walks it in full. For today's story one sentence covers it: each position blends information from the positions it is allowed to see. What GPT-1 contributes is not the machinery but what the machinery is trained on, and what happens to it afterward.

The stack ends in a softmax over 40,000 pieces — a full probability distribution over what comes next. Stage 1 turns that distribution into a score, and the score into learning.

Our labeled pair sits out stage 1, but its hypothesis is still just an English sentence — and the objective works on any sentence; that indifference is the whole point of unsupervised learning. So let's borrow it as practice text. Cover the sentence with your hand and reveal it word by word. At each position the model has already published its distribution over what comes next; then the true word is revealed, and the model is graded on one number only: the probability it gave that word.

The grade is −ln p — natural-log units, called nats. Give the true word probability 1 and you score a perfect 0; hedge at 0.5 and you pay 0.69; at p = 0.01 the bill is 4.61. Let's grade three positions of our hypothesis with made-up but realistic probabilities (each row is one calculator keystroke — try one):

Keep that 3.53 — it returns in Step 8. Training is nothing more than nudging all the weights so this bill, summed over every position in 7,000 books, goes down. Feel how the penalty moves first:

You have now run the entire objective by hand. Sum your per-position scores (as log-probabilities, sign flipped back) over a context window of k tokens, and you get Equation 1 of the paper — maximize:

You just paid 0.87 nats on “sport” — the cheapest row in the table. A fair question before going further: why would shrinking that bill teach a model to judge entailment?

This is the strangest claim in the paper, and it deserves a slow step — the authors themselves present the explanation as a hypothesis, not a theorem. Here's the way in: ask what the model must know for row 3 to be cheap. To rank “sport” above “game”, “piano” or “sandwich” after Some men are playing a…, the network needs, somewhere in its weights, the fact that the things men play together are sports. Now look back at our gold label from Step 1: that is the same fact the entailment task grades. Producing the label was never the hard part — knowing that soccer is a sport was. And that knowledge sat in raw text all along, no annotator required.

Scale the pressure up. Across 7,000 books, the cheapest way to predict next words is not memorizing text — it's tracking who did what to whom, what got negated, what a pronoun refers to, how a scene tends to end. A model that keeps a mystery novel's suspects straight pays a smaller bill on the reveal page than one that doesn't. None of these skills were the goal. Every one of them lowers L₁, so gradient descent buys them anyway.

“A hypothesis is that the underlying generative model learns to perform many of the tasks we evaluate on in order to improve its language modeling capability.” — Section 5, Radford et al., 2018

A hypothesis needs evidence, and the paper offers two pieces we'll weigh properly in Part C: peel pre-trained layers off one at a time and task performance degrades layer by layer (Step 10); and with zero fine-tuning, the raw language model can already play the tasks — and keeps getting better at them the longer it pre-trains (Step 11).

Stage 1 left us weights holding facts like soccer ⊂ sport. But the network still only knows how to emit next-word probabilities — nothing in it can say “entailment”. Stage 2 has to build that exit, with as little new machinery as possible.

First problem: entailment hands us two sentences, and the machine eats one sequence. The paper's move — borrowed from what it calls traversal-style approaches — is disarmingly direct: concatenate them. Wrap the whole thing in randomly-initialized start and end tokens, put a delimiter $ between premise and hypothesis, and feed the result through the pre-trained stack exactly as if it were a stretch of book text:

Now the exit. The paper takes h_l^m — read the sub- and superscripts slowly, they stalled us too on a first pass: the activation of the last block at the m-th (final) input token, the dashed column above ⟨e⟩ in the figure. By the top of the stack, that one 768-number vector has had the chance to attend across the entire pair — it is the network's summary of everything it read. Multiply it by one new matrix W_y (768 × 3, one column per class), softmax the three resulting scores, and you have an entailment classifier. Say the scores for our pair come out as 2.1, 0.3 and −0.8 (invented for the arithmetic, like Step 4's table):

Same scoring rule as pre-training, different question: stage 1 asked which token comes next, stage 2 asks which of three labels is right. Summed over all labeled pairs, that gives the paper's Equations 3 and 4:

Flattening premise-$-hypothesis handled entailment. The other eleven datasets arrive in different shapes — single sentences, unordered pairs, a passage with candidate answers — and each gets its own flattening, never its own network.

The four recipes are below (the paper's Figure 1, redrawn in our colors). Two deserve a closer look. Similarity: “are these two sentences paraphrases?” has no natural first-vs-second order, but a left-to-right model reads order as meaning — so the input is flattened both ways, each order runs through the Transformer independently, and the two output vectors are added element-wise before the head. Multiple choice: each candidate answer gets its own sequence — context, question, $, that answer — each runs through the same network for a score, and one softmax across the candidates picks the winner.

Contrast this with the era's main alternative: keep the pre-trained network as a frozen feature extractor and build a task-specific architecture on top of it — which reintroduces exactly the per-task engineering the paper wants to delete. Here the adaptation lives in the data format, and the format is text: the one thing the model is already fluent in.

Step 6 ended with L₂ ≈ 0.20 nats of supervised loss on our pair. The obvious plan for stage 2 is to minimize L₂ and call it done. The paper doesn't stop there.

While fine-tuning on the labeled pairs, the model also keeps playing the guessing game on those same sequences — predicting each next token of the flattened premise-$-hypothesis text, exactly the L₁ arithmetic from Step 4, just run on stage-2 data. The two bills are added with a mixing weight λ, and we've already computed both ingredients for our pair: L₂ ≈ 0.20 from Step 6's softmax, and (using our three graded positions as a stand-in for the whole sequence's next-word bill) L₁ ≈ 3.53 from Step 4. Mix them yourself:

Why bother? The paper reports that the auxiliary term improves generalization and speeds up convergence, and the intuition fits everything we've built: L₁ is what put the knowledge in, and keeping it alive during fine-tuning stops the network from trampling that knowledge while chasing a few hundred thousand labels. The ablation in Table 5 adds a caveat the abstract doesn't advertise: the auxiliary loss helps on the large datasets — QNLI drops from 88.1 to 86.9 without it, MNLI and QQP dip too — but on small ones it gets in the way: CoLA actually rises from 45.4 to 47.9 when it's removed, and the naive average across the eight GLUE tasks is a wash (75.0 vs 74.7). A useful tool, not a magic one.

The recipe is complete: pre-train (Steps 2–4), see why it stores skills (Step 5), flatten and fine-tune (Steps 6–8). Time to face the twelve datasets from Step 1's table.

The same pre-trained weights, fine-tuned per task with nothing but Step 7's flattenings, set a new state of the art on 9 of the 12 — often beating ensembles of specialized models with one network. A sample, against the best previously published results:

And the three losses, because a scoreboard you can trust must show them: on RTE, the smallest entailment set (2,490 examples), GPT-1 scored 56.0 against 61.7 for a multi-task biLSTM — too few examples to fine-tune on, a gap the authors suggest multi-task training might close. On MRPC it trailed a specialized paraphrase system (82.3 vs 86.0), and on SST-2 its 91.3 fell short of 93.2. Worth saying plainly: the bespoke-model era did not lose every battle. It lost the war — one generic recipe now matched or beat nearly all of them at once.

Nine records still admit three explanations — the pre-training, the Transformer, or the auxiliary loss. Table 5 removes them one at a time and reruns everything.

Remove stage 1 entirely — train the same architecture directly on each task — and the average score falls from 74.7 to 59.9, a 14.8-point collapse, with every single task hurt. Swap the Transformer for a 2048-unit LSTM and the average drops 5.6 points (the LSTM wins on exactly one dataset, MRPC). The auxiliary loss you already weighed in Step 8: real help on big datasets, mild drag on small ones. So the ranking of ingredients is unambiguous — pre-training first, architecture second, λ third.

The second experiment speaks directly to Step 5's hypothesis. If pre-training stored task skills throughout the network, handing over more of the pre-trained stack should help monotonically — and it does. Transfer only the embedding layer and MultiNLI sits around 73; hand over all twelve blocks and it climbs to about 82. In the figure, each dot to the right adds one more pre-trained block; there is no plateau where the extra layers stop paying.

The layer ablation says the skills are spread through the stack. One question remains from Step 5: can we catch the raw language model doing a task with zero labels — no W_y, no stage 2 at all?

You can, if you phrase the task in the only language the model speaks: probabilities over text. Take sentiment on movie reviews. Append the word very to a review, ask the model for its next-word distribution, but compare only two entries: positive and negative. Whichever gets more probability is the verdict — the review's tone was already steering the model's expectations. The paper builds one such heuristic per task: for grammaticality, threshold the average per-token log-probability (broken sentences read as improbable text); for RACE, pick the answer the model finds most probable given the passage; for Winograd pronouns, substitute each candidate referent and keep the one that makes the rest of the sentence read more naturally.

The result is the figure below. Tracked over the course of pre-training, these zero-shot scores climb steadily — no labels, no fine-tuning, just more next-word practice — and the Transformer's curves sit above an LSTM's jittery ones. This is the paper's quietest figure and, in hindsight, its loudest: it says task ability grows inside the language model itself.

Hold this figure and Step 8's slider side by side and you can see a fork in the road. If pre-training alone keeps climbing, why keep stage 2 at all? Turning that loose thread into the whole program — delete fine-tuning, scale up the reading — is, almost verbatim, the pitch of GPT-2. Before the closing bookend, four questions to close the loop. Sketch an answer from memory, then check — Steps 4, 6 and 8 hold the receipts.