Language Models are Few-Shot Learners

One frozen model, tasks ordered off a menu written in plain text — that was the wish. Before we can test it, we need the menu itself: exactly what text goes in, for each way of asking.

The paper lines up four settings on a single axis: how much task-specific data each one consumes. At one end sits fine-tuning. The other three are just prompts — the same frozen GPT-3, fed different amounts of text. Read each panel below as literally what the model sees for our word-repair task:

Look at where the coral chips live: only in the first panel. From here to the end of this page, coral never appears again — that absence is the paper. Between the other three panels the only thing that varies is how much teal text precedes the question.

What caps K? The context window. Every GPT-3 model reads at most 2,048 tokens at once (double GPT-2's window). One demonstration line of our task costs very roughly 16 tokens, so 2,048 ÷ 16 ≈ a bit over 100 examples — which is exactly the “typically 10 to 100” range the paper works in. The menu ends where the window does.

Few-shot, then, is just pasting K worked examples above your question. But why would pasting text into a frozen model teach it anything at all?

Honestly, this is where we stalled when first reading the paper: if every weight is frozen, what is there left to learn with? It feels like a contradiction. The resolution is that weights aren't the only state a network has. As the model reads your prompt, it computes activations — a fresh, throwaway state, token by token. Adaptation can live there. The paper (§7) points to a whole lineage of meta-learning work built on exactly this: an inner loop of adaptation that happens “through computation in the model's activations,” while an outer loop trains the weights.

Where would a language model pick up such an inner-loop skill? From the training text itself. Ordinary documents repeat little sub-tasks:

Follow the teal arrows inside any one card. To predict the fourth line well, given the first three, the most useful thing a model can do is pick up the pattern on the spot — the loss rewards nothing less. So the coral outer loop, in the course of ordinary pretraining, keeps paying the model for being good at the teal inner loop. The paper calls the whole two-level arrangement meta-learning, and the inner loop in-context learning. A few-shot prompt is nothing exotic: it hands the model exactly the kind of sequence it was implicitly practiced on.

The inner loop is a skill the outer loop has to build — and the paper bets that skill grows with size. Testing a bet about size means building more than one machine.

The model section of this paper is deliberately boring, and that's the point. GPT-3 reuses GPT-2's architecture — same Transformer decoder, same modified initialization, pre-normalization and tokenizer — with one listed change: it alternates dense attention layers with locally banded sparse attention layers. We won't unpack sparse attention here (it comes from the Sparse Transformer paper; §2.1 gives the one-line citation). Nothing about the method is new. What's new is the dial:

The fuel is a curated slice of the internet: Common Crawl, filtered against a quality classifier and deduplicated, then deliberately under-sampled so the cleaner corpora punch above their weight — the bars below show each source's share of training examples, not its share of bytes. The filtered Common Crawl is seen less than half of one epoch; tiny, clean Wikipedia is recycled 3.4 times.

Now the bill. Let's actually compute it — three short lines, using the paper's own accounting from Appendix D:

For calibration: GPT-2's training budget was tens of petaflop/s-days. This is roughly a hundredfold jump, run on V100 GPUs on a Microsoft-provided cluster — and 10× more parameters than any previous dense language model. One reassuring check before we use this machine: its validation loss lands right on the power-law line that the scaling-laws paper drew, now extended two more orders of magnitude (Fig. 3.1). The outer loop behaved exactly as predicted. The open question is the inner one.

Eight sizes of one machine, and a garbled-word task too artificial to have been crawled. Now the experiment the whole paper hangs on: freeze the weights, lengthen the prompt, count correct answers.

This is the paper's Figure 1.2, rebuilt so you can drive it. The dial is K, the number of worked examples pasted above the question. Nothing else changes — same checkpoint, same test words. As you drag, watch two things: the prompt on the right physically growing (that teal text is the entire intervention), and where your model's curve sits compared to the other two.

You just watched 175B pull nearly +59 points out of pure text while 1.3B barely moved. The obvious objection: maybe the big model isn't learning anything at test time — maybe it memorized more of the internet.

The paper's first defense is built into the task design. Those garbled strings are generated, by scrambling the 10,000 most common words — text like r e!c.i p r o.c a/l essentially doesn't occur in the wild. And the scores agree: with no examples, every model is at or near the floor on all five scrambling variants (reversed words: 0.09% even for 175B). If solved pairs were sitting in the training data, zero-shot wouldn't be this bad. The skill shows up only when demonstrations enter the context — which is what “learned at test time” should look like.

Second, do one last bit of arithmetic on the numbers from Step 5 — subtract each model's zero-shot score from its K = 100 score. That difference is how much the model got from the demonstrations alone:

Three subtractions, and you've reproduced the paper's headline: the value of in-context examples roughly doubles with each ~10× of model size on this task. Averaged over all 42 accuracy benchmarks the same fan appears — zero-shot climbs steadily with scale, few-shot climbs faster (Fig. 1.3).

And yet the authors stop short of declaring victory, and it's worth hearing why. In §5 they lay out a spectrum: at one end, a model that truly learns a task “from scratch” at inference; at the other, a model that merely recognizes, from your examples, a task it already absorbed during pretraining. Translation is surely learned during training; the scramble tasks look genuinely learned on the spot; most tasks sit somewhere unknowable in between. They even note we can't say which of these humans do. One boundary worth marking on our running example, too: no model can reverse a word (0.44% few-shot at 175B). GPT-3 reads BPE tokens — chunks averaging about 0.7 words — so single letters are barely visible to it; character surgery is a genuinely foreign coordinate system, and removing inserted symbols is the mildest version of it.

The garbled words were our lab probe — invented, controlled, unciteable. Time to let the probe go and ask the 2020 reviewer's question: on the field's standard benchmarks, does a prompted generalist survive contact with fine-tuned specialists?

Remember what's being compared. Every gray bar below is a model trained on that benchmark's own training set. Every red bar is the same frozen GPT-3 with a few dozen examples pasted into its prompt. The red bars have no right to be this close:

Two honest wrinkles before moving on. Few-shot isn't monotone magic: on LAMBADA, one example is actually worse than none (72.5 vs 76.2) — a single demonstration lets the model recognize a fill-in-blank format but not lock onto it, and only several examples make the format click. And one headline win comes with an asterisk we'll explain in Step 10: PIQA's 82.8 beat the fine-tuned SOTA, but its test items partly overlap the training crawl.

Benchmarks measure prepared tasks. The paper also built a battery nothing could prepare for — quick computations and adaptations where the answer must be produced, not retrieved.

Start with plain arithmetic, asked in words: Q: What is 48 plus 76? A:. Before you reveal the right-hand cell below, commit to a guess. Few-shot, two-digit addition: the 13-billion-parameter model gets 55.5%. The full GPT-3 is 13× bigger — how much better do you expect?

Not 70%. Not 85%. One hundred percent — an ability that is half-formed at 13B and simply present at 175B. Then it decays with digit count like a student running out of scratch paper (the chips above), which already hints this is procedure, not lookup. The paper checks the lookup theory directly: it searched the training data for its test problems and found 17 of 2,000 addition problems, 2 of 2,000 subtractions. And the wrong answers are the giveaway — the model often forgets to carry the 1. You don't make that mistake retrieving a memorized sum.

The same on-the-spot adaptation shows up with words. Define a term that has never existed, once, and ask for a sentence — here is the model's actual output, amber marking what it wrote:

Elsewhere in the same figure it inflects a brand-new verb (“to screeg” → “we screeghed”). And on pre-2005 SAT analogy questions — a genre with its own fussy format — few-shot GPT-3 scores 65.2% against the college applicants' average of 57%.

A model that improvises arithmetic and coins sentences for invented words will also write prose on demand. The authors measured how well people can tell — and then spent a fifth of the paper on what that implies.

Fittingly, the generation setup is itself a few-shot prompt: three real news articles in the context, then a fresh headline and subtitle. Around 80 US participants per model judged mixed batches of human and machine articles. Follow the dots down: detection accuracy slides toward the 50% coin-flip line as the generator grows.

That 52%-versus-86% pair is the finding: people can catch deliberately bad machine text, and essentially cannot catch GPT-3's. The paper treats this as a milestone to be managed, not celebrated, and §6 reads like a risk report from inside the lab:

Every score on this page assumed one thing: that the test questions weren't already in those 300 billion training tokens. On a web-scale corpus that assumption cannot be waved through — so the authors audited it, in public.

For every benchmark they built a “clean” subset — items sharing no 13-gram with the training data — and re-scored on it. Most results barely moved. Where they did move, the paper says so: PIQA and Winograd wear asterisks in its own tables. It also admits, in print, that a bug left some overlaps unfiltered and retraining was too expensive to fix it. Even our garbled-word probe makes a cameo: its flagged “overlaps” turned out to be mostly false alarms — palindromes like kayak = kayak, which look identical scrambled or not. LAMBADA overlapped substantially, yet the clean subset scored within 0.5% — the fill-in-blank framing defeats simple recitation.

The limitations chapter is just as plain-spoken. GPT-3's long text repeats itself and drifts off-topic; it flubs common-sense physics (“If I put cheese into the fridge, will it melt?”); it stays weak wherever two snippets must be compared, which §5 pins — as a conjecture — on one-directional reading. Its pretraining is spectacularly data-hungry next to a human childhood. Inference on a 175B model is expensive. And the deepest open question is the one we parked in Step 3: recognition or learning? The authors answer with the sentence quoted below — by refusing to choose.

Before the last bookend, close the loop yourself. Try each question out loud, then check: