DeepSeek-R1

One problem, one bare template, one long response. With the human teachers deleted, the next question writes itself: who decides what a good response is?

A rulebook does — two rules, equally weighted. The accuracy reward extracts the final answer (for math, the expression inside \boxed{}, parsed with SymPy) and compares it against the ground truth: our problem pays out only if the box says 211. For code problems, a compiler runs the program against test cases instead. The format reward checks that the thinking actually sits inside <think>…</think> tags, which keeps the scratchpad separable from the answer.

Notice what's missing from the dashed box: a reward model. InstructGPT taught a neural network to imitate human preferences; R1-Zero refuses that for reasoning, and the paper is blunt about why. A learned judge can be flattered. In one of their own later experiments (we'll meet it again in Step 9's fine print), a model trained against a neural reward watched its reward climb while its real Codeforces performance fell — the policy had learned to please the judge instead of solving problems. A rule can't be pleased.

A response now earns r = 2 when both the box and the tags check out, and there is nobody to charm. What remains is the machine that turns those scores into learning.

Here is the whole loop at cruising altitude. Our AIME problem goes in as the question q. The policy — the model being trained — samples 16 different attempts at it (temperature 1, so they genuinely differ). The rulebook scores each one. Then a box we haven't opened yet, GRPO (Group Relative Policy Optimization), converts sixteen scores into one adjustment of the weights. Repeat 10,400 times over a pool of 26K math questions and friends.

A few numbers to anchor the scale. Each attempt may run to 32,768 tokens of thinking (raised to 65,536 late in training). The learning rate is a cautious 3·10⁻⁶, and a small KL leash (coefficient 0.001) ties the policy to a reference copy that is refreshed every 400 steps — the InstructGPT idea, kept but loosened. Total: 10,400 steps, about 1.6 passes over the question pool, roughly 198 hours on 512 H800 GPUs.

The one box in this picture we haven't opened is GRPO — the part that decides how much each of the sixteen attempts should change the model. That box is Part B, and it's where this paper earns its keep.

Sixteen scored attempts at the b-eautiful problem go into GRPO and a weight update comes out. Before any algebra, let's settle the question GRPO exists to answer: compared to what is an attempt good?

A raw reward of 1 means nothing by itself. If the model already solves this problem nine times out of ten, another correct answer teaches almost nothing; if it nearly always fails, that same 1 is precious. RL algorithms handle this by subtracting a baseline — an estimate of the reward you'd expect anyway — and only pushing on the difference.

PPO, the incumbent, builds that baseline with a second neural network: a value model trained to predict expected reward at every point in the text. For a 671B policy that means training a second network of roughly the same size — double the memory — and here it gets worse: judging a half-written chain of thought means predicting whether the model will rescue itself five thousand tokens later, through a reflection it hasn't written yet. The paper argues this is close to unlearnable for long reasoning (supplementary A.3).

GRPO's answer is the one your math teacher used: grade on a curve. You already sampled sixteen attempts at the same question — so let the group grade itself. The baseline is simply the group's mean reward. Beat your classmates, get pushed up; lag them, get pushed down. The value model is deleted outright.

Find the coral box in the figure's top row — that is the expense in question. Now scan the bottom row: nothing sits in its place, only the sixteen attempts and two summary statistics.

“Beat the group mean, get pushed up” is the idea; the advantage A_i is the number. Let's earn it with rewards small enough to hold in one hand.

Take a group of just four attempts at our problem and score them with the accuracy rule alone: r = [1, 0, 0, 1] (attempts 1 and 4 boxed 211). Three moves, no calculator:

Why bother dividing by the spread? Because the same trick must work across 26K questions at once. Without the division, a question where the model is torn 50/50 would send much louder gradients than one it nearly always gets right, and a handful of questions would dominate training. The z-score puts every question's group on one common scale — a fairness device, not a mathematical flourish.

Now play with a full group of eight. The interesting behavior is what happens when success is rare:

Each of the eight attempts now carries one number: +1.73 if it boxed 211, −0.58 if it didn't. This is where most write-ups stop — and honestly, where we got stuck reading the paper: how does one number attached to a finished answer reach back and change 671 billion weights?

The key is to remember what an answer physically is. From the GPT-2 page: a language model writes one token at a time, each drawn from a probability distribution. Attempt o₁ is not a blob — it is a chain of thousands of individual token choices, and every choice has a probability the weights control. GRPO's move: use A_i as the learning signal for every token choice inside attempt i. The update direction is A_i · ∇ log π(o_i) — nudge the weights so each token of an above-average attempt becomes more probable, each token of a below-average attempt less.

The teal arrows in the top row are long because +1.73 is a big advantage; the coral arrows under the failed attempt are stubs because −0.58 is a small one. To see an actual number move, let's shrink the model to one knob. Suppose the only choice that matters is the final box: the model writes 211 with probability π and something else with probability 1 − π, and a single logit z controls it, π = σ(z). The real model adjusts billions of weights across every token — one knob versus billions is the honest boundary of this toy — but the direction and arithmetic of the update are exactly GRPO's.

One update, and the chance of writing 211 rose from 25% to 34%. Notice that the six failures helped: pushing down wrong answers shoves probability toward everything else, the right answer included. Now run the loop yourself and watch where it goes — and where it stalls:

Two stalls are worth hunting for. Drag π₀ below ~6% and the idealized group contains zero successes — flat rewards, zero spread, no gradient. The model never improves on a question it never solves, which is why this recipe demands a base model already strong enough to succeed sometimes. And once π climbs near 1, every attempt is correct and the question goes silent the other way: nothing left to learn here; in the real run, 25,999 other questions are waiting.

All that remains is to write down what you just did. For each question q, sampling a group {o₁ … o₁₆} from the old policy:

Every part of this line is now something you've touched. A_i at the center is the z-score; the gradient of ρᵢ·Aᵢ is the per-token push you just stepped through (at the moment of sampling ρᵢ = 1, which is why our toy could ignore it). The rest is seat-belts: the clip stops any one batch from moving a token's probability too far, and the KL term is the familiar leash to a reference model.

That update — push up what beat its group, push down what lagged — now runs 10,400 times. Watch two dials at once while it runs: the score on AIME 2024, and how long the model chooses to think.

Both climb together. Accuracy rises from 15.6% — roughly 2.3 of AIME's 15 problems — to 77.9%, sailing past the average human competitor (37.8%) with no reasoning example ever shown. And the average response stretches from a few hundred tokens to roughly 16,000. Re-read Step 2's rulebook: nothing in it mentions length. Long, self-checking attempts simply land in the correct pile more often, so every token of those habits keeps getting pushed up. Thinking time is a side effect with no line in the reward.

Follow the teal dot and you'll feel the two-act structure: a long steady grind for 8,000 steps, then a second wind once the model is allowed — and inclined — to think in 60,000-token stretches. Majority-voting 16 samples pushes the final model to 86.7%, comfortably above the human average on a competition built for the best high-school mathematicians in America.

The length curve says the model chose to think longer. The transcripts show what the extra tokens were spent on.

Midway through training, an intermediate checkpoint produced this, on an algebra problem (we've trimmed the routine lines; the re-derivation runs on for pages):

Nobody wrote “pause and re-derive” into any template, rule, or dataset — check Steps 1 and 2 again if you doubt it. The researchers went counting: across training, reflective words (“wait”,“verify”,“mistake”,“check”…) grow 5- to 7-fold. And the word “wait” specifically follows a schedule — nearly absent before step 4,000, occasional through 7,000, then erupting after 8,000, right where the accuracy curve found its second wind.

So where did the habit come from? The paper's own answer has two honest halves. First, the raw material was already there: V3-Base's pre-training web text is rich in worked math and code — including, the authors note, solution text generated by other models — so fragments of “wait, let me re-check” existed at low probability from day one. Second, RL made those fragments profitable. Attempts that happened to re-check landed in the correct pile more often, collected positive advantages, and Step 6's machinery pushed up every token of the habit. What looks like a lesson learned is selection working at scale.

The same logic explains a failure the authors report openly: they first tried this recipe on a 7B dense model and a 16B MoE. Responses grew longer but degenerated into repetition, and AIME barely moved. A base model that never stumbles onto success gives the z-score nothing to amplify — the Step 5 edge case, at model scale. Emergence here is real, but it is not free.

Pure RL grew a strong reasoner with bad manners. R1-Zero mixes English and Chinese mid-thought and formats its work carelessly — the grader never charged a single point for any of that.

So the team ran the experiment again, this time engineering for a usable product. Four stages, alternating the two tools this page has kept apart on purpose — SFT for style, RL for skill:

Two details repay attention. Stage 3 does not continue training Dev-2 — it restarts from a clean V3-Base, using Dev-2 only as the author of its textbook. The skill survives the reset because it lives in the 800K traces. And the honest wrinkle: watch AIME across the checkpoints below. The cold start initially costs 19 points — a few thousand polite examples dilute raw problem-solving — and RL wins it back with interest.

Four stages later the manners are fixed. Time to grade the graduate — on the same benchmark we've tracked since the Step 7 curves.

Where it trails, we should say so: GPQA Diamond (PhD-level science) reads 71.5 against o1's 75.7, and on the engineering-flavored Aider benchmark o1 leads 61.7 to 53.3 — software-engineering tasks got little RL because their test suites are slow to run in the loop. The headline is narrower and more interesting than “beats o1”: on checkable reasoning, the open recipe caught the closed one.

One expensive model now writes reasoning traces worth studying — the same 800K samples that retrained V3-Base in Step 9. The cheapest thing to do with a good textbook is hand it to more students.

The team fine-tuned six open models — Qwen and Llama, 1.5B to 70B — on those 800K traces. Plain SFT, no RL at all. The smallest student embarrasses much larger closed models on math:

The controlled question hiding here: is copying really better than learning first-hand? Run Part B's whole loop directly on Qwen2.5-32B and you get 47.0 on AIME — respectable, level with QwQ-32B-Preview. Distill R1 into the identical base and you get 72.6. The comparison is stark in a bar:

The reading the paper gives: patterns a 671B model discovers transfer down cheaply, while a 32B model exploring alone hits the Step 5 wall — too few successes on hard problems to amplify. Discovery seems to need scale; the discovered patterns don't.

Everything above is the branch that worked. The authors also published the branches that didn't, and they teach as much as the trunk.

And the standing limits, in the authors' own accounting: rule rewards only reach domains with answer keys, so open-ended writing still depends on the hackable kind of judge, used sparingly; the model overthinks simple questions; languages beyond English and Chinese trigger mixing; and few-shot prompting actually hurts it — ask plainly, zero-shot.

Before the bookend, six questions. If you can answer them without scrolling up, the machinery is yours: