The model wasn't broken. The objective was.
GPT-3 was trained to do one thing: predict the next token of internet text. That objective built astonishing capability, but it never once asked “is this what the user wanted?” — the paper's word for this mismatch is misaligned. When a model ignores your instruction, invents facts, or turns toxic, it may be doing its actual job perfectly; the job was just the wrong one.
The bet this paper makes: human judgment can become the training signal itself, at a price that rounds to nothing. Hire ~40 people, collect demonstrations and preference rankings, distill those rankings into a reward model, then let reinforcement learning optimize against it. Total cost for the largest model: 60 petaflop/s-days — under 2% of the 3,640 spent pretraining GPT-3.
Labelers preferred the 1.3B InstructGPT over the 175B GPT-3. Alignment bought more than a 100× increase in size.
The gap between predicting and obeying
What GPT-3 actually optimizes — and how you'd even define “do what I meant”
Step 1 · The failure, live
Ask GPT-3 to explain the moon landing — it hands you more homework instead.
If you've been through our GPT-3 page, you know the deal: a 175B model, trained on next-token prediction, that can be coaxed into almost any task with a few examples. The key word is coaxed. Give it a bare instruction and things get strange. Here is the prompt this paper uses in its own Figure 2, and the one we'll keep on the desk for the whole lesson:
Look at cards A and B on the right: the model didn't answer the instruction — it continued it, riffing out more “Explain X to a 6 year old” exercises. Card C changes the subject to a bare fact. Only D even attempts the task.
Here's the thing: this isn't stupidity. On internet text, a lone instruction most plausibly sits inside a worksheet or a list of writing exercises, so “another exercise” genuinely is a likely next token sequence. The model is answering the question it was trained on — what text comes next? — not the question you asked. The paper says the objective is misaligned: predicting webpages and “follow the user's instructions helpfully and safely” are two different targets that only sometimes coincide.
You could patch around this with few-shot prompting — and we'll see in Step 9 that a carefully engineered prompt does help GPT-3 a lot. But the paper's move is more direct: change what the model is trained to do.
The A–D samples are the paper's own condensed illustrations (Figure 2), not full transcripts; the deployed data came from real API traffic. Also worth saying: GPT-3 with a good few-shot prefix behaves far better than bare GPT-3 — the paper evaluates both, and both lose to what's coming.
Step 2 · Defining the target
“Helpful, truthful, harmless” — rewritten as a checklist forty people could apply.
Step 1 showed the objective aiming at the wrong target. Before any training trick can help, someone has to write down what the right target is — concretely enough that strangers agree on it.
The paper wants models that are helpful (solve the user's task), honest (don't fabricate), and harmless (no physical, psychological or social harm). Nice words; the real work is the labeling instructions that turn them into judgments. A few lines from the actual instruction sheet in the paper's appendix — each pane below quotes the flavor of one criterion:
Helpful
Follow the user's intent, not just the literal text: “answering the question they meant to ask, even if they mis-asked it”. Write clearly. No rambling, no invented context.
Truthful
On summarization-style tasks, use only what's in the input. Don't state clearly false things; refute a question's false premise rather than dodging with “it's not totally clear”.
Harmless
No denigrating language, no violence, no sexual content that wasn't asked for, no bad real-world advice. Treat people with respect and consideration.
Criteria collide, of course — a maximally helpful answer to a dangerous request is not harmless. The sheet even gives a tie-breaker for borderline cases: “which output would you rather receive from a customer assistant?” During training data collection, helpfulness was ranked first; for the final evaluations, truthfulness and harmlessness took priority.
Who applies this checklist? About 40 contractors, hired through Upwork and Scale AI — screened with a test for spotting sensitive content and for agreement with the researchers' own judgments (soft cutoffs: 75% agreement, 6/7 on writing demonstrations). Even after screening, labelers agree with each other only 72.6 ± 1.5% of the time. Hold that number: everything downstream is built on judgments that genuinely differ, and Step 12 comes back to what that means.
“Alignment” stops being philosophy the moment it becomes a labeling instruction sheet. The model will end up aligned to these forty people, following this sheet — not to “human values” in the abstract. The paper is unusually honest about this, and we'll weigh it in Step 12.
Step 3 · The pipeline from orbit
Three stages: show it, score it, then train against the score.
We now have a target (the checklist) and judges (the labelers). The engineering question: 40 people can produce maybe tens of thousands of judgments — how do you steer 175 billion parameters with that?
The answer is a three-stage relay, where each stage squeezes more model improvement out of a cheaper kind of human work. Here's the whole machine at low zoom — we'll spend all of Part B inside these boxes:
Read the caption strip along the bottom of the figure: the human effort per data point falls stage by stage — write a full answer, then merely drag cards into an order, then nothing at all. Meanwhile the number of model updates grows: stage 3 runs 256k practice episodes against the learned scorer. That asymmetry is the whole economic argument of the paper.
Where do the prompts come from? Mostly from real users: text submitted to early InstructGPT models on the OpenAI API Playground (with notice that it could be used for training, deduplicated, PII-filtered), plus prompts the labelers wrote to bootstrap the very first model. It's a working distribution, not an academic one — 45.6% open-ended generation, 12.4% open QA, 11.2% brainstorming, and so on down a long tail.
| Stage | Human gives | Prompts | Produces |
|---|---|---|---|
| 1 · SFT | written demonstrations | ~13k | SFT model (teal) |
| 2 · RM | rankings of K = 4–9 outputs | ~33k | reward model (amber) |
| 3 · PPO | nothing — the RM stands in | ~31k | InstructGPT (coral) |
Stage 2 converts expensive, one-shot human time into a reusable artifact: a model of the labelers' preferences. Once that artifact exists, stage 3 can practice on 31k prompts nobody ever answered — and could keep practicing on millions more — paying humans nothing further.
Rankings into a reward, a reward into a policy
Feel each stage · run its arithmetic yourself · then read its formula
Step 4 · Stage 1 — supervised fine-tuning
First, just show it: thirteen thousand prompts, answered by hand.
The orbit view promised that stage 1 uses the most expensive human signal — full written answers. Let's watch one get used.
A labeler opens our moon-landing prompt and writes the answer they wish the model had given: “Some people went to the moon, and they took pictures of what they saw, and sent them back to earth so we could all see them…” (that wording is from the paper's Figure 2). Collect ~13k prompts with demonstrations like this, then fine-tune GPT-3 on them with the same next-token objective it was pretrained with.
Notice what changed and what didn't. The loss function is untouched — predict the next token, exactly as on our GPT-1 and GPT-2 pages. Only the text changed: instead of the whole internet, the model now predicts tokens of answers that forty screened people considered helpful, truthful and harmless. Aim the old objective at better text and you get the SFT model — the teal box from Step 3, filled in.
So why isn't the paper done at stage 1? Two reasons, and they motivate everything ahead. First, cost: writing a good answer takes a labeler minutes; we'll see next step that judging answers is much cheaper per data point. Second, ceiling: imitation can at best clone the demonstrators. A model that only imitates never gets feedback on its own outputs — on the weird, model-flavored mistakes no human demonstration anticipates.
An honest wrinkle from §3.5: on validation loss, the SFT model overfits after just 1 epoch — yet training for 16 epochs kept improving human preference ratings anyway. Loss on held-out text and “humans like it” are already, measurably, different targets. Compute cost: 4.9 petaflop/s-days at 175B.
Step 5 · Stage 2a — collecting rankings
Don't ask people to write. Ask them to rank — one screen can yield 36 comparisons.
Stage 1's data cost a written answer per prompt, and imitation caps at the demonstrator. Stage 2 switches to a cheaper question: not “what should the model say?” but “which of these is better?”
The labeling screen shows one prompt and K model
outputs, where K runs from 4 to 9. The labeler rates each output on
a 1–7 scale, flags problems (hallucination? fails the instruction?
unsafe?), and finally drags the outputs into an order, ties
allowed. Our four moon-landing cards from Step 1 are exactly such a
screen with K = 4 — and the paper's Figure 2 shows the
ranking a labeler produced:
The right column of the figure is the payoff: a single ranking of
K outputs contains C(K,2) = K(K−1)/2 pairwise
verdicts. Ranking 4 cards took one drag; it produced 6 pairs. Rank
9 and you get 36. Judging effort grows gently with K while
comparisons grow quadratically — that's why the paper pushed K up
to 9, and why 33k prompts became a much larger pile of
comparisons. Try the arithmetic:
Drag K — how many lessons does one screen teach?
The paper's own ceiling: 64 prompts per batch × C(9,2) = 36 pairs ≤ 2,304 comparisons per batch (§C.2). Slide K down to 4 and watch the same screen yield six times less signal.
Quality control on all this judging: labelers agree with each other 72.6 ± 1.5% of the time (held-out labelers: 77.3 ± 1.3%). Perfect consistency was never on offer; the machinery below has to work with majority tendencies, not ground truth.
Step 6 · Stage 2b — the reward model
Nobody ever wrote down a score — one sigmoid conjures the scores from “D beats C”.
We now hold five comparisons like “D beats C” for the moon prompt, and tens of thousands more. But stage 3 will need a number for a single response, standing alone. Rankings contain no numbers.
This is the step where we slowed down when first reading the paper, so let's take it carefully. The gap is real: labelers only ever said which of two texts is better, never how good either is. The trick is to train a model that outputs a score, and grade the scores by whether their differences predict the human's picks. Get the pairwise predictions right often enough and the scores become a usable ruler — calibrated by nothing but comparisons.
First, the machine itself. Take the 6B SFT model, remove its
final unembedding layer (the one that maps a hidden vector to
50k-ish vocabulary logits — see our GPT-2 page), and bolt on a head
that outputs a single number. Feed it a prompt x plus
a candidate response y, out comes
rθ(x, y) — a scalar reward.
Now the training move, on our own pair. Say the current RM scores response D at 1.6 and response C at 0.4 (toy values — the paper doesn't publish per-example scores). The labeler said D beats C. Run the numbers:
Score both responses
Two forward passes through the RM — same prompt, different response.
Take the gap — only the difference matters
The winner's score minus the loser's. Positive gap = the RM currently agrees with the human.
Squash the gap into a probability
The sigmoid σ(Δ) = 1/(1+e−Δ) maps any gap to (0, 1). Read it as: “the probability the RM assigns to the human picking D over C.”
Loss = − log of that probability
The human did pick D, so the loss is small when σ is near 1 and explodes as it nears 0.
Flip the scores and feel the gradient
Had the RM scored C above D (Δ = −1.2), the loss would be −ln(0.23) = 1.46 — over five times larger. Shrinking it pushes r(D) up and r(C) down. Every pair pulls the two scores apart in the order the human chose.
Drag r(D) — watch the sigmoid grade the reward model
What to watch: around Δ = 0 the amber curve is steep — the loss changes fast, so ambiguous pairs push hard on the weights. Out past Δ ≈ 2 it flattens: once the RM is confidently right, that pair has little left to teach. Drag r(D) below 0.4 and the loss chip jumps — disagreement with a human is what this model feels as pain.
You've now run every part of the reward-model loss by hand. Stacked into a formula — this is Equation 1 of the paper, and each symbol is a step you just did:
loss(θ) = −1C(K,2)E(x, yw, yl)[log σ( rθ(x, yw) − rθ(x, yl) )]
yw = the winner (D) · yl = the loser (C) · rows 1–2 = the subtraction · row 3 = σ · row 4 = −log
The one symbol you haven't met is the 1/C(K,2) out
front, and it hides a war story. The team first shuffled all pairs
from all prompts into one big dataset — and the RM overfit within a
single epoch, because each response reappears in K−1 different
pairs. The fix: keep all pairs from one prompt inside one batch
element, normalized by C(K,2) so 9-output prompts don't shout
louder than 4-output ones. Bonus: each response is now embedded
once, not K−1 times.
One last touch. Since only score differences are trained, the absolute scale floats free — so before RL, a bias is added to center the scores: labeler demonstrations get mean 0. Remember that zero; it matters in the next step.
The RM generalizes past its teachers: cross-validated across labeler groups, it predicts held-out labelers' preferences at 69.6 ± 0.9% — barely below the 72.4 ± 0.4% on its own training group, and close to the labelers' own 72.6% agreement with each other. It learned the panel's shared taste, not one person's quirks.
Step 7 · Stage 3a — the heart of PPO
A score becomes a direction: multiply the reward into the gradient.
The RM can now score any (prompt, response) pair in one forward pass — for free, forever. Stage 3 makes those scores the only teacher: from here on, no human writes or ranks anything.
The setup is the simplest reinforcement learning there is. Draw a prompt from the 31k pile, let the policy (the model being trained, initialized from the SFT model) write one response, have the RM score it, episode over. The paper calls this a bandit environment — one decision, one payoff, no multi-turn game.
And now the question this page refuses to hand-wave, because nearly every summary of RLHF does: the RM says “+0.8”. A score is not a gradient. Which of the policy's billions of weights should move, and in what direction? Follow the dashed coral arrow in the figure back to the policy — here is what actually travels along it.
Shrink the world so we can see it whole. Suppose for our moon prompt the policy chooses between just three replies, from raw preference numbers (logits) z = (1.0, 0.5, −0.5), passed through softmax to make probabilities:
- a — “Explain the theory of gravity to a 6 year old.” (the riff) · π(a) = 0.55
- b — “People went to the moon in a big rocket…” (an answer) · π(b) = 0.33
- c — “The moon is a natural satellite.” (true, useless) · π(c) = 0.12
We sample — it comes up b. The RM scores it +0.8. The
policy-gradient rule then updates the logits as
z ← z + η · r · ∇log π(b), and for a softmax that
gradient has a beautifully readable form:
∇log π(b) = onehot(b) − π = (−0.55, +0.67,
−0.12). In words: raise b's logit, and lower everyone else's in
proportion to their current share — softmax bookkeeping, since
probabilities must still sum to 1.
With step size η = 0.5 and r = +0.8, the logits move by 0.4 × (−0.55, +0.67, −0.12), and the new probabilities come out (0.44, 0.44, 0.12). One judged episode, and the answer's share rose from 33% to 44% — that is, literally, the model learning from feedback. Now drive it yourself:
Drag the reward — one gradient step, recomputed live
Before the update
After one step of z ← z + 0.5 · r · (onehot(b) − π)
r = +0.8 → the sampled answer b climbs: 33% → 44%.
Three things to try. Push r negative: the same arrows flip, and b — the response we happened to sample — gets suppressed instead. Set r = 0: nothing moves at all, which is why the RM was centered so that average behavior scores zero — no push either way. And notice a and c: they shrink or grow without ever being sampled, because softmax shares the probability budget.
The honest boundary: real PPO wraps this heart in plumbing we are not opening here. The response isn't one choice among three but hundreds of tokens, each nudged as above with the reward granted at the end; a learned value function (initialized from the RM) estimates the expected reward so updates push on reward minus expectation rather than raw reward; and a clipping rule stops any single batch from yanking the policy too far. All of it lives in §C.4 and Schulman et al. (2017). The multiply-the-score-into-the-gradient move you just performed is the part that makes it learning from human feedback.
Scale, for the record: 256k episodes over ~31k prompts, batch size 512, and one shared 6B RM + 6B value function grading policies of every size — which is how a 1.3B policy gets exactly the same exam as the 175B one.
Step 8 · Stage 3b — the KL leash
Left alone, the policy learns to fool its examiner — so every token pays a distance fee.
Each nudge from Step 7 makes high-scoring responses more likely. Loop 256k times and a danger appears: the policy doesn't chase quality, it chases the RM's opinion of quality — and those diverge.
The RM is a 6B proxy trained on 33k prompts of human data. The policy gets 256k episodes to probe it for blind spots — phrasings that score high without being anything a labeler would actually prefer. The paper calls this over-optimization of the reward model. Its defense: punish the policy for drifting away from where its training data came from. Every token pays a fee proportional to how much more likely the policy makes it than the frozen SFT model would:
reward = rθ(x, y) − β · log(πRL(y | x)πSFT(y | x))
β = 0.02 · the log-ratio is summed per token over the whole response — that's the leash
β = 0.02 looks like a rounding error. The catch is in the fine print under the formula: the fee is charged per token. Suppose the policy has drifted until its average token is e0.5 ≈ 1.6× more likely under πRL than under πSFT — a log-ratio of 0.5 per token. Over a 100-token response: 0.02 × 100 × 0.5 = 1.0 knocked off the reward. Compare that to the reward gaps of a point or two we saw in Step 6 — the “tiny” leash bites at exactly the scale that matters.
So the policy lives in a tug-of-war: drift buys raw RM score (that's what optimization is for), but drift also runs up the per-token bill. Somewhere in between sits a sweet spot. The chart below makes the trade visible — drag β and watch the sweet spot slide:
Drag β and the drift — where does net reward peak?
What to watch: the dashed amber curve (score gained by drifting) is schematic — the paper doesn't publish it — but the gray fee line is exact arithmetic: β × 100 × d. Raise β to 0.06 and the sweet spot collapses to d* = 0: the leash is so short the policy barely leaves the SFT model. Drop β to 0.005 and it's free to roam — and to hack the RM. β = 0.02 is where the paper parked.
One term remains, and then you've seen the entire training signal. The full objective the paper maximizes (its Equation 2) is:
objective(φ) = E[rθ(x, y) − β · log(πRL(y | x)πSFT(y | x))] + γ · Ex ~ pretrain[log πRL(x)]
term 1: the RM's opinion · term 2: the leash you just priced · term 3: keep predicting pretraining text — an IOU we redeem in Step 11
That third term simply mixes plain old next-token prediction on pretraining data back into the updates, weighted by γ. With γ = 0 the paper calls the model “PPO”; with γ = 27.8 it's “PPO-ptx”, and InstructGPT means PPO-ptx. Why anyone would bolt the pretraining objective back on is a Step 11 story — it fixes a real casualty we haven't met yet. For now, here is the whole machine again, one zoom deeper than Step 3, with everything you've since learned drawn in:
Look at what stage 3 bought that stage 1 could not: the model is now judged on its own outputs, on prompts no human ever answered. The RM's opinion is cheap enough to consult 256k times — human judgment, amortized.
The verdict — and who gets to give it
Preference wins, taxes paid, and the fine print on the word “human”
Step 9 · The headline result
Labelers preferred the 1.3B InstructGPT to a GPT-3 one hundred times its size.
The machine is assembled: an SFT starting point, a frozen examiner, a leash, and a pretraining IOU. Grading time — by the only metric this paper ultimately trusts, fresh human preference.
The protocol: take prompts from held-out API customers (nobody whose data was trained on), generate outputs from each model, and ask labelers which they prefer. Everything is measured as a win rate against a fixed reference — the 175B SFT model, the dashed line at 0.5 in the chart below:
Two readings of the same chart. Vertically: at every size, each stage of the recipe adds a visible jump — prompting helps GPT-3, demonstrations beat prompting, and RL on the reward model beats demonstrations. Horizontally, the punchline: the coral line's leftmost point (1.3B, circled) sits above the gray lines' rightmost points (175B). On this axis — “does it do what I asked?” — a hundred-fold parameter advantage failed to close the gap that alignment training opened.
Report card · 175B InstructGPT vs. 175B baselines, % preferred by labelers
vs. GPT-3 (bare)
vs. GPT-3 + few-shot prompt
vs. FLAN-tuned 175B
vs. T0-tuned 175B
0% of comparisons won →100
A detail we enjoy: the few-shot prefix for “GPT-3 (prompted)” was chosen by an hour-long competition between two of the authors — best RM score won. Even the strongest hand-crafted prompt they could find loses 71% of matchups to fine-tuning on feedback.
Step 10 · Truthfulness & toxicity
Half the hallucinations, less venom on request — and a bias needle that didn't move.
Winning preference votes could just mean “sounds nicer”. Recall from Step 2 that the final evaluations put truthfulness and harmlessness first — so let's check those axes specifically.
Truthfulness. On closed-domain tasks from the API (answer only from the given text — summarization, closed QA), InstructGPT invents unsupported information about half as often as GPT-3. The two bars below are that result; the gap is the paper's cleanest honesty win:
GPT-3 · hallucination rate
InstructGPT · hallucination rate
0closed-domain tasks, API distribution →50%
On the adversarial TruthfulQA benchmark, InstructGPT is truthful and informative about twice as often as GPT-3 — and when an honesty-priming instruction allows “I have no comment”, it prefers being truthfully uncertain to confidently wrong more than the baseline does.
Toxicity is a conditional story. Told to be respectful, InstructGPT produces ~25% fewer toxic completions than GPT-3 (RealToxicityPrompts, both automatic and human evals). With no instruction at all, the advantage evaporates. And told to be toxic, it complies — more effectively than GPT-3. The same obedience that fixes the moon-landing prompt cuts in every direction; the paper names this its “greatest limitation”, and it's worth sitting with rather than smoothing over.
Bias: on Winogender and CrowS-Pairs, no improvement. Following instructions and shedding stereotyped associations are apparently different problems — RLHF as run here solved only the first.
Numbers: hallucinations 21% vs 41% (§1, Figure 4); TruthfulQA ≈ 2× truthful + informative (§4.2); toxicity −25% under “respectful” instructions (§4.2). All three from human evaluation.
Step 11 · The alignment tax
Alignment initially cost benchmark skill — mixing pretraining gradients back in refunded most of it.
Time to redeem Step 8's IOU: why does Equation 2 carry a term that just re-runs pretraining?
Because pure PPO fine-tuning broke things. As the policy chased reward, its scores slid on public NLP benchmarks — SQuADv2 (reading comprehension), DROP (numeric reasoning), HellaSwag, WMT 2015 French→English — relative to plain GPT-3. The paper names this an alignment tax: a capability fee paid for obedience. And it treats the tax as a strategic problem, not a cosmetic one — if aligning a model makes it worse at things people measure, teams will quietly ship the unaligned one.
The fix is the γ term you already saw: for every RL episode, also sample pretraining text (8× more of it than RL prompts) and push the policy to keep predicting it, with γ = 27.8 scaling the gradient. The two pushes are simply added before each update — “get high reward” and “stay good at being a language model”, averaged into one step direction.
Does it work? Largely: the regressions shrink on all the affected benchmarks, SQuADv2 and DROP recover at the tuned γ, and on HellaSwag PPO-ptx even overtakes GPT-3. Not fully, though — DROP, SQuADv2 and translation still trail GPT-3, and the paper says so plainly. They also tried the obvious alternative — just tighten the β leash from Step 8 — and it failed: pushing β up crushed the reward without ever recovering the benchmarks. Distance from the SFT model and forgetting-how-to-language turn out to be different quantities; you leash the first, you rehearse the second.
Step 12 · Aligned — to whom?
The paper's own closing question: whose preferences did we just optimize?
Every result in Part C says the model got better at what its judges valued. Section 5.2 of the paper — unusually, for a paper this triumphant — spends pages asking who those judges actually are.
Trace the chain in the figure below, left to right. Researchers wrote the instruction sheet (Step 2). Roughly 40 hired contractors applied it — a group the paper describes concretely: mostly under 35, majority from the US or Southeast Asia, English-speaking, screened partly for agreeing with the researchers. The prompts came from early API customers, not the general public. Every arrow is a place where “human values” quietly became “these particular humans, on these particular tasks”:
The authors don't claim these were the right people to align to — they claim the machinery works given some chosen group, and that choosing is a separate, unsolved, political problem. With labelers agreeing only ~73% of the time, “the human preference” was always a majority vote, not a fact.
And the aligned model still stumbles in characteristic ways. Feed it a false premise — the paper's example: “Why is it important to eat socks after meditating?” — and it earnestly surveys theories about sock-eating instead of challenging the question. Ask something simple and it may over-hedge, refusing to commit when one answer is clearly right (likely, the authors suspect, because labelers rewarded epistemic humility and the RM learned to overpay for it). Stack several constraints in one instruction and performance degrades. Aligned, yes; finished, no.
That's the full arc. Six questions to close the loop — if the machinery from Part B stuck, the answers should come to you before you open them:
How can 1.3B beat 175B in a human eval?
Because the eval measures alignment, not knowledge. The 175B model knows more but aims worse — it answers “what text comes next?”. The 1.3B model was trained on the actual question, “what did the user want?”. Different axes; size only buys one of them.
Labelers never wrote a score — where do rewards come from?
From the pairwise loss. The RM assigns tentative scalars; the loss −log σ(r(yw) − r(yl)) is small when the winner's score exceeds the loser's. Thousands of pairs pull the scalars into an order that reproduces the rankings — you verified σ(1.2) = 0.77, loss = 0.26 in Step 6.
How does “+0.8” move billions of weights?
The reward multiplies the gradient of the sampled response's log-probability: z ← z + η·r·∇log π(y). Positive reward raises that response's probability (Step 7's 0.33 → 0.44); negative suppresses it; zero does nothing — which is why RM scores are centered.
What stops the policy from gaming the RM?
A per-token KL penalty against the frozen SFT model, β = 0.02 — small per token, but ~1.0 over a 100-token response at moderate drift. It prices distance-from-home so hacking the examiner stops paying. Mitigation, though, not proof — the RM remains a fallible proxy.
Why mix pretraining gradients into RL?
Pure PPO paid an alignment tax — regressions on SQuADv2, DROP, HellaSwag, translation. Adding γ = 27.8 worth of next-token gradient on pretraining text (PPO-ptx) refunded most of it; tightening β instead lost reward without recovering the benchmarks.
Which failures survived alignment?
False premises get accepted (the sock question), simple questions get over-hedged, multi-constraint instructions degrade — and harmful instructions get followed better than before. Obedience was trained; judgment wasn't.
“In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters.” — the abstract, Ouyang et al., 2022
The recipe outlived the model.
Nobody deploys InstructGPT anymore. Everybody deploys its training pipeline: within a year, SFT → reward model → RL had become the standard final mile between a raw language model and a usable assistant.
ChatGPT
Announced as trained with RLHF, “using the same methods as InstructGPT” with tweaks to data collection — this paper's pipeline is what a billion people ended up talking to.
Post-training, everywhere
Claude, Llama, Gemini — model cards differ in details, but demonstrations-then-preferences-then-optimization became the assumed recipe for turning a base model into an assistant.
DPO & successors
Direct Preference Optimization showed the RM + PPO stages can collapse into one classification-style loss on the same rankings — a whole literature of simplifications aimed at this paper's Part B.
Alignment became a training stage you can budget — about 2% of pretraining, by this paper's ledger. “Aligned to whom” stayed a question no budget answers.