Performance is a straight line — if you plot it right.
Before 2020, building a better language model felt like a craft: tune the depth, the width, the number of attention heads, try architectural tricks. This paper measured something cleaner. A model's loss — how surprised it is by held-out text — falls as a smooth power law in each of three scale factors: model size, dataset size and training compute. On log-log axes that power law is a straight line, unbent across more than seven orders of magnitude.
And the architecture's exact shape — depth versus width, head count, feed-forward ratio — barely moves that line. Performance depends strongly on scale, and only weakly on shape. So the lever that matters isn't cleverness; it's how much you scale, and where you spend it.
Plot loss against scale on log-log axes and you get a ruler — one you can extrapolate before you train.
The question
What actually makes a language model better?
Step 1 · The black box & the surprise
A box that predicts the next word — and a number that grades it.
Here is the whole subject in one picture. A decoder-only Transformer language model reads some text and predicts the next token; do that across a passage and you can score it. That score — the loss — will follow us through every step:
The grade is the cross-entropy loss: the average negative log-probability the model assigned to the words that actually came next, measured in nats (natural-log units) and averaged over a 1024-token context. Lower is better — a low loss means the model wasn't surprised.
Here's the surprise the paper delivered. You'd expect a better grade to come from a cleverer design. Instead, quality is set almost entirely by scale — and barely at all by architectural cleverness. The same loss curve appears whether you build a deep-narrow model or a shallow-wide one of the same size.
One number, the loss in nats, summarizes the whole model. The paper's entire story is the shape of how that one number falls when you turn up the scale.
From the abstract: “Loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.”
Step 2 · Three knobs
Everything you can turn up reduces to three knobs.
You've met the loss L — the one number we want to push down. Now: what are you actually allowed to turn up to move it? The paper found that all of scaling collapses to exactly three knobs.
Define each plainly, because every law from here on is written in these three letters:
Knob 1 · model size
N
N — non-embedding parameters. The model's learned weights, counting the Transformer body but not the vocabulary/position embeddings. Excluding those gives much cleaner laws.
Knob 2 · dataset size
D
D — dataset size, in tokens. How much text the model trains on. A token is a sub-word chunk; the paper used a 50,257-word BPE vocabulary on WebText2.
Knob 3 · compute
C
C — training compute, in PF-days. Total work done. One PF-day = 8.64×10¹⁹ floating-point operations. A handy rule: training costs about 6N operations per token (forward + backward).
Three knobs — N, D, C — and they're linked: spending compute C buys you steps on a model of size N over data D, roughly C ≈ 6·N·(tokens seen). The rest of this page is the law each knob obeys.
Setup: decoder-only Transformers on WebText2 (~2.29×10¹⁰ tokens), 1024-token context, Adam, a fixed 2.5×10⁵ steps at batch 512×1024 tokens, 3000-step warmup then cosine decay.
The laws
A straight line · for all three knobs · and why shape doesn't bend it
Step 3 · A straight line on a log-log plot
Double the model, and the loss drops by a fixed percentage. Every time.
You have three knobs. Start with the first — model size N — and watch what the loss does as you turn it up. The answer is almost suspiciously regular.
First, the concrete rule, no formula yet. Take any model and double its size. Its loss is multiplied by about 0.95 — a roughly 5% drop. Double again, another 5%. Make it 8× bigger and the loss is multiplied by about 0.85. The percentage drop per doubling is the same no matter where you start — whether you're at a thousand parameters or a billion.
A relationship where multiplying the input always multiplies the output by the same factor is a power law. Plot it on log-log axes — log of N across, log of L up — and it becomes a perfectly straight line. Drag the slider and ride the point along it:
Drag to change the model size N — watch the point ride the line
×2 params → L × 0.95 (about a 5% drop). ×8 params → L × 0.85. The same percentage every time — that's what makes the line straight, and what lets you extrapolate it.
The point never leaves the line. Slide N over seven decades and the loss keeps falling at the same gentle, fixed rate — the hallmark of a power law. Values shown are illustrative of the trend.
Now the formula arrives — and it should feel like relief, not new difficulty. It just writes the line you've been dragging in symbols:
L(N)=(NcN)αN
αN ≈ 0.076 · Nc ≈ 8.8×10¹³ params · (the slope is what matters; the constant Nc just depends on the tokenizer)
That tiny exponent αN ≈ 0.076 is the doubling rule: doubling N multiplies the loss by 2−0.076 ≈ 0.95. A small exponent means a gentle slope — but a slope that never flattens out across the whole studied range.
Checking the doubling rule: 2−0.076 ≈ 0.949 (≈ 5% drop); 8−0.076 = 2−3·0.076 = 2−0.228 ≈ 0.85.
Step 4 · Same story for data and compute
Three knobs, three power laws — three parallel straight lines.
You watched the loss fall as a straight line in model size N. The quietly remarkable part: the other two knobs obey the very same shape. When a model isn't bottlenecked by the other two factors, each knob gets its own power law.
Same equation, different letters and slopes. Here they are side by side — note the tiny exponents, and that all three lines slope gently down:
with model size
L(N) = (Nc/N)αN
αN ≈ 0.076 · Nc ≈ 8.8×10¹³
Limited params, trained to convergence on plenty of data.
with data
L(D) = (Dc/D)αD
αD ≈ 0.095 · Dc ≈ 5.4×10¹³
Large model, limited data, with early stopping to cut off overfitting.
with compute
L(Cmin) = (Cc/Cmin)αC
αC ≈ 0.050 · Cc ≈ 3.1×10⁸
Compute spent optimally — the gentlest slope of the three.
A small exponent (0.05–0.10) means each doubling buys a modest but reliable gain — diminishing returns, yes, but never-ending and perfectly predictable. No floor appears within the studied range. You keep scaling, the loss keeps sliding.
The exponents are the robust, transferable part. The constants Nc, Dc, Cc depend on the vocabulary and tokenization and carry no fundamental meaning of their own.
Step 5 · Shape barely matters
Reshape the model however you like — the loss hardly notices.
The three laws all care about how much you scale. So here's the natural worry: surely how you arrange those parameters — deep or wide, more heads or fewer — matters too? The paper's answer: barely.
Hold the total non-embedding size N fixed and rearrange it. Vary the aspect ratio (depth vs. width) by as much as 40×, change the feed-forward ratio, change the attention-head dimension — and the loss moves only a few percent. Two wildly different shapes at the same N land in almost the same place:
deep & narrow
(nlayer, dmodel) = (48, 1600)
loss ≈ L
same total N
same parameters,
different arrangement
≈ equal
shallow & wide
(nlayer, dmodel) = (6, 4288)
loss ≈ L
Scale N; don't fiddle with shape. The architecture is effectively settled, and the only lever that reliably moves the loss is how big you go. That's exactly why the core idea said: depends strongly on scale, weakly on shape.
From Figure 5: at fixed N, a (6, 4288) model lands within ~3% of the (48, 1600) model. Number of attention heads has a similarly minimal effect across a wide range.
Step 6 · Bigger models learn faster
The big model is the cheap one — it gets there in fewer steps.
So scale is the lever and shape is settled. But surely a bigger model costs more to train to any given quality? Here's the counterintuitive twist that turns out to be central.
Bigger models are more sample-efficient: they reach a given loss after seeing fewer tokens and taking fewer optimization steps than smaller models. Each parameter update extracts more from the data. So the large model isn't the expensive option — for a target loss, it's frequently the frugal one.
“Bigger models are wasteful” is the intuition this paper overturned. The opposite holds: spend parameters generously, and you spend data and steps sparingly. That single fact sets up the whole payoff in Part C.
Training curves themselves follow predictable power laws roughly independent of model size — you can extrapolate the early part of a curve to predict the converged loss: L(N, Smin) = (Nc/N)αN + (Sc/Smin)αS, with Sc ≈ 2.1×10³, αS ≈ 0.76.
The payoff
Predict overfitting · spend a budget · and why it mattered
Step 7 · Overfitting is predictable
Grow the model and the data together, by a known ratio.
Bigger models learn faster — but if you keep growing N while the data D stays fixed, something has to give. That something is overfitting, and the paper made even that predictable.
Overfitting is when a model starts memorizing its training text instead of learning the pattern, so its loss on new text stops improving. The paper found the overfitting penalty depends almost entirely on a single ratio:
penalty ∝N0.74D
keep this ratio in check and the loss keeps following the clean law
N and D fold into a single equation — L(N, D) = [ (Nc/N)αN/αD + Dc/D ]αD — so you can predict performance for any pairing, not just the bottlenecked extremes.
Combined-fit constants: αN = 0.076, αD = 0.103, Nc = 6.4×10¹³, Dc = 1.8×10¹³. “Safe” means within the ~0.02-nat loss variation between random seeds (Figure 9).
Step 8 · How to spend a compute budget
Got 10× more compute? Pour almost all of it into a bigger model.
You can now predict the loss and the overfitting. The final question is the practical one a lab actually faces: given a fixed pile of compute C, where do you spend it? This is the paper's most famous answer.
When you're limited only by compute, the optimum is to train a very large model and stop well short of convergence. As the budget C grows, the optimal split is lopsided: most goes to model size N, a little to batch size, and almost none to more serial steps. Click to multiply the budget and watch where it lands:
Add compute and watch the optimal split — bars are relative growth, log scale
Compute ×1 — the starting point. Add compute to see the optimal split.
Spend compute on a bigger model; stop well before convergence. Serial training time (steps) barely grows, so wall-clock stays manageable even as the model balloons. Bars use a log scale so the lopsided split is visible.
Very large model · modest data · stop early. Maximally compute-efficient training is far more sample-efficient than training a small model to convergence — and you have slack: any model within 0.6×–2.2× of the optimal size costs only ~20% more compute.
Optimal allocation (Eqs 1.7): N ∝ C0.73, B ∝ C0.24, S ∝ C0.03, and data D = B·S ∝ C0.27 — so a 10× budget gives N≈5.4×, B≈1.7×, S≈1.07×, D≈1.9×.
Step 9 · Why it mattered & recap
The landscape — and can you now explain it?
You've seen the straight line, the three laws, why shape doesn't matter, and how to spend a budget. Final question: how solid is all this, and have you actually got it?
Report card · the empirical landscape
Range spanned · power laws unbent
Model size N · non-embedding
Dataset size D · tokens
prior intuition · “big data wins”
Now you can explain it. Five questions — answer each out loud before opening it. If all five come easily, you've genuinely got this paper.
What is a power law here, in plain words?
Multiply the scale by some factor and the loss is multiplied by a fixed amount — e.g. double N and loss × ≈0.95, every time. On log-log axes that's a straight line, which stayed straight across more than seven orders of magnitude.
Why does architecture shape barely matter?
At a fixed total size N, rearranging the model — deep vs. wide (up to 40×), head count, feed-forward ratio — moves the loss only a few percent. The loss is set by how many parameters you have, not how you stack them.
Why train a huge model and stop early?
Under a fixed compute budget, the optimum puts most compute into model size (N ∝ C0.73) and stops short of convergence. A big model trained briefly reaches a lower loss than a small model trained to the end on the same compute.
Why are bigger models more sample-efficient?
Empirically, larger models extract more from each token and each update: they hit a target loss after seeing fewer tokens and taking fewer optimization steps than smaller models (Figures 2, 4).
What predicts overfitting?
The single ratio N0.74/D. Grow N and D in tandem and the clean law holds; to stay safe, D ≳ (5×10³)·N0.74 — so an 8× bigger model needs only ~5× more data.
What happened next — these laws turned “scale” into an engineering plan: predict the loss with the law, then budget the compute, then train. That recipe — very large models, modest-by-comparison data, stop before convergence — is the one GPT-3 followed the same year.
“Model performance depends most strongly on scale, which consists of three factors: the number of model parameters N, the size of the dataset D, and the amount of compute C used for training.” — Kaplan, McCandlish et al., 2020
This is the blueprint for the LLM era.
Before this paper, scaling a model was a hunch. After it, scaling was a plan you could draw on paper and predict — and that plan reshaped how the whole field spends its money.
GPT-3 and after
Scale-first training — very large models, modest data, stop before convergence — became the standard recipe, directly motivating models like GPT-3.
Compute as a plan
Teams began budgeting compute and predicting the final loss before training — turning a costly gamble into engineering.
Chinchilla
A later follow-up (Hoffmann et al.) rebalanced the optimal split toward more data per parameter — a refinement of, not a break from, the scaling-law idea.
Every frontier model now estimates loss and budgets compute with scaling laws before spending it — alchemy turned into arithmetic.