DeepSeek-V3

So the machine somehow keeps 634 billion parameters asleep per token. To see where the sleeping ones live — and which parts of the report touch which — we need the floor plan.

V3 is a standard Transformer stack, 61 blocks tall. Every block asks the same two questions about our sentence. First: what context matters here? — that's the attention room, where derivative goes looking for x². Second: which specialists should process this token? — that's the expert hall, where a small router picks 8 of 256 specialist FFNs (plus one shared generalist that serves everybody).

In the zoomed block on the right of the figure, note that the router sits in front of the expert row like a maître d'. Almost everything this report is famous for happens at that little desk.

One honest note before we zoom in: MLA and the expert layout are inherited from DeepSeek-V2 (2024), where they were first validated. What this report adds on top is the balancing trick, the extra prediction head, and the systems work — the four tags above. We'll take the rooms in order.

The floor plan showed two rooms per block. The expert hall holds the parameters; the attention room holds a different cost — memory that grows with every token you've already read.

When a model generates, each new token attends over all previous ones, so it keeps their attention keys and values (K and V) in a cache. With V3's dimensions — 128 heads, 128 numbers per head — a vanilla Transformer would store 2 × 128 × 128 = 32,768 numbers per token, per layer. At a 128K-token context that snowballs into hundreds of gigabytes.

Multi-head Latent Attention (MLA) refuses to store K and V at all. It squeezes each token's hidden state down to a latent vector of 512 numbers, caches that, and re-expands it into full keys and values only at the moment of use. One extra 64-number vector carries the token's position (its RoPE key, shared across heads). In the figure, everything inside the dashed cinnabar box is the entire per-layer memory footprint of derivative: 512 + 64 = 576 numbers.

The ratio is worth checking yourself: 2 × 128 × 128 = 32,768 raw numbers versus 512 + 64 = 576 cached ones — a factor of 56.9. Slide the context length below and watch what that factor does to real memory. Remember the model weights alone occupy roughly 671 GB in 8-bit form, and the cache comes on top, per sequence being served.

One question we'll leave at the door: how can attention run on compressed latents without wastefully re-expanding every cached token? The answer is a neat piece of algebra — the up-projection matrices can be absorbed into the query and output projections — and it lives in the DeepSeek-V2 paper (§2.1 there). For this page, "cache 576, expand on demand, lose nothing measurable" is all we need.

MLA just cut attention's memory 57-fold. Now for the expert hall — the room holding roughly 97% of V3's parameters — and the desk that decides which of them wake up.

The router owns one learned vector per expert — think of it as the expert's job description. For each token it computes an affinity score: the dot product of the token's hidden state with each job description, squashed through a sigmoid so it lands between 0 and 1. High score means "this looks like my kind of token".

The real model scores 256 experts and keeps 8. That's too many arrows to draw, so from here on we shrink the hall to four stand-in experts, keep the top 2 — same rules, same math, napkin-sized. Watch derivative get scored below: the math expert E-M wants it badly (0.90), the general expert E-G shrugs (0.55), and the two dashed outlines mark the winning shortlist.

Here is the full routing computation, small enough to redo on paper:

The router sent derivative to E-M with a 0.90. Nothing stops it from sending x², 2x, and every other math token there too — and in our sentence, that is exactly what happens.

Route all six tokens of our sentence with the recipe from Step 4 and count the assignments: E-M gets 4, E-L gets 3, E-G gets 3, E-C gets 2. Twelve slots, so a fair share is 3 each — E-M is already over quota after one math-flavored sentence. Now run a math-heavy batch, a million times. Two bad things compound:

The spiral. Whoever wins a token receives its gradient and gets better at tokens like it, which raises its score, which wins it more tokens. Taken to the limit the router collapses onto a few superstar experts, and most of your 671B parameters become dead weight — trace the circle in the left pane below.

The stragglers. Experts live on different GPUs. If E-M's GPU gets 4 tokens while E-C's gets 2, the busy one finishes last and everyone else waits. At 2,048 GPUs, imbalance is not an aesthetic problem; it is idle silicon you are paying for by the hour.

The standard remedy since 2017 has been an auxiliary loss: add a term to the training objective that grows when expert loads drift apart, scaled by a coefficient α. It works — but look at the right pane: the balance gradient flows into the same router and expert weights that are trying to predict "2x". Set α small and imbalance persists; set it large and you've told a language model to care about queue fairness more than language.

An auxiliary loss buys balance by re-aiming gradients, and pays in model quality. V3's answer: get balance from something that gradients never see.

Give each expert one extra number: a bias b, starting at 0. When the router draws its shortlist, it ranks experts by s + b instead of s. And after every training step, a bookkeeper checks the batch statistics: experts that got more than their fair share have their bias cut by a small constant γ; experts that got less have it raised by γ. That's the entire mechanism — a thermostat bolted onto the router. Overworked experts cool down, idle ones warm up, no loss term anywhere.

Two details carry all the cleverness, so let's run one update by hand before the machine does it for us.

Now the part that stopped us cold on first read. If the bias can overrule the scores — E-C just took a slot from E-G while scoring 0.25 less — hasn't the balancing hand already distorted the model, exactly what we blamed the auxiliary loss for? It took us a few passes through equation (16) to see why not, and the answer is in your simulator panel: after E-C won, the mixing weights were recomputed from the raw scores — 0.90∕1.20 = 0.75 and 0.30∕1.20 = 0.25. The bias decided who; it never touches how much.

So tally what the bias never enters: not the loss (no gradient ever points toward "be uniform"), not the mixing weights (raw s only), not the expert parameters themselves. Its one lever is the shortlist — occasionally you consult the #3 expert instead of #2 and pay a small, local price in fit. Compare that to the auxiliary loss, which re-aims every gradient of the network, all the time. V3 chooses the cheaper distortion.

You've run the thermostat by hand and seen what it never touches. A knob this simple still owes us evidence at scale — what improves, and what secretly breaks?

The paper trains matched pairs: same data, same architecture, one model balanced by auxiliary loss and one by the bias controller. At the larger scale tested (228.7B parameters, 578B tokens), the bias version wins most of the board — and, in the interest of honesty, loses one line. MMLU dips by 1.1 points; the paper claims a win on most benchmarks, not all, and the table backs exactly that:

There's a second, quieter payoff. The bias controller balances over the whole batch, not within every sequence — a looser leash. Freed from having to spread every single math document across all experts evenly, experts drift toward genuine specialties. In the paper's Figure 9 the effect is stark; our sketch below copies its shape: under the auxiliary loss (top strip) every expert carries a faint, similar load on every domain, while the bias-balanced model (bottom strip) shows a few hot cells per domain — the math specialists our toy's E-M was standing in for, now real.

The bias controller keeps all 256 experts earning their keep. The next idea squeezes more teaching out of every token the model reads — the second architecture contribution on the report's list.

Normally each position gets one lesson per pass: standing at "is", predict "2x". Multi-Token Prediction (MTP) adds a second assignment: a small bolt-on module — one extra Transformer block, borrowing the main model's embedding table and output head — takes the main model's representation of "is" together with the embedding of "2x" and predicts what comes after that: the "." in our sentence. Same data, nearly double the training signal. Because the module receives the true next token as input, the causal chain stays intact — it predicts two ahead given one ahead, not by guessing blindly.

Below is Step 2's floor plan, one module richer — the dashed box on the right is everything MTP adds. At inference you can simply unbolt it: the main model runs alone, at exactly its usual cost. Or keep it as a draft-writer: it proposes the token after next, the main model verifies, and when the guess sticks you emit two tokens for one pass. The paper measures the guess sticking 85–90% of the time, for 1.8× tokens per second.

Architecture done: compressed attention, balanced experts, a second prediction head. The other half of the $5.576M story is plumbing, and we'll walk it briskly — two ideas, one pane each.

FP8 training. Store and multiply most tensors as 1-byte floats (format E4M3: 4 exponent bits, 3 mantissa bits) instead of 2-byte BF16 — roughly double the throughput on the same silicon, half the memory. The danger is outliers: one huge activation stretches the scale and squashes everything else to mush. V3's counter is fine-grained scaling — every 1×128 tile of activations and every 128×128 block of weights gets its own scale factor (left pane) — plus a precision patch: H800 tensor cores accumulate at only ~14 bits, so every 128 multiplies the partial sums are promoted to full FP32 registers. If that last sentence felt dense, keep just this: the loss curve of the FP8 model stays within 0.25% of the BF16 one — inside run-to-run noise, validated to about a trillion tokens. Nobody had shown that at this scale before.

DualPipe. Experts live spread across 8-GPU nodes, so routing tokens to them is an all-to-all network storm — naively, compute and communication take about equal time, meaning half your cluster-hours would go to waiting. DualPipe feeds micro-batches into the pipeline from both ends and schedules each forward chunk so its network phase runs underneath another chunk's compute phase (right pane). Only 20 of each GPU's 132 streaming multiprocessors babysit communication; tokens travel to at most 4 nodes.

We are folding whole subsections here — the paper spends six pages on custom communication kernels, recomputation tricks, and online quantization (§3.2–3.3). The honest one-line summary: none of it changes what the model computes; all of it changes what an H800-hour costs.

FP8 halves the arithmetic cost; DualPipe stops the cluster from waiting on its own network. Time to add the whole thing up — Table 1 of the paper, redrawn.

The bar below is the entire official training run. Almost all of it is pre-training on 14.8T tokens; the thin band is the long-context extension to 128K, and the sliver on the right — squint — is all of post-training, SFT and RL together. That last one deserves a pause: turning the base model into a chat model cost about 0.2% of the budget.

Read the number the way the authors themselves qualify it: this is the cost of the final, official run only — "excluding the costs associated with prior research and ablation experiments on architectures, algorithms, or data." The exploration that found these settings, the V2 lineage this builds on, and the internal R1-style model that generated reasoning data for post-training all sit outside the invoice. $5.576M is the marginal cost of reproducing the recipe, not the cost of discovering it — still a number that reset the industry's intuitions.

A cheap model is only news if it is also a good one. Our running sentence retires here; the paper's own benchmark tables take the stand.

Two versions get graded. The base model comes out of pre-training as, in the paper's words, "the strongest open-source base model currently available", ahead of Llama-3.1-405B while activating a tenth of the parameters per token. The chat model — after SFT and RL, including distilled reasoning from an internal DeepSeek-R1 model — is compared against GPT-4o and Claude-3.5-Sonnet directly. Read the card as a profile, not a coronation: it wins math and competitive coding by wide margins, splits knowledge benchmarks, and concedes software-engineering agent tasks (SWE-bench) to Claude.

One caveat the paper itself measures: distilling from the reasoning model lifts scores and lengthens answers — in the V2.5 ablation, MATH-500 jumps 74.6 → 83.2 while average response length doubles. Quality bought partly with verbosity; they tuned the trade-off rather than pretending it away.

Close the loop before you go — six questions, answers folded underneath. Use them to check what stuck: