DeepSeek-V2

Step 1 promised a model that reads your whole report and answers cheaply. Before trusting any of the discounts, let's compute what “reads your whole report” costs a standard Transformer at answer time.

You already know the attention recipe from the Transformer: each token makes a query, compares it against every other token's key, and uses the resulting weights to mix their values. During generation there's a crucial asymmetry: the keys and values of tokens already read never change. Recomputing them for every new token would be absurd, so every serving system stores them — that store is the KV cache.

Now the bill, with DeepSeek-V2's own shapes. Its attention has 128 heads, each 128 dimensions wide, across 60 layers. Cached the standard multi-head way, one token deposits a key and a value per head per layer:

At two bytes per number (BF16) that's 3.93 MB for a single token. Multiply by the report — 131,072 tokens — and one conversation holds ≈ 515 GB of cache. An entire 8-GPU H800 node carries 640 GB, and the model's weights already want most of it. The picture below is that arithmetic drawn out; watch the bag on the right fill up.

And memory is throughput. A GPU serves many users by batching their conversations; the cache decides how many fit. Halve the cache and you can roughly double the batch — which is why this page keeps talking about memory when the headline you may care about is tokens per second.

The bill is real: ≈ 515 GB of cache for one 128K conversation, on hardware that carries 640 GB per node. DeepSeek-V2's response is not one clever trick but two separate surgeries.

Zoom into the box from Step 1 and you find nothing exotic: 60 identical Transformer blocks, each one an attention module followed by a feed-forward network (FFN), with normalization and residual connections in the usual places. The paper explicitly keeps every “tiny detail” identical to DeepSeek 67B. The novelty is concentrated in exactly two places, and the diagram below marks both — attention is replaced by MLA, and the FFN by DeepSeekMoE.

The two surgeries split the two bills between them. MLA attacks the serving bill — the 515 GB bag from Step 2 — and it is the invention this paper is remembered for, so Part B takes it apart slowly. DeepSeekMoE attacks the training bill — how a model stores 236B parameters while charging 21B per token — and Part C walks it at a brisker pace. If you came here from our DeepSeek-V3 page: this is the origin story of both organs that V3 later scaled up.

Step 3 put the serving bill on attention's desk. Attention already had two well-known diets before this paper — worth one step, because MLA is built to escape their trade-off.

The cache is huge because every head keeps its own keys and values. So the obvious surgery is sharing. Grouped-Query Attention (GQA) makes groups of heads share one key/value pair — 8 groups instead of 128 heads cuts the cache 16×. Multi-Query Attention (MQA) pushes to the limit: all 128 heads share a single key/value pair, a 128× cut. LLaMA-class models shipped with GQA; it works.

But heads are the whole point of multi-head attention — each one watches the sentence differently, and sharing K/V forces them to watch through the same pair of glasses. The paper measures the damage directly, training three otherwise-identical 7B models:

Seven MMLU points between MQA and MHA — that's not a rounding error, that's a smaller model wearing a bigger model's name. So the question DeepSeek-V2 actually asks is sharper than “how do we shrink the cache?” It is: can the cache shrink like MQA while every head keeps its own view, like MHA? The fourth panel below is the answer, and the rest of Part B earns it.

GQA and MQA cut the bag by deleting heads' individual keys and values, and Step 4's table showed the seven-point bill for doing so. MLA wants the small bag and the 128 distinct views.

Here's the observation that makes it possible. A token's 128 keys and 128 values are not 256 independent facts — every one of them is computed, by fixed weight matrices, from the same hidden vector h. They are 32,768 numbers of outfit generated from 5,120 numbers of person. The wardrobe is hugely redundant, and low-rank compression is the standard tool for redundancy: pass h through a skinny down-projection and keep only that.

Concretely — and this is the paper's equation 9 — the token's hidden vector is squeezed to a latent (a compact learned summary): c = W^DKV h, taking 5,120 numbers down to 512. Two up-projection matrices, W^UK and W^UV, can regenerate all 128 heads' keys and values from c whenever they're wanted. So the cache keeps the 512-number note and throws the 32,768-number wardrobe away.

Watch our running example go through it: the token “liability”, sitting at position 91,204 of the report, arrives at some layer as a 5,120-number vector — the long bar at the bottom of the figure. One matrix multiply later it is the short violet note, and only the note enters the dashed bag. The amber and coral rows above it are never stored.

Two honest debts before moving on. First: why should 512 numbers preserve enough of 32,768 to keep quality? The intuition is the redundancy above, but intuition isn't evidence — Step 8 pays this off with the paper's ablation, which is stranger than “quality survives.” Second, and more urgent: regenerating keys and values “whenever they're wanted” should worry you. The model wants them at every generation step, for every past token. Didn't we just trade a memory disaster for a compute disaster?

Step 5 shrank the bag to a 512-number note per token — at the apparent price of re-inflating notes into keys and values for every past token, at every one of the thousands of steps it takes to write an answer. That price would sink the whole idea.

It never gets paid, and the reason is one line of linear algebra. An attention score is a dot product, q·k. Write both sides out: the query is q = W^Qh (from the current token, the one being generated), and the key would be k = W^UKc (regenerated from a past token's cached note). Chain them:

Matrix multiplication is associative: it doesn't care where the parentheses sit. Grouped the left way, you rebuild k for every past token — the compute disaster. Grouped the right way, the two weight matrices merge into one matrix M before serving even starts, and at run time the current token's h is compared directly against the cached notes. The keys are never materialized. Neither are the values — the same move folds W^UV into the output matrix W^O. The paper calls this absorbing one matrix into another.

If that felt too cheap to be legal, you're in good company — we had to write it out with actual numbers before we stopped distrusting it. So let's do exactly that, with 2-dimensional toys. Take W^Q = [[2,0],[1,1]], W^UK = [[1,2],[0,1]], a current token h = (1, 2) and a cached note c = (3, 1). The slow path: q = (2, 3), k = (5, 1), score = 2·5 + 3·1 = 13. The absorbed path: M = W^Q⊤W^UK = [[2,5],[0,1]] (computed once), then Mc = (11, 1) and h·Mc = 1·11 + 2·1 = 13. Same score, and the second path never touched k. Now break it yourself:

The absorption in Step 6 worked because nothing stood between W^Q⊤ and W^UK — two constants, merged once. Position information is precisely the thing that has to stand between them.

Attention as computed so far is order-blind: “liability appears at position 91,204” is information no dot product has seen. DeepSeek models encode order with RoPE — rotary position embedding, which rotates each query and key by an angle proportional to its position, so that a score depends on how far apart two tokens are. Fair warning before the next paragraph: §2.1.3 is the densest spot in this paper, and we went around it three times before the shape of the problem clicked. Here is the slow version.

With RoPE, the score between our current token t and the cached “liability” at position j becomes q^⊤ R_j−t k — a rotation matrix wedged between the query side and the key side, parameterized by the distance j−t. Now try to absorb: the merged matrix would be W^Q⊤ R_j−t W^UK. But t is the token being generated — it changes at every single step, so the distance changes, so the rotation changes, so there is no fixed M to precompute. Matrix multiplication is associative, not commutative: you can't slide the rotation out of the way. The escape routes are all bad — rebuild every past key each step (the compute disaster returns), or cache the rotated keys (the 32,768-number wardrobe returns).

The paper's exit is almost rude in its simplicity: if position jams the compressed channel, stop sending position through it. Split the job in two:

The content channel — everything from Steps 5 and 6, RoPE-free, absorbable, 128 dims per head served from the cached violet note. The position channel — a small side-car that carries RoPE and nothing else: each head gets a 64-dim decoupled query q^R, and the token contributes one 64-dim rotated key k^R shared by all 128 heads. Per head, the two channels just add up inside the softmax:

Why is this cheap? Sharing one rotated key across all heads is an MQA move — but applied only to position, the one signal where Step 4's quality tax has nothing to bite on (a rotation angle is the same fact for every head). The bag gains a single 64-number green strip per token per layer. In the figure, follow the two channels bottom to top: the violet note and the green strip are the only things inside the dashed bag.

Content rides the cached note (Steps 5–6); position rides a shared 64-number strip (Step 7). Time to re-total the bill from Step 2 and audit the paper's headline numbers.

Per token, per layer, the bag now holds 512 + 64 = 576 numbers. Run the same arithmetic you ran in Step 2:

Against the 1,966,080 numbers of Step 2 that is a factor of 56.9 — call it 57×. Our 128K report drops from ≈ 515 GB of cache to ≈ 9.1 GB: from “six GPUs' entire memory for one user” to “a corner of one GPU.” Here is where that lands in the paper's comparison table, with V2's shapes plugged in:

The paper's own summary of that last column: MLA costs the same as GQA would with just 2.25 groups — deep in quality-tax territory for actual GQA — yet outperforms full MHA. That is the IOU from Step 5, and the receipt is Appendix D.2: two MoE models, ~250B parameters, identical except for the attention mechanism, trained on the same 420B tokens:

A 25× smaller cache and higher scores across the board. Why compression would ever help quality, the paper doesn't really explain — low-rank bottlenecks can act as a regularizer, but treat that as our speculation, not the paper's claim. What the paper does own is the measurement.

One audit remains: the abstract says 93.3%, but 34,560 ÷ 1,966,080 says 98.2%. Different comparison. The 93.3% is measured against DeepSeek 67B as actually served — a 95-layer GQA model caching about 195K numbers ≈ 389 KB per token — while deployed V2 also squeezes each cached number to ~6 bits, landing near 26 KB per token. 26 ÷ 389 ≈ 6.7%, hence “reduced by 93.3%.” Same-architecture, same-precision, the honest number is the 57× you computed yourself. Now stress-test it:

MLA settled the serving bill. The other headline — training 42.5% cheaper — comes from the block's other organ, the FFN, which is where nearly all of V2's 236B parameters actually live.

A Mixture-of-Experts (MoE) layer replaces the one big FFN with many smaller FFNs — experts — plus a router that sends each token to a few of them. Parameters stored: all of them. Parameters computed per token: only the chosen few. That's the whole decoupling.

DeepSeekMoE (adopted from the lab's earlier MoE paper) sharpens the recipe in two ways, both visible in the figure. First, fine-grained experts: instead of a handful of giants, each layer has 160 small experts (hidden width just 1,536) and the router picks 6. Picking 6 of 160 gives combinatorially more distinct teams than picking 2 of 8 — sharper specialization from the same parameter budget. Second, 2 shared experts that every token visits unconditionally, so grammar-level common knowledge settles there once instead of being copied into all 160 specialists. When “liability” from our report reaches this layer, the router scores all 160 (a softmax over dot products with each expert's learned centroid), and the output is: input + the 2 shared + the 6 winners, gate-weighted.

Where do “236B stored, 21B active” come from? Back-of-envelope, with rounding: each expert is 3 matrices of 5,120 × 1,536 ≈ 23.6M parameters; kept across 59 MoE layers, one expert-slot ≈ 1.4B. Then 162 experts ≈ 226B stored, plus ≈ 9B of attention machinery and ≈ 1B of embeddings → ≈ 236B. Active per token: 6 routed ≈ 8.4B + 2 shared ≈ 2.8B + the same 9B + 1B → ≈ 21B. The knob that sets the price is K, the number of routed experts per token:

Step 9 handed the router 160 choices per layer. Routers left to themselves develop favorites — and a favorite expert is a failure mode with a name.

Routing collapse: early in training a few experts get slightly better, so the router sends them more tokens, so they get better still, until most experts sit untrained and the “160 specialists” are really five. Sparse compute also has a plumbing problem — experts live spread across 8 GPUs, and a token whose 6 experts sit on 6 different devices pays 6 network hops.

V2's countermeasures, at a brisk walk. Three auxiliary losses added to training: one nudging tokens to spread across experts (weight 0.003), one across devices (0.05), one balancing communication traffic (0.02). Device-limited routing: each token's experts may span at most M = 3 devices — the paper finds M ≥ 3 matches unrestricted routing. And a token-dropping escape valve: when a device still overflows its compute budget, the lowest-affinity tokens are skipped that layer (about 10% of training sequences are exempted so the model also learns the no-drop regime).

Every one of these balance losses is a small tax — gradient spent on logistics instead of language. V2 accepts the tax; a year later V3's aux-loss-free balancing showed how to stop paying it. We fold the three loss formulas here (equations 23–31 in §2.2.3 if you want them); nothing downstream depends on their exact form.

Both organs are priced: attention remembers in 576-number notes, the FFN wakes 21B of 236B. The remaining question is whether the discounted model is actually good.

On MMLU it lands within half a point of LLaMA 3 70B while activating less than a third of the parameters per token — the bars below keep each model's activated count next to its score. Two honest asterisks before you read them. LLaMA 3 saw over four times V2's English data, and on English commonsense and trivia it still wins; V2's edge is the bilingual trade — on Chinese benchmarks it and Qwen1.5 trade blows at the top of the open field. And every economics headline on this page is measured against DeepSeek's own dense 67B, not some universal baseline.

The long-context claim closes a loop from Step 7: because position lives only in the decoupled RoPE strip, stretching the context from 4K to 128K meant adjusting that one channel (with YaRN) and training a further 1,000 steps at 32K — the compressed content channel never had to know. Alignment gets one sentence and a pointer: the base model was instruction-tuned on 1.5M examples, then aligned with GRPO — the critic-free RL method DeepSeek later made famous — and that story is told properly on our DeepSeek-R1 page.

One report, two surgeries, one receipt. Before you close the tab, try to rebuild each piece from memory — peek only after you've committed to an answer.