DeepSeek-V4

You've seen the box win on cost. Before we open it, sit with the problem it solves: where does the bill for long context actually come from?

In standard attention, generating one new token requires comparing its query against the key of every earlier token. Read 1,000 tokens, that's ~1,000 comparisons per new token; read a million, that's a million. The compute per generated token grows linearly with the context already seen — and over a whole sequence that's the famous quadratic cost (work proportional to length squared).

There's a second, sneakier bill: the KV cache. To avoid recomputing the past on every step, the model stores each earlier token's key and value vectors. That store grows one entry per token — so at a million tokens it can swell to tens of gigabytes of fast memory per request, which is what really caps how long a context you can afford to serve.

We just saw both bills are driven by the number of cache entries. So the move is simple to state: have fewer entries.

Instead of keeping one KV entry per token, V4 takes every m consecutive tokens and compresses them into one summary entry, using a small learned, softmax-weighted blend of those tokens. A sequence of length n becomes roughly n / m cache entries. Attention then runs over the compressed entries, not the raw tokens — so both the comparisons and the stored cache shrink by the same factor m.

That's the whole idea. The next two steps are just the two recipes V4 uses to do this compression — one gentle and selective (CSA), one aggressive but complete (HCA) — and why it interleaves both.

Step 3 gave us summary entries. Compressed Sparse Attention (CSA) uses a light compression and then adds a second saving on top: sparsity.

CSA first compresses gently — every m = 4 tokens into one summary (so 1M tokens → 250K summaries). Then a tiny, fast scorer called the lightning indexer ranks those summaries for the current query and a top-k selector keeps only the best k (V4-Pro uses k = 1024; Flash uses 512). Core attention runs over just those k — the rest are skipped entirely. A small sliding window of the most recent raw tokens is added back so nearby detail isn't lost.

CSA compressed lightly and skipped most entries. Heavily Compressed Attention (HCA) takes the opposite trade: compress brutally, but attend to everything that's left.

HCA consolidates every m′ = 128 tokens into a single summary entry — 32× harder than CSA's m = 4. After that crush, so few entries remain that V4 can just run dense attention over all of them, with no top-k selector at all. The win comes entirely from the heavy compression; the simplicity (no sparse selection) keeps it cheap and stable. Like CSA, it keeps a small sliding window of recent raw tokens for local detail.

Now we have two attention recipes. V4 doesn't choose one — it alternates them layer by layer.

After the first couple of layers (pure sliding-window attention to lock in local structure), V4 stacks CSA and HCA layers in an interleaved pattern. CSA layers give fine, selectable detail; HCA layers give a cheap, complete bird's-eye view of the whole context. Together with the per-token RoPE (rotary position encoding, telling tokens how far apart they are) and an attention sink (a learned slot that lets a head choose to attend to almost nothing), the hybrid keeps attention cost bounded as the context grows.

With attention efficient, the second upgrade is on the residual path you saw threading the stack — the lane that carries each block's input straight across.

Hyper-Connections (HC) widen that residual lane into several parallel streams and learn how to mix them. It helps — but stacking many such layers tends to blow up numerically. V4's fix, Manifold-Constrained Hyper-Connections (mHC), forces the mixing matrix B to be doubly stochastic (every row and column sums to 1, all entries ≥ 0), projected there by the Sinkhorn–Knopp algorithm. That guarantees its spectral norm (the most it can stretch any vector) is ≤ 1, so the residual transform is non-expansive: it can never amplify the signal, only redistribute it. Multiply many such matrices and the result stays well-behaved — stable to any depth.

Architecture done. Two training-side choices let a 1.6T-parameter model train and serve at this scale.

First, the Muon optimizer replaces most of Adam. Where Adam rescales each weight individually, Muon orthogonalizes the whole update matrix (via fast Newton–Schulz iterations) so all directions get a balanced step — giving faster convergence and steadier training. (Embeddings, norms and a few other modules still use AdamW.)

Second, the routed MoE expert weights — the bulk of the parameters — are trained and stored in FP4 (4-bit floating point) via quantization-aware training, slashing memory and, on future hardware, compute. The training itself was not painless: trillion-parameter MoE runs hit loss spikes traced to routing outliers, which the team tamed with two tricks (Anticipatory Routing and SwiGLU Clamping) that they note work empirically but aren't yet fully understood.

We've built the machine. Now compare its bill to V3.2's across context length — this is the chart from the paper's Figure 1, made draggable.

At short contexts the savings are modest; the further you push, the wider the gap. At a million tokens V4-Pro lands at 27% of V3.2's per-token FLOPs and 10% of its KV cache; V4-Flash, with fewer active parameters, reaches 10% of FLOPs and 7% of cache.

Efficiency and architecture explain the cost; one last piece explains the skills — V4's two-stage post-training.

First, specialist training: for each domain (math, code, agent, instruction-following) a separate expert is built — supervised fine-tuning to lay foundations, then reinforcement learning with GRPO (a reward-driven policy-optimization method) to sharpen it. Second, on-policy distillation: one unified model acts as a student, generating its own answers and learning to match the specialist teachers on them — folding many narrow experts into a single generalist. Notably, this replaced V3.2's mixed-RL stage entirely.