Long-horizon agent workloads drive input-heavy model usage where prefill cost and KV cache size become primary deployment bottlenecks. This paper's core insight is that combining an architecture tuned for low-prefill activation with aggressive cross-layer KV reuse and compact numeric formats can cut both in-memory (HBM) and persistent (SSD/host) KV footprints by substantial factors without sacrificing overall performance.
Key Findings
- Architecture and compute balance: adopting a Causal Encoder–Decoder (CED) layout lets the system activate ~8B parameters per token during prefill and ~16B during decoding — reducing prefill compute for input-heavy agent workloads.
- KV-cache compression techniques: Compressed Sparse Attention 2 (CSA2) enables cross-layer KV reuse and index sharing; combined with FP4 (E2M1/E4M3-style) KV storage, the global (HBM) KV footprint is reduced to ~890 bytes/token (≈1/4 vs prior generation).
- Persistent-storage optimization: SWA Bounded Replay reconstructs missing SWA KV states by replaying only a recent window, cutting persistent KV footprint on SSD/host to ~1/8 of the prior model while avoiding full SWA persistence.
- Scale and scope: DeepSeek-V4.1-Flash is a multimodal MoE backbone with 552B parameters, trained on a 45T-token multimodal corpus (sparse attention at 64K, context extended to 1M at 34T tokens), aiming at agentic and long-context multimodal tasks.
Who it's for and tradeoffs
Great fit if you operate or research long-horizon multimodal agents and need to reduce HBM/SSD KV costs at large scale while keeping high context lengths. It is attractive to infra teams optimizing large-model serving, and to researchers exploring sparse attention, MoE scaling, and numeric-compression tradeoffs. Look elsewhere if you need a small-footprint model for edge devices, if you cannot accept MoE/FP4-related engineering complexity, or if high-precision (non-quantized) KV storage is mandatory; the approach assumes infrastructure capable of managing MoE routing, FP4 compute paths, and replay-based persistent-state reconstruction.
Where it fits
This work sits between advances in long-context sparse-attention and deployment-focused compression: it targets the deployment bottlenecks (prefill compute, HBM bandwidth, SSD capacity) rather than raw benchmark SOTA, offering concrete reductions in KV storage and prefill activation for large-scale agentic workloads.
Method and notable mechanisms
- CED layout: a 20-layer causal encoder + 20-layer decoder where decoder KV is projected from final encoder states to lower per-token activation during prefill.
- CSA2: assigns static attention modes (Full/Reindex/Reuse) across layers to share main KV and reuse Top-K sparse indices, plus a hierarchical indexer to bound deeper index costs.
- FP4 KV caching: low-bit KV storage format with per-channel scaling to compress HBM footprint.
- SWA Bounded Replay: a deployment technique to avoid persisting full SWA KV by reconstructing recent windows on demand.
The paper emphasizes practical deployment tradeoffs and presents detailed design choices for teams aiming to lower the cost of serving extremely long contexts at scale.