Attention isn't compute-bound at decode time — it's memory-bound, throttled by how fast you can stream the KV cache. FlashMLA's whole design orbits that one bottleneck: these are the production decoding kernels DeepSeek itself runs to serve V3 and V3.2, pushing an H800 to roughly 3000 GB/s of effective KV bandwidth before compute ever becomes the limit.
What Sets It Apart
- Battle-tested in DeepSeek's own serving stack, not a reference toy — what ships here is what runs in production behind V3/V3.2.
- Built specifically for MLA, the compressed-KV scheme that makes DeepSeek's long-context inference cheap; the kernel exploits MLA's tiny per-token footprint rather than treating it as generic attention.
- The sparse decode path stores the KV cache in FP8 (656 bytes/token: 512 quantized values + 16 scale + 128 unquantized RoPE) while keeping matmuls in bf16, cutting memory traffic without collapsing accuracy.
- Spans SM90 (Hopper) and SM100 (Blackwell), with published figures around 660 TFLOPS dense and 410 TFLOPS sparse decode on H800.
Who It's For
Great fit if you're building an inference engine around MLA models and want kernels already tuned for the exact KV layout DeepSeek uses. Look elsewhere if you serve standard MHA/GQA models, don't have a Hopper-or-newer GPU, or aren't on CUDA 12.8+ — the kernels are narrowly specialized and won't generalize to arbitrary attention shapes.
