The bottleneck in attention was never the math — it's the memory traffic. A standard implementation writes the full N×N score matrix out to slow GPU HBM and reads it back, so attention is IO-bound long before it is compute-bound. The trick here is to tile the computation and keep intermediate scores in fast on-chip SRAM, streaming through the sequence and never materializing that matrix in HBM — producing the numerically exact same output.
What Sets It Apart
- Exact, not approximate. Unlike sparse or low-rank attention, outputs are bit-for-bit equivalent to naive attention, so it drops into an existing model with zero accuracy trade-off — no retraining, no quality regression.
- Linear memory in sequence length. Removing the quadratic activation term is what made training at 32k+ context practical on ordinary hardware, rather than a research stunt.
- Each version chases the silicon. v2 rebalanced work across GPU warps for higher occupancy; v3 exploits Hopper (H100) async copies and FP8; Triton and ROCm backends extend reach beyond hand-written CUDA and NVIDIA.
- A dedicated decode path. KV-cache and paged variants target the memory-bound single-token generation of inference, not just the throughput-bound training case.
Great Fit If / Look Elsewhere
Great fit if you train or serve transformers on modern NVIDIA GPUs and want speed and memory wins with no model changes — it is already the default kernel inside PyTorch, vLLM, and most training stacks, so you may be using it without knowing. Look elsewhere if you are on pre-Ampere cards (support is partial or absent), need CPU inference, or run short sequences where the score matrix already fits comfortably in cache — there the payoff shrinks toward noise.
