Most clever efficient-attention ideas die in a research notebook: a neat recurrence that nobody can plug into a real training run. What actually made linear attention usable wasn't new math — it was a chunkwise-parallel, Triton-level implementation fast enough to compete with FlashAttention on wall-clock time. That implementation lives here, and it has quietly become the reference codebase the whole sub-field builds on.
What Sets It Apart
- One home for a fragmenting field. GLA, RWKV-6/7, Mamba2, GSA, DeltaNet, HGRN2 and more share a single set of fused kernels and
nn.Moduletoken mixers, so switching architectures is a config change rather than a rewrite of custom CUDA. - The hard kernel is already written. The chunkwise-parallel form — the trick that lets these models train as fast as they infer — is implemented and tuned across NVIDIA, AMD and Intel hardware, not left as an exercise.
- Hugging Face-native. Models expose standard
generate()APIs alongside fused ops (rotary embeddings, RMSNorm, fused cross-entropy), so a linear-attention model slots into an existing training and eval stack instead of forcing a parallel one.
Great Fit / Look Elsewhere
Great fit if you research sub-quadratic sequence models, want to benchmark a new linear / SSM / sparse variant against strong baselines, or need to scale a non-Transformer LLM without hand-writing kernels. Look elsewhere if you want a turnkey chatbot or a frozen, stable API — this is a fast-moving research library where module signatures and kernels shift between releases, and correctness on less common hardware still warrants your own verification.
