Long-context models often face a trade-off: attend to the full history at high cost, or discard context to save compute. This model takes a middle path: it retains the full-history KV cache while limiting each attention group to selected prefix, nearby, and learned remote blocks. Combined with 3B active parameters per token, that design aims to make long histories practical without making every token pay for dense attention over the entire sequence.
Key Capabilities
- Selective long-context attention: At up to 256K tokens, each KV group attends to a bounded set of 128-token blocks. The team reports a 6.7× kernel speedup over dense GQA at 256K; that is a kernel comparison, not a guarantee of equivalent end-to-end speedups across hardware and workloads.
- Sparse compute, broad capacity: The model has 35B total parameters but activates 3B per token, routing among fine-grained experts while keeping shared experts active. This separates stored capacity from the amount of expert computation used for each token.
- Reasoning and agent training: Post-training combines supervised fine-tuning, separate reinforcement-learning branches, and multi-teacher on-policy distillation. The team reports strong results on its selected math and search evaluations, including 92.61% mean@8 across six mathematics benchmarks and 76.30% pass@1 on BrowseComp.
- Multilingual coverage: Pre-training uses approximately 26T multilingual tokens, and the tokenizer is designed to remain compact across scripts, including several Southeast Asian languages.
Who It’s For
A good fit for teams exploring open-weight models for long-context reasoning, multilingual text, or tool-using agents, especially when they can run a large checkpoint on compatible NVIDIA infrastructure. Look elsewhere if you need multimodal input, a lightweight consumer deployment, or independently verified benchmark comparisons: this is text-only, its weights are substantial, and the published performance figures are the model team’s own evaluations.