The paper argues that compact foundation models can exceed memory-limited performance by combining deliberate internal reasoning with active external tool use, and demonstrates this at 7B scale with extreme long-context and training efficiency. The surprising claim is empirical: a carefully designed small dense model can rival much larger models on math and agentic search benchmarks when paired with long contexts and tool interaction.
Key Findings
- Architecture & efficiency: interleaved gated sliding-window local attention with a small number of global attention layers reduces KV-cache and raises throughput for 256K contexts (reported ~3.9× throughput gain vs full attention at 256K).
- Training recipe: hybrid FP8 precision with a Muon optimizer and a progressive context curriculum (pretrain → midtrain 16K→64K→256K) yields an estimated ~4.2× improvement in 16K pretraining time-to-loss.
- Capabilities: strong performance on math and agentic-search tasks (examples reported: MATH-500 97.13%, AIME 2026 75.00%, WebWalkerQA ~63%). The model supports both internal “thinking” and direct-response modes and is trained with execution-verified trajectories for agentic behavior.
- Reproducibility: authors open-source weights, intermediate checkpoints, per-stage data recipes, training code, and W&B logs to enable community follow-up.
Who it's for and tradeoffs
Great fit if you are a researcher or practitioner who needs an open, long-context LLM optimized for reasoning and tool-assisted search and who values reproducible training artifacts. The release is valuable for people evaluating architectural long-context tradeoffs, hybrid-precision training, and agentic co-training dynamics. Look elsewhere if you need a multimodal foundation model, extremely large-capacity models (>100B) for purely memorization-heavy tasks, or turnkey hosted inference APIs — this work focuses on training recipes, model design, and datasets rather than managed serving.
Method and practical notes
The development lifecycle combines ~4.19T tokens of pretraining, a mid-training phase of ~600B tokens with explicit context scaling, and supervised fine-tuning that mixes thinking/direct-response examples and execution-verified tool trajectories. The model is a 7.39B decoder-only dense Transformer (32 layers, 4,096 hidden) with 27 gated sliding-window layers and 5 global layers and a 262,144-token context length target. The paper also distills practical empirical findings about SFT quality pruning, long-context generalization, and agentic co-training dynamics that can guide follow-up experiments.