Why this release matters
Long-context multimodal models change what agents can hold in memory: multi-session tool traces, entire repositories, long videos and prolonged agent rollouts. This checkpoint experiments with scaling not just model size but reinforcement-learning infrastructure—mixing thousands of diverse agent environments, groupwise grading, and very large asynchronous RL batches—to nudge a policy toward shorter, higher-quality solution paths rather than relying solely on single-turn supervised fine-tuning.
Key Capabilities
- Architecture and scale: sparse Mixture-of-Experts backbone with ~1.02 trillion total parameters and ~42B activated (per token), 70 transformer layers (60 sliding-window, 10 global-attention) and 384 routed experts (8 active). Hidden size ~6144, sliding window mechanics and a 5-layer speculative decoder for multi-token drafting.
- Long context and modalities: native 1,048,576-token context window and 131,072-token max completion; accepts text, image, video and audio inputs via dedicated vision and audio encoders (vision encoder ≈681M params; audio encoders ≈435M params). Output is text-only.
- Reinforcement learning for self-improvement: single mixed RL run ("You Only RL Once") across code, general agents, visual tasks and cybersecurity using Group Relative Policy Optimization (GRPO) with groupwise grading (Groupwise Reward Synthesis and Groupwise Advantage Redistribution) to produce richer rewards and rank high-quality rollouts within groups.
- Practical posture: MIT-licensed weights published on HuggingFace in multiple precisions (BF16/FP8/INT8/safetensors) and recipes for serving (SGLang, vLLM). Designed for agentic workflows, tool-calls, reasoning traces and long-horizon evaluations.
Who it's for and tradeoffs
Great fit if you need a self-hostable multimodal foundation model with a million-token context for long-running agents, repository-scale code reasoning, or multimodal visual+code workflows and you have large GPU/TPU infrastructure to run sparse MoE models. The model's RL-centered training and groupwise grading are tailored to research into agentic self-improvement and long-horizon task optimization.
Look elsewhere if you require native image or audio generation (this checkpoint emits text only), low-inference-latency on commodity hardware, or turnkey cloud endpoints across all major CSPs—top closed-source frontier models may still surpass it on some benchmarks. Operational costs, deployment complexity (MoE routing, tensor-parallel setups) and safety/red-teaming requirements are nontrivial.
Where it fits
Practically positioned as an open-weights alternative for teams and researchers exploring long-context multimodal agents and RL-driven capability scaling. Use it for agent research, benchmark studies on long-horizon behavior, multimodal code/visual reasoning, and custom tool-enabled agent stacks when you can handle the infra and safety overhead.
