A large parameter count does not necessarily mean every generated token pays the full compute cost. This model activates 14B of its 280B parameters per token, trading a substantial model footprint for sparse computation; that distinction matters when weighing training flexibility against deployment resources. Its released checkpoint is pre-training-only, so it is a starting point for adaptation rather than a ready-made instruction-following assistant.
Key Capabilities
- Sparse compute with a large expert pool: 42 MoE layers use 512 routed experts, selecting 10 per token alongside a shared expert. This offers a route to scaling capacity without activating every expert on every token, though the full model still demands significant memory and serving infrastructure.
- Long-context design: Training used sequences up to 128K tokens, while fine-tuning and inference support up to 1M. Hybrid attention combines compressed attention layers with sparse latent attention, targeting lower cache and attention costs as context grows.
- A research-oriented starting point: Pre-trained on 33.5T tokens, the base checkpoint can be adapted through domain training, instruction tuning, or reinforcement learning. The model card reports results across knowledge, math, code, multilingual, reasoning, and long-context evaluations; those results are useful for comparison but reflect the publisher's evaluation pipeline.
Who It's For
A good fit for teams researching continued pre-training, model adaptation, or long-context and MoE systems that can provision substantial memory and distributed inference. Look elsewhere if you need a small, immediately usable chat model, limited-resource local inference, or a checkpoint already tuned to follow instructions. The model is provided under its own license agreement, so review those terms before using or redistributing it.
