An open model release is only reproducible when researchers can inspect how the models were made, not just download their weights. This work’s central contribution is therefore as much about exposing the development process as about model performance: it releases training data and construction pipelines, checkpoints, logs, and training and evaluation code alongside a six-model family.
Key Findings
- The models span 0.9B to 375B-A23B parameters and target reasoning, coding, and agentic work, giving researchers a shared family to study across scales.
- Mixture-of-Value Attention (MoVA) routes experts through attention value projections. The 36B-A4B model combines MoVA with softplus attention gating and nearly matches a 32B dense model trained on the same data, using roughly one-seventh as many active parameters.
- Uno uses diffusion adapters to generate token blocks in parallel while leaving the base model’s output distribution intact. The paper reports up to 2.2× faster decoding without a separate draft model or a loss of response quality.
- The release includes roughly 10T synthetic pre-training tokens and documents the pipelines that produced them, making synthetic data generation itself available for scrutiny and reproduction.
Who It’s For—and the Trade-Off
A strong fit for researchers studying open model training, scaling, sparsity, reasoning, or agentic behavior, especially those who need intermediate checkpoints and data provenance. It is less suitable for users seeking a small, turnkey hosted model: the largest checkpoints demand substantial infrastructure, and reproducing the full training history requires significant compute and engineering effort.
Method in Context
MoE usually sparsifies feed-forward layers; MoVA extends expert routing into attention, where tokens combine information across context. That creates another way to scale capacity without activating every parameter for every token. Uno addresses a separate bottleneck—autoregressive generation’s one-token-at-a-time latency—through lightweight diffusion adapters rather than replacing the base model’s decoding distribution.