AIAny
Icon for item

K2-Horizon: A Frontier Fleet of Open Source Language Models

Provides six open language models from 0.9B to 375B-A23B for reasoning and agentic tasks. The paper also introduces sparse attention and a decoding speedup that needs no separate draft model.

Introduction

An open model release is only reproducible when researchers can inspect how the models were made, not just download their weights. This work’s central contribution is therefore as much about exposing the development process as about model performance: it releases training data and construction pipelines, checkpoints, logs, and training and evaluation code alongside a six-model family.

Key Findings
  • The models span 0.9B to 375B-A23B parameters and target reasoning, coding, and agentic work, giving researchers a shared family to study across scales.
  • Mixture-of-Value Attention (MoVA) routes experts through attention value projections. The 36B-A4B model combines MoVA with softplus attention gating and nearly matches a 32B dense model trained on the same data, using roughly one-seventh as many active parameters.
  • Uno uses diffusion adapters to generate token blocks in parallel while leaving the base model’s output distribution intact. The paper reports up to 2.2× faster decoding without a separate draft model or a loss of response quality.
  • The release includes roughly 10T synthetic pre-training tokens and documents the pipelines that produced them, making synthetic data generation itself available for scrutiny and reproduction.
Who It’s For—and the Trade-Off

A strong fit for researchers studying open model training, scaling, sparsity, reasoning, or agentic behavior, especially those who need intermediate checkpoints and data provenance. It is less suitable for users seeking a small, turnkey hosted model: the largest checkpoints demand substantial infrastructure, and reproducing the full training history requires significant compute and engineering effort.

Method in Context

MoE usually sparsifies feed-forward layers; MoVA extends expert routing into attention, where tokens combine information across context. That creates another way to scale capacity without activating every parameter for every token. Uno addresses a separate bottleneck—autoregressive generation’s one-token-at-a-time latency—through lightweight diffusion adapters rather than replacing the base model’s decoding distribution.

Information

  • Websitearxiv.org
  • OrganizationsInstitute of Foundation Models
  • AuthorsZhengzhong Liu, Haonan Li, Xuezhe Ma, Alexander Moreno, Subham Sekhar Sahoo, Rupesh Kumar Srivastava, Mikhail Yurochkin, Hadi Abdine, Yash Akhauri, Adharsh Babu …
  • Published date2026/11/16

More Items

Blocks tool-using agents from directing emails, payments, and other side effects to unauthorized targets by checking typed tool-return fields. Deterministic rules also remove instruction-like text before the model reads it.

Hugging Face
AI Audio2026

Turns English text into speech with two American English voices, running on CPUs or in a browser. Its 4.5M-parameter model was trained from scratch on synthetic speech.

Hugging Face
AI Model2026

Simulates how a scene changes in response to text-described actions, optionally conditioned on a starting image, and generates videos up to 20 seconds long. It pairs language-model reasoning about events with a 29B diffusion model for visual prediction.