AIAny
Icon for item

Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning

Separates knowledge storage (a global Memory) from iterative reasoning operators (multiple Reasoners) to improve knowledge compression and inference efficiency; reports a 7B model matching baseline with 62.6% of training data and a 35B Intern-S2-Mobius achieving ~4x end-to-end speedup.

Introduction

Most large-model work tightly couples knowledge storage with local computation, which inflates parameter redundancy and makes multi-step deliberation costly. This paper argues that explicitly decoupling where knowledge lives (a global, addressable Memory) from how reasoning is performed (iterative Reasoners) yields a compact representation of knowledge and much shorter internal reasoning traces, cutting end-to-end inference time.

Key Findings
  • Architecture: Mobius-v0 replaces layer-bound FFN knowledge storage with a globally shared Memory (knowledge vectors) and uses multiple Reasoners (self-attention blocks) that iteratively query and refine hidden states against that Memory. This separation enables backward residual access and dynamic latent reasoning, letting reasoning stages access knowledge beyond local layers and internalize multi-token deliberation into dense continuous states.
  • Empirical gains: A 7B model trained from scratch on the Mobius layout reaches comparable downstream performance to a Transformer 7B baseline while using 62.6% of the baseline's training data. Intern-S2-Mobius (a 35B model continually pretrained from Qwen3.5-35B) preserves downstream capability and reports nearly 4× end-to-end inference throughput improvement in the authors' evaluation, with shorter reasoning traces on benchmarks.
  • Deployment and tooling: The released 35B weights use bfloat16 and are demonstrated with common inference frameworks (LMDeploy, vLLM, Transformers, etc.), and include optimizations such as speculative decoding and multi-token drafting to realize practical speedups.
Who it's for and trade-offs

Great fit if you need lower-latency, reasoning-heavy LLM inference or want to explore architectures that compress knowledge representations for scientific and reasoning tasks. The approach is attractive for production deployments where throughput and concise internal deliberation matter. Look elsewhere if you require off-the-shelf compatibility with every Transformer toolchain: the Mobius layout introduces new runtime and implementation considerations (specialized memory/query patterns, support in certain inference stacks), and the gains depend on matching workloads where iterative querying and internal deliberation are beneficial. As with any novel architecture, maturity, tooling, and broad benchmarking beyond the authors' reported suites will affect adoption.

Where it fits

Mobius sits between standard decoder-only Transformers and memory-augmented models: it keeps Transformer-style reasoners but relocates knowledge to an addressable global module. This positions it as a candidate architecture for next-generation foundation models focused on compact knowledge storage and efficient multi-step reasoning.

Method highlights

The paper emphasizes (1) a global FFN-like Memory storing key/value knowledge vectors, (2) multiple Reasoners that query and refine hidden states iteratively against Memory, and (3) training and deployment recipes (continual pretraining from a large base model and inference optimizations) that demonstrate practical speed/quality trade-offs on downstream benchmarks.

Information

  • Websitearxiv.org
  • AuthorsKai Chen, Jifeng Ding, Ning Ding, Jiaye Ge, Lixin Gu, Yicheng Gu, Qipeng Guo, Ermo Hua, Haian Huang, Haozheng Hou …
  • Published date2026/08/14

More Items

Evaluates fast-decision models across 2,919 tasks in 11 domains, measuring both capability and reliability through option-order changes and cases with no valid answer.

Studies how to simplify Hyper-Connections and reduce their training overhead. SimpleHC removes residual mixing and shares state access, matching mHC performance on a 30B MoE model.

Lets AI agents read task-relevant evidence from external content instead of receiving entire documents by default. Intent tracking, adaptive readers, and an exposure budget reduce token use and latency while retaining a faithful full-view fallback.