Most large-model work tightly couples knowledge storage with local computation, which inflates parameter redundancy and makes multi-step deliberation costly. This paper argues that explicitly decoupling where knowledge lives (a global, addressable Memory) from how reasoning is performed (iterative Reasoners) yields a compact representation of knowledge and much shorter internal reasoning traces, cutting end-to-end inference time.
Key Findings
- Architecture: Mobius-v0 replaces layer-bound FFN knowledge storage with a globally shared Memory (knowledge vectors) and uses multiple Reasoners (self-attention blocks) that iteratively query and refine hidden states against that Memory. This separation enables backward residual access and dynamic latent reasoning, letting reasoning stages access knowledge beyond local layers and internalize multi-token deliberation into dense continuous states.
- Empirical gains: A 7B model trained from scratch on the Mobius layout reaches comparable downstream performance to a Transformer 7B baseline while using 62.6% of the baseline's training data. Intern-S2-Mobius (a 35B model continually pretrained from Qwen3.5-35B) preserves downstream capability and reports nearly 4× end-to-end inference throughput improvement in the authors' evaluation, with shorter reasoning traces on benchmarks.
- Deployment and tooling: The released 35B weights use bfloat16 and are demonstrated with common inference frameworks (LMDeploy, vLLM, Transformers, etc.), and include optimizations such as speculative decoding and multi-token drafting to realize practical speedups.
Who it's for and trade-offs
Great fit if you need lower-latency, reasoning-heavy LLM inference or want to explore architectures that compress knowledge representations for scientific and reasoning tasks. The approach is attractive for production deployments where throughput and concise internal deliberation matter. Look elsewhere if you require off-the-shelf compatibility with every Transformer toolchain: the Mobius layout introduces new runtime and implementation considerations (specialized memory/query patterns, support in certain inference stacks), and the gains depend on matching workloads where iterative querying and internal deliberation are beneficial. As with any novel architecture, maturity, tooling, and broad benchmarking beyond the authors' reported suites will affect adoption.
Where it fits
Mobius sits between standard decoder-only Transformers and memory-augmented models: it keeps Transformer-style reasoners but relocates knowledge to an addressable global module. This positions it as a candidate architecture for next-generation foundation models focused on compact knowledge storage and efficient multi-step reasoning.
Method highlights
The paper emphasizes (1) a global FFN-like Memory storing key/value knowledge vectors, (2) multiple Reasoners that query and refine hidden states iteratively against Memory, and (3) training and deployment recipes (continual pretraining from a large base model and inference optimizations) that demonstrate practical speed/quality trade-offs on downstream benchmarks.