More decoder computation does not have to mean more distinct layer weights. This preview explores that tradeoff by running a shared block twice, reaching 56 layer executions from 32 distinct layers. The result is aimed at workloads where long multimodal context and multi-step tool-oriented tasks matter, though the deeper execution path also makes compatible inference support important.
Key Capabilities
- Shared-weight looping: Layers 4–27 run twice, so the model adds a second pass through a substantial decoder block without storing a second copy of its weights.
- Two-stream mHC: Learned routing mixes information across two residual streams in the repeated section, giving the loop a structured way to carry and update state.
- Text-and-image input with long-context training: Documented training sequences reach 262,144 tokens and include text, images, code, and supervised task trajectories—useful for document-heavy or tool-assisted workflows.
- Hybrid attention: Three linear-attention layers are followed by one full-attention layer in repeating groups, balancing attention mechanisms across the decoder.
Who It’s For
Great fit if you are evaluating long-context multimodal models or agentic workflows and can use an inference stack that supports its loop and mHC execution configuration. Look elsewhere if you need a mature, broadly integrated checkpoint: this is a preview, custom model code is required, and the technical report and some deployment integrations are not yet published. The extra layer executions should not be read as a guarantee of faster or cheaper inference; actual performance depends on hardware and serving setup.
