Hyper-Connections add flexibility by expanding and mixing residual streams, but that flexibility can be expensive: repeatedly reading and writing the expanded state creates overhead even when mixing is removed. This paper’s central insight is that effective residual-stream interactions can be retained without paying for either full mixing or repeated state access.
Key Findings
- Learned read/write gates with identity transitions perform comparably to mHC in the authors’ experiments, suggesting that residual mixing is not always necessary for competitive quality.
- SimpleHC conditions multiple sublayers’ gates on a shared group-entry state, batches reads, and defers writes. This cuts repeated state traffic while preserving sequential output contributions through scalar corrections.
- At the largest tested scale, training overhead is 2.4% over a vanilla residual baseline. On a 30B MoE model, SimpleHC matches mHC’s loss and slightly exceeds its reported average benchmark score, 63.1 versus 62.9.
- The results also point to a trade-off: unconstrained mixing can help, but may amplify or attenuate signals across depth; constrained Sinkhorn mixing gives little benefit over identity transitions in the reported comparisons.
How to Read the Results
The contribution is both architectural and systems-oriented: it links a simpler additive residual structure to an execution strategy that reduces state movement. The reported evidence covers experiments scaling from 15B to 30B MoE models and stress tests, not every model family or training setup. The gains therefore support SimpleHC as a practical alternative worth evaluating, rather than proving that mixing is unnecessary in all architectures.
Who It Suits—and the Trade-off
Great fit if you are researching residual-stream architectures or trying to reduce the training cost of large language models without sacrificing benchmark quality. Look elsewhere if you need evidence across diverse non-MoE models or a general recipe for replacing residual mixing: the paper’s results are tied to its tested configurations, and its shared-state gate generation is an approximation to per-sublayer conditioning.