Why this matters
Most residual variants in deep autoregressive Transformers treat depth-wise shortcuting as a fixed or single-stream retrieval problem. Multi-Stream Attention Residuals (MSAR) flips that assumption: it deterministically expands a single residual source into several sequence-context views via small causal depthwise convolutions, then performs independent softmax attention over network depth for each stream. The core insight is that different local sequence contexts are useful for retrieving different historical layer states, and combining these retrieved streams yields richer, content-dependent layer inputs without persistent multi-stream storage.
Key Findings
- Better pretraining loss and downstream performance: MSAR achieved lower final training loss in 18B and 28B MoE settings and led across 12 evaluated benchmarks (knowledge, reasoning, math, code). This indicates the method improves representation utility across tasks.
- Low runtime and memory overhead: in a 28B MoE setup MSAR adds only ~0.5% training FLOPs and ~0.3% activated parameters; it keeps storage cost at O(Nd) by recomputing streams on demand, matching AttnRes at the storage level.
- Concrete architectural choices: MSAR typically uses the original residual plus three causal depthwise convolutions (kernel sizes 12, m=4 streams) as distinct depth-attention sources; per-stream retrieved states are fused by a lightweight mixer into the layer input.
- Scaling gains quantified: MSAR reaches the same loss with substantially less compute—iso-loss compute speedup of 1.58× over the standard residual baseline, compared with 1.35× for AttnRes and 1.30× for mHC—showing practical benefits for large-scale pretraining budgets.
Who it's for and tradeoffs
Great fit if you train large autoregressive Transformer or MoE models and care about faster loss scaling and improved downstream results with minimal engineering changes. MSAR is compatible with existing AttnRes block modes, cache-based pipeline communication, and two-phase computation strategies, so it integrates into common large-scale training stacks.
Look elsewhere if your workload is tiny (single-layer toy models) or if implementation simplicity is paramount: MSAR adds modest additional kernels (causal depthwise convolutions and a small mixer) and slightly more operator complexity than vanilla residuals. Also, the benefits are presented for autoregressive pretraining at large model scales—gains on very small models or non-autoregressive regimes are not the focus here.