Adding width or depth is not the only way to spend a language model’s compute budget. This paper explores a less conventional option: run independently parameterized Transformer blocks in parallel, then fuse their representations. The key insight is that branching can improve quality without simply making a sequential model wider or deeper—and shared KV construction can make the resulting parallel architecture practical at inference time.
Key Findings
- Across four MoE scales from 0.8B to 2.1B total parameters, two-branch Wide-Shallow models outperform matched sequential Transformers; Balanced variants improve further. This suggests that the gains are not explained only by reallocating width and depth.
- Joint-KV builds a shared cache at each branch depth while keeping queries branch-local. It cuts full-model KV-cache storage by up to 34%, addressing a major cost of long-context serving.
- The final Balanced + Joint-KV models reach lower validation loss than their Standard baselines. Fitted scaling curves estimate that Standard needs up to 1.26× the training-body compute to match their validation loss; on one H200, the reported peak decoding-throughput gain is 51.4%.
How the Architecture Works
Inputs first pass through shared layers, then through parallel branches of sequential Transformer blocks. A fusion step combines branch outputs before shared final layers. The paper compares Wide-Shallow, Narrow-Deep, and Balanced designs, and finds branching most effective in middle layers. Joint-KV is a model-level sharing strategy, not simply a serving trick: branch representations jointly form the cache at each depth, while attention queries remain local.
Who It Suits—and the Trade-off
A useful read for researchers and infrastructure teams evaluating architecture changes for MoE language models, especially when long-context KV memory and decoding throughput matter. The evidence is controlled but bounded: four model scales, specific training and serving setups, and H200 inference measurements. It does not establish that branching wins at every scale, hardware target, or workload, and the parallel execution and fusion add implementation complexity.