AIAny
Icon for item

Multi-Branch Transformer

Studies scaling language models with parallel Transformer branches and introduces Joint-KV to share key-value caches across branches. Experiments report improved modeling quality at comparable training compute and faster long-context decoding.

Introduction

Adding width or depth is not the only way to spend a language model’s compute budget. This paper explores a less conventional option: run independently parameterized Transformer blocks in parallel, then fuse their representations. The key insight is that branching can improve quality without simply making a sequential model wider or deeper—and shared KV construction can make the resulting parallel architecture practical at inference time.

Key Findings
  • Across four MoE scales from 0.8B to 2.1B total parameters, two-branch Wide-Shallow models outperform matched sequential Transformers; Balanced variants improve further. This suggests that the gains are not explained only by reallocating width and depth.
  • Joint-KV builds a shared cache at each branch depth while keeping queries branch-local. It cuts full-model KV-cache storage by up to 34%, addressing a major cost of long-context serving.
  • The final Balanced + Joint-KV models reach lower validation loss than their Standard baselines. Fitted scaling curves estimate that Standard needs up to 1.26× the training-body compute to match their validation loss; on one H200, the reported peak decoding-throughput gain is 51.4%.
How the Architecture Works

Inputs first pass through shared layers, then through parallel branches of sequential Transformer blocks. A fusion step combines branch outputs before shared final layers. The paper compares Wide-Shallow, Narrow-Deep, and Balanced designs, and finds branching most effective in middle layers. Joint-KV is a model-level sharing strategy, not simply a serving trick: branch representations jointly form the cache at each depth, while attention queries remain local.

Who It Suits—and the Trade-off

A useful read for researchers and infrastructure teams evaluating architecture changes for MoE language models, especially when long-context KV memory and decoding throughput matter. The evidence is controlled but bounded: four model scales, specific training and serving setups, and H200 inference measurements. It does not establish that branching wins at every scale, hardware target, or workload, and the parallel execution and fusion add implementation complexity.

Information

  • Websitearxiv.org
  • OrganizationsUniversity of Illinois Urbana-Champaign, Rice University, University of Michigan, Cornell University
  • AuthorsGaotang Li, Lingjie Chen, Chunyuan Deng, Zhiwei Xu, Hongzheng Chen, Christopher De Sa, Jingrui He, Yingheng Wang, Hanghang Tong
  • Published date2026/11/15

More Items

Examines whether Transformers are necessary when LLMs write model weights without gradient training. When allowed to design the architecture, the tested LLMs compile facts into sparse tables, reaching up to 54.4% accuracy on ARC.

Evaluates how well models and coding agents optimize operators for real Ascend NPU workloads, accounting for both speedups and token cost. Test cases preserve tensor shapes and data types from real model execution contexts.

Details the Xing 4 model architecture, Ascend-scale pretraining systems, and agentic post-training, including a 285B-parameter training run on 8,192 Ascend 910C cards.