AIAny
Icon for item

Loopback Transformer: Towards a Unified View of Residual Looped Transformers

Reduces Transformer parameter count by repeatedly applying shared blocks, then combines their updates with geometric decay and learned attention. On ClimbMix, it beats standard baselines in validation loss with fewer Transformer parameters.

Introduction

Parameter sharing saves weights, but it does not by itself explain how repeated computation should reuse its history. The key observation here is that weighting a block’s full output also changes the weight of everything already carried in its residual stream. Separating each new update from the loop’s original input gives the model more direct control over what to retain.

Information

  • Websitearxiv.org
  • OrganizationsUniversity of Wisconsin–Madison, University of California, Santa Barbara, California Institute of Technology, Microsoft
  • AuthorsXuan Ouyang, Zefan Cai, Chenlong Zhang, Cheng Luo, Ke Wan, Junjie Hu
  • Published date2026/11/04

More Items

Speeds up lossless speculative decoding for vision-language models by focusing drafter training on visually dependent tokens. Sparse attention also reduces redundant work on long video inputs; reported speedups reach 3.58×.

Proposes a framework for robots that combine foundation models with an embodied runtime to interpret intent, call skills, verify outcomes, and adapt. It also outlines an evaluation framework and a roadmap toward sustained autonomy.

Turns screen observations into keyboard and mouse actions to complete long-horizon game tasks. Sparse reasoning, persistent memory, and online reinforcement learning support real-time interaction and multi-step decisions.