AIAny
Icon for item

Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It

Demonstrates that pretrained transformers typically use only ~1–3 lines of depth to follow reference chains, and that a task‑trained rank‑8 LoRA applied at one early layer (with all other weights frozen) can extend reference‑following to dozens or hundreds of lines while adding only a few ten‑thousand parameters.

Introduction

Oops! Something went wrong

[next-mdx-remote-client] error compiling MDX: Unexpected character `<` (U+003C) before name, expected a character that can start a name, such as a letter, `$`, or `_` More information: https://mdxjs.com/docs/troubleshooting-mdx

Information

  • Websitearxiv.org
  • OrganizationsGeorgia Institute of Technology
  • AuthorsZehao Jin, Ruixuan Deng, Junran Wang
  • Published date2026/09/29

More Items

Provides six open language models from 0.9B to 375B-A23B for reasoning and agentic tasks. The paper also introduces sparse attention and a decoding speedup that needs no separate draft model.

Reduces Transformer parameter count by repeatedly applying shared blocks, then combines their updates with geometric decay and learned attention. On ClimbMix, it beats standard baselines in validation loss with fewer Transformer parameters.

Speeds up lossless speculative decoding for vision-language models by focusing drafter training on visually dependent tokens. Sparse attention also reduces redundant work on long video inputs; reported speedups reach 3.58×.