AIAny
Icon for item

Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs

Shows that Transformer LLMs output a superposition of next-token distributions when token embeddings from distinct text streams are linearly combined, finds this property is architectural but weakened by pretraining, and demonstrates it can be restored by lightweight fine-tuning and disentangled at decode time.

Introduction

Oops! Something went wrong

[next-mdx-remote-client] error compiling MDX: Unexpected character `0` (U+0030) before name, expected a character that can start a name, such as a letter, `$`, or `_` More information: https://mdxjs.com/docs/troubleshooting-mdx

Information

  • Websitearxiv.org
  • AuthorsPavel Tikhonov, Anton Korznikov, Matvey Mikhalchuk, Nikita Dragunov, Temurbek Rahmatullaev, Polina Druzhinina, Anton Razzhigaev, Ivan Oseledets, Elena Tutubalina
  • Published date2026/09/24

More Items

Tests whether large language models can tell when a simulator’s modeling flaw changes which system design should be chosen. Paired workload comparisons reveal when models reject valid recommendations or retain invalid ones.

Accelerates decision-model inference by computing a shared context once and forking it across questions. In tests with multiple questions per state, it runs up to 9.5× faster than vLLM and SGLang.

Trains a single agent policy with successful trajectories from specialized sibling harnesses when attempts under the base harness all fail. The resulting policy improves under the original harness, with optional task-specific harness routing at inference.