AIAny
Icon for item

The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation

Extrapolates RL-induced representation residuals into student hidden states during on-policy distillation: at each layer and token, it regresses the student beyond the teacher along the teacher’s RL-induced direction to improve stability and empirical performance.

Introduction

Why this matters Many on-policy distillation methods extrapolate an implicit reward in output (logit) space, but the LM head both attenuates and anisotropically filters representation changes and sampled log-ratio signals amplify noise. That makes output-space extrapolation unstable, especially when the RL teacher is close to its base checkpoint. The key insight of this paper is that RL produces a measurable directional shift in internal representations, and extrapolating that shift in representation space avoids the LM-head bottleneck and the high-variance token-probability noise.

Key Findings
  • RIDE (RL-Induced Direction Extrapolation) computes the per-layer, per-token residual between an RL-trained teacher and its pre-RL checkpoint, and regresses the student’s hidden states toward targets displaced beyond the teacher along that residual. This moves the student along the teacher’s learned direction while keeping it near the teacher.
  • The regression objective is equivalent (conditioned on a sampled trajectory) to maximizing a linear directional reward defined by the residual with a quadratic penalty centered on the teacher; this makes the optimization explicit about directionality and deviation control.
  • Across four base/teacher pairs spanning different scales, architectures, and pretraining lineages, RIDE approaches or exceeds the RL-trained teacher on every pair and is the only method whose mean performance does so; it consistently outperforms output-space extrapolation, which can degrade students when the teacher is close to its base checkpoint.
  • RIDE reduces sensitivity to noisy sampled-token log-probabilities and to the LM-head’s anisotropic mapping, improving stability when applying extrapolation-style objectives.
Who it’s for and tradeoffs

Great fit if you want to transfer RL gains from a teacher into a stronger student without running sparse-reward RL on the student, and you can access both the RL-trained teacher and its pre-RL checkpoint. RIDE is especially relevant when output-space extrapolation is unstable or when internal representation shifts are expected to carry the RL signal. Look elsewhere if you cannot access the teacher’s pre-RL checkpoint or you need a purely output-space method compatible with workflows that only expose logits; RIDE also requires incorporating representation-level losses into training, which changes memory/computation trade-offs compared to purely output-space objectives.

Information

  • Websitearxiv.org
  • AuthorsHao Li, MeiJia Chen, Weijie Ren, Donghan Li, Zijun Tian, Jingchun Huang, Naibo Wang
  • Published date2026/09/29

More Items

Tests decision models as actors, trajectory judges, and reinforcement-learning teachers for long-horizon agents. It finds that a model can be a weak actor yet still improve policy learning by evaluating actions after seeing their consequences.

Turns GTA V missions into automatically verifiable agent benchmarks and reinforcement-learning environments, with 150 navigation, interaction, and shooting tasks. Results expose persistent difficulty with dynamic control, while training on generated tasks improves some capabilities.

Trains language models to retrieve, study, and apply knowledge from textbooks through reinforcement learning. The approach raises overall mathematical proof accuracy from 51.5% to 79.5% and also improves results on two long-context benchmarks.