Why this matters: on-policy distillation (OPD) trains students under their own generated data while supervising them with stronger teachers, but this loop can unexpectedly produce extremely long, budget-exhausting rollouts. The paper's core insight is that surface-level EOS token disagreements between student and teacher can suppress the student’s native termination action and unintentionally incentivize longer generations.
Key Findings
- Termination-token mismatch: base students and post-trained teachers often allocate stopping probability to different surface EOS tokens even when their declared stop sets match. This mismatch can collapse the student’s termination probability and allow rollouts to balloon.
- Semantic EOS aggregation: treating functionally equivalent EOS tokens as one shared stopping action (aggregating probability mass) substantially reduces redundant tokens and truncation across model families including Qwen3, Llama, and Gemma.
- Decoding fixes insufficient alone: simply enlarging the decoding stop set does not reliably transfer termination mass; probability-level corrections that remap or aggregate EOS probabilities are more effective.
- Dynamics over training: termination preferences shift during OPD; a late-stage re-inflation can appear even after EOS alignment, so EOS mismatch is an important but not sole cause of length inflation.
Who should read this / Tradeoffs
Great fit if you train or fine-tune LLMs with on-policy distillation or diagnose runaway generation in distillation workflows — the paper offers a concrete, implementable correction and an open implementation. Look elsewhere if your setup does not use sampled-token OPD or if length issues stem primarily from other causes (e.g., reward/design incentives); the proposed fix reduces but does not fully eliminate late-stage inflation, so additional stabilization techniques may still be needed.