Why this matters
As practitioners seek sample-efficient ways to improve LLM reasoning without relying on larger teacher models, on-policy self-distillation (OPSD) has emerged as a lightweight alternative that gives dense token-level feedback by letting the model act as its own teacher with privileged context. This review argues that the central risk is not a missing optimizer detail but a behavioural failure—“collapse,” the progressive loss of diverse reasoning paths—and that collapse should be studied as a symptom controlled by a small set of design choices.
Key Findings
- Collapse is the dominant failure mode observed in OPSD-style methods: dense on-policy signals can narrow the model’s output distribution and reduce reasoning diversity. This matters because it can degrade generalization even if short-term accuracy improves.
- Three levers explain when and how collapse occurs: (1) where the signal is applied — token weighting and credit assignment; (2) what the teacher is shown — the nature and specificity of privileged information; (3) when the signal changes — teacher dynamics and decay of guidance over training.
- Privileged references help produce dense supervision but also bias the teacher’s behavior; some gains attributed to passing the correct solution may instead arise from context-induced teacher behaviors rather than target-specific information.
- The paper is a structural review: it synthesizes prior results, proposes a shared vocabulary, draws a clear line between settled and open questions, and reports no new experiments.
Who it's for and tradeoffs
Great fit if you research LLM reasoning, distillation, or on-policy training and need a conceptual framework to diagnose degeneration modes and design mitigations. Use it to reason about experiment design (what supervision to expose, how to weight tokens, schedule teacher guidance). Look elsewhere if you need empirical benchmarks, new algorithmic recipes with experimental validation, or implementation guidance—this work organizes and critiques existing literature rather than presenting new experimental results.
Where it fits
Positions OPSD relative to on-policy distillation and RL: OPSD trades a larger teacher for privileged context in the same model, gaining token-level supervision and sample efficiency but introducing unique bias risks. The review highlights open research directions: rigorous ablations for each lever, metrics for diversity-preserving supervision, and practical schedules to avoid catastrophic narrowing of reasoning paths.