Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models

Why this matters OPD is widely used to transfer capabilities from stronger teacher LLMs to smaller students by supervising trajectories sampled from the student. Yet prior evaluations focus narrowly on single domains or benchmarks close to training data. This paper isolates one generalization factor at a time to reveal when OPD truly transfers reasoning behavior versus when it merely fits the trained distribution.

Key Findings

• OPD transfers reasoning patterns, not just solutions: Training-problem difficulty has little effect — even problems the teacher never solves can help the student learn teacher-like reasoning. So what: supervision from OPD shapes the student’s internal inference style, reducing reliance on memorized answers.
• Origin matters for breadth of transfer: Same-origin teacher-student pairs move the student close to the teacher across languages (English→Chinese), reasoning horizons (short→long), and other domains; cross-origin teachers mainly improve performance on the trained distribution. So what: model lineage and pretraining/architecture similarity can be more decisive than raw teacher performance when aiming for broad generalization.
• Multi-teacher OPD is a double-edged sword: routing prompts to domain experts cannot fully confine each teacher’s influence, so combining experts produces a mixture-dependent seesaw among capabilities rather than independent composition. So what: naive expert routing can cause unexpected capability tradeoffs and requires diagnostic strategies.

Who it helps and tradeoffs

Great fit if you need to understand or deploy OPD for capability transfer across languages, reasoning horizons, or domains and want principled guidance on multi-teacher setups. Look elsewhere if you only care about short-term, within-distribution gains from offline distillation—OPD’s strengths lie in behavioral transfer rather than brute-force performance on a single benchmark.

Where it fits

The study complements OPD and model-distillation literature by emphasizing mechanism and practical diagnostics: it explains when OPD will generalize beyond training prompts and when teacher selection and origin considerations are more important than training-set difficulty.