Why this matters
Frontier-scale post-training (e.g., verifier-driven RL) is expensive to repeat on every new large model, and conventional distillation can lock a student into a weaker teacher's capacity ceiling. The central insight of this paper is that the useful supervision from weaker teachers can be extracted as a teacher policy shift relative to a reference checkpoint, evaluated on the student's own on-policy states, and then used to amplify only those verifier-supported gradient directions that help the student — accelerating progress without forcing the student to imitate the weak teacher.
Key Findings
- Extracting the teacher's policy shift (difference in logit/policy space between post-RL and pre-RL checkpoints) and evaluating it on student rollouts yields a dense supervision signal that can be transferred across scales. This converts weak-model RL outcomes into a reusable signal for stronger students.
- On-Policy Reverse Distillation (OPRD) rescales verifier-driven policy-gradient updates only along components supported by the teacher's shift; this preserves the stationary points of the underlying policy objective while boosting learning speed beyond the teacher's final policy.
- Empirically, OPRD attains higher performance with fewer student updates than prior RL and distillation baselines in successive-model transfer and multi-teacher distillation, and it continues to improve students even when supervision sources are weaker overall.
- Response-style analysis indicates OPRD-trained students stay closer in behavior to models trained with verifier-based RL alone than to their weak teachers, suggesting teacher guidance primarily accelerates the student's own optimization rather than redirecting it.
Who it's for & trade-offs
Great fit if you want to reuse cheap RL runs on smaller models as supervision for larger models (e.g., accelerate post-training of a new generation without rerunning expensive RL), or if you need a practical way to compose multiple weak supervision sources. OPRD reduces compute by avoiding sparse-reward RL on the target and by needing fewer student updates.
Look elsewhere if your verifier signal is unreliable or if teacher and student have fundamentally incompatible "thinking patterns"; OPRD relies on verifier-supported signals and paired teacher/reference checkpoints, and it can amplify spurious directions if the verifier does not correctly identify desirable behaviors. It also assumes access to both pre- and post-RL teacher checkpoints (or equivalent contrast pairs) to form a meaningful policy shift.
Where it fits (brief)
OPRD sits between imitation-style distillation and full sparse-reward RL: instead of copying a weaker teacher's final policy, it reuses the weak teacher's RL-induced policy shift as an implicit reward-like signal evaluated on the student's own states, combining verifier-based objectives with teacher guidance to speed up convergence without imposing the weak model's capacity ceiling.