Repetition of expensive reward-driven fine-tuning for each new, larger model becomes a major bottleneck as models scale. The core insight of Direct On-Policy Distillation (Direct-OPD) is to treat the policy change produced by running RL on a cheaper, smaller “weak” model as an implicit, dense reward signal that can be applied to a stronger target without re-running sparse-reward RL.
Key Findings
- Directly computes the log-ratio between a weak model's post-RL and pre-RL policies and uses that scalar as a dense reward for the student on the student's own on-policy states — so what: it captures which actions RL made more or less likely, abstracting away the weak model's capacity limits.
- Substantially faster and more compute-efficient than running RL on the target: example result shows Qwen3-1.7B improving from 48.3% to 58.3% on AIME 2024 in ≈4 hours on 8 A100 GPUs — so what: practical gains for large-model post-training without full RL rollouts.
- Outperforms step-matched direct RL and supports sequential composition of multiple policy shifts — so what: you can chain multiple weak-to-strong transfers to accumulate improvements.
Who it's for and tradeoffs
- Great fit if you need to transfer sparse-reward RL gains from cheap, smaller models to expensive large models, or you want to avoid running many target-model rollouts while preserving RL-derived behavior changes.
- Look elsewhere if the weak teacher is poorly aligned with target tasks or the RL improvements rely heavily on exploration patterns that do not transfer via log-ratio signals; the method assumes the teacher's policy shift encodes transferable preferences rather than model-specific failure modes.
How it works (brief)
Direct-OPD requires a paired checkpoint (pre-RL, post-RL) of a weak teacher. For states sampled on the student's on-policy rollout, it evaluates the teacher's probability ratio log(pi_post / pi_pre) for the action taken and uses that scalar as an implicit reward to train the student. This reframes the weak teacher's RL outcome as dense supervision mapped onto the student's state distribution, avoiding sparse-reward optimization on the target while preserving the direction of the RL-induced policy shift.