Most policy-shift distillation methods transfer only the endpoint change in a teacher’s token log-probabilities after RL, but that endpoint can hide whether early and late training steps changed local action preferences in consistent or opposing directions. Waypoint-OPD shows that a single intermediate checkpoint (a "waypoint") supplies temporal agreement evidence that can selectively weight endpoint shifts at student-generated prefixes, improving transfer without additional teacher training or verifier calls.
Key Findings
- Temporal gating improves on-policy distillation: Waypoint-OPD retains the centered initial-to-final log-probability shift only when the early (Initial→Waypoint) and late (Waypoint→Final) relative changes point in the same direction; it suppresses the endpoint signal when they oppose each other.
- Small computational overhead: the method adds one frozen-teacher scoring pass per student batch (the Waypoint checkpoint) and no extra trainable parameters or verifier calls beyond standard Direct-OPD.
- Empirical gains across scales and domains: evaluated with Qwen3-4B RL teachers and Qwen3-8B/14B/30B-A3B students on mathematics and code reasoning, Waypoint-OPD achieved the highest domain-average scores among compared distillation baselines; ablations show the gating is causally important.
- Simple, interpretable gate: compute centered segment vectors over top-k next-token candidates at each student prefix, use their (positive) cosine agreement as a gate g_t ∈ [0,1] with a small-norm fallback; g_t scales the endpoint shift applied to student token choices.
Who it's for and tradeoffs
Great fit if you want to transfer RL-discovered directional improvements from smaller RL-trained teachers to larger student models without forcing endpoint imitation, and you can afford one additional frozen teacher scoring pass. Look elsewhere if you cannot score an extra checkpoint per batch or if your workflow requires zero extra inference cost from frozen teachers. The method assumes availability of multiple checkpoints from the same RL run and focuses on weighting endpoint signals rather than providing new verification or reward signals.
How it works (brief)
At each student-generated prefix, Waypoint-OPD: (1) selects top-k candidate next tokens from the student; (2) scores these candidates under three frozen teacher checkpoints (Initial, Waypoint, Final); (3) constructs centered relative-preference segment vectors for Initial→Waypoint and Waypoint→Final and computes their cosine agreement c_t; (4) sets gate g_t = max(0, c_t) (with a low-norm fallback g_t=1) and multiplies the centered Initial→Final log-ratio signal by g_t as the dense on-policy supervision applied to the student. This preserves the endpoint transfer when temporal evidence supports it and suppresses potentially reversed local shifts.