Most on-policy distillation (OPD) assumes a teacher can recover a correct solution from any student prefix — an assumption that fails in practice because teachers often follow student errors rather than explicitly repairing them. CriticOPD reframes that failure: instead of only scoring, the teacher first generates a critique of an incorrect rollout, the student rewrites its solution using that signal, and the teacher then scores the revised output while keeping the critique as privileged context during supervision. This restores the teacher’s practical recoverability and produces a more corrective supervision signal.
Key Findings
- Critique-as-privileged-context: Conditioning the teacher’s scoring on a generated critique localizes errors and produces corrective gradients that encourage backtracking tokens (e.g., “Wait”, “Actually”), rather than merely amplifying high-scoring but erroneous continuations.
- Empirical gains on math reasoning: Evaluated with two Qwen3 student models across six mathematical reasoning benchmarks, CriticOPD improves on-policy distillation baselines, with the largest relative improvements on competition-style problems where recoverability failures are common.
- Better than scoring-only supervision: A scoring-only privileged teacher often fails to identify actionable error spans; allowing the teacher to generate a critique converts the scoring signal into one that can reliably guide rewrites and improve sampling-based solving.
Who it's for and tradeoffs
Great fit if you train or fine-tune LLM policies via on-policy distillation and need denser, error-aligned supervision for chain-of-thought or multi-step reasoning tasks. It is especially useful when student rollouts frequently contain recoverable errors (e.g., math competition problems) and when a frozen teacher can produce critiques.
Look elsewhere if your training setup cannot sample student retries or cannot expose privileged teacher context during scoring, or if deployment constraints forbid multi-stage critique-and-rewrite loops. CriticOPD adds complexity (critique generation, guided resampling, privileged-context scoring) and requires infrastructure to sample and evaluate rewrites under a fixed budget.