Training task-specific teacher models for on-policy distillation incurs high compute and can propagate teacher biases into student trajectories. Self-OPD sidesteps that dependency by converting the student’s own local stochastic exploration into dense, stepwise supervision: at each denoising step it samples K SDE branches from the student, rolls them to terminals, scores them versus a deterministic ODE baseline, and uses normalized advantages to shape the velocity field.
Key Findings
- Teacher-free per-step supervision: Replaces teacher velocity targets with advantage-weighted velocity targets discovered via the student's local branches, avoiding the cost and bias of training specialized teachers.
- All-branch pull–push objective: High-advantage branches attract the velocity field while low-advantage branches repel it; direction-aware attenuation and SDE-variance normalization stabilize learning and connect the loss to a reward-tilted KL interpretation.
- Reward-level fusion for multi-objectives: Normalizes and fuses scalar rewards before ranking branches, avoiding direct gradient conflicts across objectives and enabling a single image to satisfy multiple metrics simultaneously.
- Empirical gains: On single- and mixed-reward benchmarks, Self-OPD matches or outperforms prior RL and teacher-based OPD approaches while remaining teacher-free.
Who It's For and Tradeoffs
Great fit if you work on aligning flow-matching or continuous denoising models to black-box, possibly heterogeneous rewards and want dense, stable per-step supervision without training task-specific teacher networks. Look elsewhere if your priority is minimal sampling overhead at each step—Self-OPD requires K stochastic branches and deterministic rollouts per timestep, which raises per-step compute and evaluation costs compared to purely terminal RL approaches.
Where It Fits
Positions itself between teacher-based OPD (which provides dense supervision but needs pretrained experts) and terminal-policy RL (which is sample-inefficient and sparse): Self-OPD keeps dense, on-policy feedback while removing external teachers, trading extra local sampling for reduced dependence on specialized teacher models and better multi-objective joint optimization.