Most online RL fine-tuning methods for diffusion models trade off efficiency and stability: forward-process approaches (e.g., DiffusionNFT) are sample-efficient but unstable without careful EMA tuning, while reverse-process approaches (e.g., Flow-GRPO) suffer poor credit assignment. The core insight of this paper is that NFT-style objectives hide a forward–reverse consistency force that joint positive/negative training only partially cancels, and that EMA stabilizes training by providing a restoration force rather than via off-policy rollouts.
Key Findings
- Introduces DiffusionART (Aggregation-First Reward Tilting): aggregate each prompt's generated samples and rewards into a shared reward-improving direction G(c), then regress visited reverse-process actions toward guided targets derived from G(c), eliminating the implicit forward–reverse consistency force by construction.
- Adds a trust-region–masked second update per rollout that restores states proportional to their drift, replacing EMA and removing the need for per-task EMA schedules.
- Requires no differentiable reward model, supports pairwise reward models (DiffusionART-Pairwise), and remains effective with fewer samples per prompt.
- Empirical gains on image and video generation: on HunyuanVideo-13B with equal visual and motion quality rewards, reaches late-stage mean rewards of DiffusionNFT and Flow-GRPO-Fast with 12.02× and 11.72× fewer GPU hours, and achieves 28.8% and 53.0% higher mean training rewards respectively.
Who it's for and tradeoffs
Great fit if you want a reinforcement approach to directly optimize black-box quality metrics for pretrained diffusion models (images or videos) with better sample/GPU efficiency and fewer per-task hyperparameters. Look elsewhere if you need a turnkey training pipeline with minimal engineering overhead—DiffusionART introduces new update mechanics (aggregation + trust-region restoration) that require integration into reverse-process rollouts and careful reward design. It also assumes access to meaningful per-prompt reward signals (including pairwise comparisons) rather than perfect differentiable critics.
Where it fits
DiffusionART occupies a middle ground between forward-process NFT-style optimization and reverse-process policy-gradient methods: it keeps NFT’s supervised-style velocity regression intuition but moves updates into reverse rollouts with an aggregated, explicit reward direction and a drift-adaptive restoration that removes EMA tuning. This makes it particularly attractive for large-scale generative fine-tuning where stability and GPU cost are primary concerns.