Most text-to-image RL approaches rely on supervised preference data or trained reward models. This paper shows a different lever: reuse a pretrained multimodal LLM's image–text alignment to rate images without any reward-model fine-tuning. The core idea is to measure prompt recoverability — how likely the MLLM would generate the original prompt given the image — and use that average image-conditioned prompt log-likelihood as a reward.
Key Findings
- SpectraReward is training-free: it uses a single image-conditioned, teacher-forced forward pass through an MLLM to compute reward, so no preference labels or reward fine-tuning are required. This simplifies pipelines and reduces reliance on labeled human preferences.
- Broad compatibility and empirical gains: evaluated across two diffusion backbones, three RL algorithms, nine reward MLLM backbones (4B–235B) from four families, and five out-of-distribution benchmarks; SpectraReward and Self-SpectraReward consistently improve generation quality and outperform prior MLLM-derived reward methods.
- Self-SpectraReward insight: letting the policy’s own understanding branch score its generation (a closed-loop setup) can match or beat much larger external reward models, highlighting that reward–policy alignment matters more than absolute reward-model scale.
- Not monotonic with model size: larger reward MLLMs do not always yield better RL outcomes, implying selection/compatibility of reward model and policy is crucial.
Who it's for and trade-offs
Great fit if you want to tune text-to-image models via RL without collecting preference labels or training extra reward models, or if you want a simple plug-in reward signal that leverages existing multimodal LLMs. Look elsewhere if you require human-aligned, safety-verified preference judgments (human labels remain the gold standard) or if your deployment constraints prevent using large pretrained MLLMs for scoring (compute and latency costs for large MLLMs can be nontrivial).
Where it fits
SpectraReward sits between heuristic automated metrics and fully supervised reward learning: it reuses pretrained multimodal capabilities to provide a principled, likelihood-based reward that scales across image generators and RL algorithms, offering a low-friction alternative for improving fidelity to prompts.
Method overview
Compute the average log-likelihood that the MLLM assigns to the original text prompt conditioned on the generated image (teacher-forced). Use that score as the scalar reward to optimize a diffusion-based image generator via standard RL algorithms. The Self-SpectraReward variant uses a unified multimodal model where the policy’s perception/understanding branch scores outputs from its generation branch, forming a closed feedback loop without external reward models.
