The expensive part of alignment can be generating fresh samples after every policy update. Separating generation from learning lets one batch support several improvements; the key question becomes how well the reward filter represents what people actually prefer.
Key Findings
- Grow creates candidate translations; Improve repeatedly filters and trains on that batch, amortizing generation costs.
- On IWSLT 2014, one Grow and four Improve steps reach 77.8 average reward versus 70.9 for supervised training; these are reward-model scores.
- Human raters also prefer the improved translations, but their ranking differs from the reward model: more reward optimization does not guarantee more human benefit.
Who Should Read It
Great fit if you are choosing between online alignment and reward-filtered self-training, especially with a usable translation scorer. Look elsewhere if you need evidence for general agent tasks: evaluation centers on translation, and repeated optimization can overfit the reward model.