Most RL fine-tuning for video MLLMs trades off reliable supervision (long chain-of-thought oracles) against on-policy exploration, making post-training slow and data-inefficient. The surprising observation behind this paper is that annotations can serve as direct positive rollouts if integrated without corrupting on-policy advantage estimates; naive mixing raises the baseline and inverts advantage signs, which suppresses useful exploratory rollouts.
Key Findings
- Core idea: serialize each annotation as an oracle rollout but exclude the oracle from the on-policy baseline. This decouples policy-relative advantages (computed among on-policy rollouts) from oracle guidance (applied as a detached optimization target).
- Optimization mechanics: introduce a decoupled advantage estimator that yields two update terms — a directional gain that amplifies above-average policy rollouts and a detached oracle update whose weight decays as the oracle-policy gap closes.
- Efficiency tricks: sign-balanced pruning retains the oracle plus strongest rollouts of each sign, followed by moment correction; selective backprop through this subset reduces step time and preserves contrast. Reported runtime: OraRL needs ~2.2× SFT step time (vs 4.9× for GRPO with CoT) and selective backprop gives ~1.48× speedup over full-group updates.
- Empirical scale and gains: Video-ORA models trained with OraRL improve consistently from 0.8B to 9B and across data budgets up to 100k prompts. Example improvements include higher temporal mIoU, tracking AO, segmentation scores, and a VSI-Bench average reported at 73.1 for Video-ORA-9B.
Who it helps and trade-offs
Great fit if you train or fine-tune video-capable multimodal LLMs and need better RL sample efficiency without relying on expensive chain-of-thought decoding; useful when annotations can be serialized into model responses. Look elsewhere if your setup cannot generate on-policy rollouts or if your objective demands full-group gradient updates without selection (OraRL uses pruning and detached updates, which introduce a different optimization dynamic). OraRL emphasizes preserving exploratory signal while giving annotations a direct positive role — that trade-off is by design and may need tuning for new task families.
Method sketch
OraRL (1) appends the serialized annotation as an oracle rollout to each on-policy group, (2) computes policy advantages using only on-policy rollouts to avoid advantage inversion, (3) encodes the oracle-policy gap into a directional gain and a separate oracle advantage, and (4) applies sign-balanced pruning plus post-selection moment correction to accelerate training without biasing updates.
Overall, the paper reframes annotations from mere scorers to usable oracle rollouts and supplies practical algorithmic components to maintain exploration while harvesting strong supervised signals, delivering measurable speed and metric gains for unified video perception post-training.