Most RL fine-tuning for LLMs relies on learned value critics or per-step advantage estimates, which add cost and can be unstable on reasoning tasks. The core insight of this paper is that for autoregressive generation with terminal (verifiable) rewards, the Bellman equations let token-level advantages telescope into a trajectory-level residual that depends only on the terminal reward and the prompt-level expected reward—eliminating the need for intermediate value estimates.
Key Findings
- Critic-free reformulation: The authors derive a trajectory-level PMD objective that provably has the same unique optimal solution as the original advantage-based PMD on states reachable by the rollout policy. This means you can target the same KL-regularized optimum without training a value model.
- Practical token loss: By approximating the trajectory objective and using a binary KL approximation, they obtain a token-level loss whose per-token weight is a smoothed ratio of complementary token probabilities (a mismatch-correction weight). This replaces importance-sampling ratios and stabilizes training.
- Empirical gains: On Qwen3-30B-A3B-Base trained on a math dataset, BPO reaches a peak average accuracy of 50.5% across AIME 2024–2026, outperforming strong baselines (gains of 3.1–11.0 percentage points versus CISPO, GSPO, GRPO-ClipHigher, DPPO).
Who It's For and Trade-offs
Great fit if you fine-tune LLMs with verifiable terminal rewards (e.g., math problem verifiers) and want to avoid training an auxiliary critic or running costly per-step rollouts. The method reduces memory and critic-instability risks and can simplify pipelines for RL with verifiable rewards. Look elsewhere if your reward is dense or intermediate (non-terminal) across tokens, since the derivation relies on terminal rewards and initial-value estimation; also note BPO uses group-based sampling and approximations (binary KL, smoothing, clipping) that introduce hyperparameters and potential bias compared to exact PMD.