Why this matters
Value estimates from learned critics are widely used to reduce variance in policy optimization, but when those critics produce “flat” predictions across states that actually have sharply different empirical returns, advantage estimates become misleading and policy learning degrades. This paper pinpoints that failure mode (Value Flattening), analyzes its causes, and recommends a simple change in supervision that restores informative value signals without large computational cost.
Key Findings
- Empirical phenomenon: Monte Carlo continuations show sharp changes in true state values while learned critics remain relatively flat, a gap that grows with state-space size and temporal correlation. This mismatch harms advantage estimation and downstream policy updates.
- Root causes: an implicit variance penalty in standard critic loss and redundant updates from temporally correlated states with similar gradients, both of which bias the critic toward conservative, flattened predictions.
- Practical fix: SP³O (SParse Proximal Policy Optimization) applies the value loss to only a few well-separated states per response (e.g., three states), reducing redundant gradient signals and the implicit variance penalty.
- Empirical impact: on experiments with Qwen3-Base and controlled environments (including FrozenLake), sparse supervision markedly reduces Value Flattening and yields consistent improvements in learned policies across model sizes and evaluation suites.
Who it's for and tradeoffs
Great fit if you train or align LLMs with reinforcement learning and use token- or state-level critics (PPO-style): SP³O is a low-friction modification that preserves single-sample training throughput while improving value fidelity. It is especially relevant for long-horizon or sparse-reward tasks where critic noise can dominate the true signal. Look elsewhere if your setup already uses reliable multi-sample empirical baselines or critic-free methods with provably better variance (those alternatives may already avoid the specific failure mode). SP³O trades denser supervision for targeted supervision of fewer states per trajectory—this reduces redundant updates but may require tuning the spacing/selection policy for supervised states.