Long-horizon, multi-turn agentic tasks hide which few decisions actually determine outcomes because verifiable rewards are sparse and delayed. AgentOPSD turns sparse outcome supervision into interpretable turn-level credit by aggregating token-level teacher–student log-probability gaps into turn evidence and recursively updating a Bayesian belief state in log-odds, allowing standard policy updates to spotlight pivotal turns without adding a critic or extra rollouts.
Key Findings
- Recursive Bayesian aggregation: token-level teacher–student log-prob gaps are summed into turn evidence and used to update a belief state in log-odds; the marginal belief revision between consecutive states identifies pivotal turns.
- Critic-free integration: the method yields turn-level advantage reweighting that plugs into standard policy optimization (e.g., GRPO) without requiring an auxiliary critic or additional environment interactions.
- Empirical improvements: outperforms GRPO and strong self-distillation baselines on ALFWorld, WebShop, and Search-QA; reported 89.1% success on ALFWorld with Qwen2.5-7B. Ablations attribute gains to turn-level aggregation and history-dependent recursion.
- Practical trade-offs: uses teacher–student log-probabilities as supervision, so it leverages privileged teacher signals rather than changing rollout dynamics.
Who it's for & trade-offs
Great fit if you train LLM-based agents on long-horizon, multi-turn tasks where final outcomes are verifiable but intermediate credit is unclear, and you can evaluate a teacher policy to obtain token-level log-probabilities. It is especially relevant when you want to keep rollout behavior unchanged and avoid learning a separate critic.
Look elsewhere if you cannot run a teacher model to supply log-prob gaps, if environment rewards are already dense and informative, or if you need methods that explicitly learn value functions for off-policy correction. Computational overhead is mainly teacher evaluations and bookkeeping of recursive belief updates; no extra environment samples are required.
Where it fits
Compared with contemporaneous self-distillation approaches, AgentOPSD emphasizes a recursive, history-dependent Bayesian belief in log-odds to produce turn-level marginal revisions. That contrasts with methods that localize supervision to action spans or apply alternate Bayes-calibrated scores: AgentOPSD’s recursion explicitly models how each turn changes posterior belief about the final outcome, yielding principled reweighting for policy updates.