Most outcome-based RL for interactive agents trains from sparse episode returns, leaving little guidance for intermediate token-level decisions. SEED flips that gap into a source of supervision: the policy itself analyzes finished trajectories to extract reusable, natural-language skills, then distills the behavioral effect of those skills back into the policy as a dense on-policy supervisory signal that co-evolves with the policy distribution.
Key Findings
- Hindsight-skill extraction: Treating completed trajectories as analyzable artifacts yields compact natural-language skills that capture reusable workflows, decisive observations, or failure-avoidance heuristics — these are used as alternate contexts to re-score actions.
- On-policy distillation signal: The method converts the probability shift induced by skill-augmented contexts into a token-level distillation target, producing dense supervision aligned with the sampled trajectory distribution instead of relying only on sparse episode rewards.
- Self-evolving loop: Because the current policy both collects trajectories and plays the role of analyzer, skill generation and policy improvement bootstrap each other; as the policy improves, extracted skills become more relevant to new data.
- Empirical benefits: Across text-based and vision-based agentic tasks, the approach improves sample efficiency and generalization to unseen scenarios compared with outcome-only RL baselines.
Who it fits and trade-offs
Great fit if you train interactive LLM agents or multimodal agents on long-horizon tasks where trajectory-level rewards are sparse and you can afford additional compute for trajectory analysis and distillation. It is less suitable if you cannot reliably collect complete on-policy trajectories, if running extra analysis per trajectory is prohibitively expensive, or if you require pure offline policy distillation without online data collection.
How it works (brief)
SEED fine-tunes the policy to act as an analyzer that, given a finished trajectory, writes natural-language skills that summarize actionable patterns. During RL rollout the same analyzer generates skills from recent trajectories; the system re-scores sampled actions under ordinary vs skill-augmented contexts and converts the induced logit/probability differences into a dense token-level loss term that is optimized jointly with the outcome-based RL objective. The pipeline keeps the auxiliary supervision synchronized with the current trajectory distribution rather than using static hindsight heuristics.