A model that chooses actions poorly may still be a good teacher: acting requires anticipating consequences, while judging can use consequences already observed. This study shows that these roles reward different abilities, so actor scores alone can miss a decision model’s value for agent training.
Key Findings
- Across three decision models and three general-purpose LLMs on ALFWorld, actors tended to abandon a prerequisite subgoal for an easier one. The failure points to a challenge specific to long-horizon tasks: locally attractive actions can derail progress toward a necessary later outcome.
- When shown the full trajectory, the same models could credit actions that advanced the prerequisite subgoal. Asking whether an individual action helped was more useful than asking whether the task would ultimately succeed, separating action quality from the outcome of the whole episode.
- Used as a teacher, JEV improved a Qwen2.5-1.5B policy trained with GRPO from 72.8% to 82.9% success on ALFWorld, and from 56.8% to 71.4% on WebShop. The gains suggest that post-episode evaluation can provide learning signals that a model’s own online action choices do not reveal.
Who Should Read It
A useful read for researchers building long-horizon agents or comparing decision models for action selection, evaluation, and policy supervision. The results are promising but come from a limited set of models and two environments; they do not establish that weak actors will generally make strong teachers, or that the same gains transfer to other tasks.