Gameplay data is unusually hard to scale: videos omit the actions that produced them, while successful play is difficult to verify. The paper’s key insight is to make environment construction part of data collection: players expose flaws in game interfaces and task definitions as they play, and those environments are repaired before their trajectories are used for training.
Key Findings
- The framework produces 568 accepted web-game environments and 3,436 tasks, with 15.7K trajectories totaling 307 hours. This provides broader coverage than a small set of hand-authored game tasks.
- A Builder exposes game state and defines measurable progress and success checks; a screenshot-playing VLM and a state-reading coding player test the same tasks and report problems. Their different observation interfaces catch different failures.
- Progress-weighted supervised fine-tuning gives greater weight to actions that move a task forward. The 9B model raises average progress on the held-out Player Bench from 14.3% to 31.2%, and on GameWorld from 30.0% to 53.1%.
- The results suggest that a useful training signal can come from verified progress, not just successful demonstrations. The external benchmark improvement is notable because it shares no games, tasks, or success checks with the collection.
Who It’s For—and the Trade-offs
A good fit for researchers studying visual agents, interactive control, and scalable training data, especially those who need task-level progress and verifiable outcomes. It is less suited to claims about real-time play: the VLM pauses the game while reasoning, and the collection focuses on offline, single-player browser games. Acceptance is not a guarantee of correctness; audits found that some generated tasks were implemented incorrectly.