Computer-use agents can only improve as fast as their interaction data lets them. The central insight here is to make that data flywheel draw on real software workflows, rather than relying only on an agent’s own generated experience, then use model weaknesses to steer what gets collected next.
Key Findings
- Instructional videos provide authentic workflows; visual and transcript cues jointly locate task boundaries, helping reconstruct trajectories from noisy demonstrations.
- Extracted actions become reusable primitives that can be recombined into tasks with controllable complexity. This turns a finite set of demonstrations into a broader, testable training curriculum.
- A snapshot-and-rollback retry mechanism supports more reliable trajectory collection in real environments, while task-specific verifiers provide outcome-based rewards for reinforcement learning.
- The reported 27B model scores 80.2 on OSWorld Verified, 36.9 on OSWorld 2.0, 43.9 on WeaveBench, and 25.0 on Agents’ Last Exam. Results span GUI-only and GUI-plus-CLI settings, suggesting the data pipeline benefits more than a single interaction style.
Method and Trade-offs
The framework nests two feedback loops: open-world seeding supplies new workflows, while primitive-based propagation amplifies their coverage and difficulty. An outer loop uses observed model weaknesses to guide subsequent data construction and training. This is a useful distinction from recursive pipelines that mostly recycle self-generated experience. The cost is substantial infrastructure and compute for environment rollouts and iterative optimization; data quality also depends on capable annotator models.
Who It Suits
A strong fit for researchers studying computer-use agents, data synthesis, or iterative agent training who can support large-scale sandboxed evaluations. Look elsewhere if you need a lightweight training recipe or a low-infrastructure benchmark result: the method’s gains depend on costly environment interactions and annotation capacity.