A convincing action sequence can still leave the wrong spreadsheet or browser state. Evaluating the resulting files and application state makes completion measurable without requiring an agent to follow one reference trajectory.
Key Findings
- The 369-task Ubuntu suite spans desktop software, files, and multi-application workflows, testing more than browser navigation.
- In the original experiments, the best agent reached 12.24% success versus 72.36% for humans; these are historical baselines.
- Failures in screen grounding and application knowledge show why stronger text reasoning alone may not fix desktop control.
Who Should Read It
Great fit if you study computer-use agents and need reproducible, outcome-based evaluation. Look elsewhere if you need a lightweight text-only test: virtual machines and application-specific evaluators add setup costs, and the published benchmark focuses on Ubuntu.