A browser agent can read every word on a page and still choose the wrong product when the decisive clue is in its image. The central insight is that web competence depends on visual grounding and reliable action selection, alongside language reasoning.
Key Findings
- All 910 tasks require visual understanding across three self-hosted websites, making visual evidence central to evaluation rather than optional context.
- In the paper's reported experiments, GPT-4V with Set-of-Marks reaches 16.37% success versus 88.70% for humans; this measures a 2024 baseline, not today's best systems.
- Numbering clickable elements improves GPT-4V over the accessibility-tree variant, suggesting that the action representation can matter alongside model capability.
- Checks inspect answers or the resulting website state, so fluent narration cannot substitute for completing the requested operation.
Who Should Read It
Great fit if you study multimodal browser agents and need reproducible tasks that expose perception and action failures. Look elsewhere if you need desktop-wide coverage or live-web reliability estimates: three controlled sites cover only a slice of computer use, and reproducing evaluations requires hosting the environments.