Computer-use agents kept failing on one quiet problem: a model can read a screenshot, but it can't reliably say "this 18-pixel icon is the save button, click here." OmniParser attacks grounding rather than reasoning — it turns raw pixels into a structured list of interactable regions plus functional captions, so any vision LLM can act on coordinates instead of guessing.
Key Findings
- Grounding is the bottleneck, not the brain. Feeding GPT-4V parsed elements (screenshot-only, no HTML) beats GPT-4V baselines on ScreenSpot, Mind2Web, and AITW — the gain comes from better perception, not a bigger model.
- Two specialized models beat one general one. A detector fine-tuned on ~67K labeled screenshots finds clickable regions; a separate caption model trained on ~7K icon-description pairs explains what each does. Splitting detection from semantics is what makes small-icon grounding work.
- Model-agnostic plugin. It feeds Phi-3.5-V, Llama-3.2-V, GPT-4o and others; V2 pushed ScreenSpot Pro accuracy far above standalone GPT-4o while cutting caption latency ~60%.
Who It's For
Great fit if you're building pure-vision GUI agents or computer-use automation and need a drop-in grounding layer that works across LLM backends. Look elsewhere if your target app exposes a clean DOM or accessibility API — parsing pixels adds latency and error you don't need when structured metadata already exists.
