Most automation tools break the moment an app updates its layout, because they lean on brittle accessibility trees or hard-coded element IDs. This project makes a different bet: treat the screen as an image and let a vision-language model click what a human would, so one agent generalizes to apps it has never seen — across Android, Windows, macOS, and the browser.
What Sets It Apart
- Pure visual grounding: perception comes from screenshots, not the app's internal UI hierarchy, so it still works on closed apps and custom UIs where accessibility APIs return nothing useful.
- Open GUI-Owl models: the underlying 7B and 32B vision-language checkpoints are released on ModelScope and HuggingFace, so you can self-host instead of renting a proprietary computer-use API.
- A loop, not a single prompt: distinct planner, decision, reflection, and memory roles let it recover from misclicks on long-horizon tasks rather than dying on the first wrong step.
Great Fit If / Look Elsewhere
Great fit if you research GUI or computer-use agents, need an on-prem alternative to hosted computer-use, or want a base model to fine-tune for a specific device fleet. Look elsewhere if you want a turnkey consumer product: this is a fast-moving research family (v1 through v3.5) where interfaces shift between versions, and the Chinese-ecosystem demos (Bailian, ModelScope) are first-class while English packaging trails behind.
