Why this matters: interaction on personal computers is shifting from pure GUI automation to agents that choose how to observe and act across visual and programmatic interfaces. The paper presents an open stack—OpenGCUA—that exposes agent design, a preliminary real-device harness, a cross-device SDK, and execution trajectories from multiple frontier models so researchers can study and iterate on system-level choices rather than black-box demos.
Key Findings
- Code execution as a first-class capability: letting the model generate and run code (shell, API calls, scripted GUI operations) lets intermediate results guide subsequent actions, reducing brittle batched GUI-only workflows and cutting reported steps per task by ~53% in their setting — so what: agents can leverage programming constructs (loops, conditionals, variable state) to handle complex, branching tasks more compactly.
- Harness-to-model decision transfer: shifting selected observation and task-state heuristics from fixed harness rules into the model improves flexibility and raises mean partial scores by +19.6 points on OSWorld-v2 — so what: moving some decision logic into the model reduces brittle rule engineering and enables model-specific strategies.
- Real-device harness and cross-device SDK: the release includes a macOS harness and a preliminary SDK that binds operations to explicit devices/apps/tabs, plus trajectories from eight frontier models — so what: this surfaces engineering challenges of deploying agents on real machines and provides a common API for multi-device tasks.
- Execution-efficiency differences across models: runs show large variance (e.g., one model using ~19.4× more reasoning tokens than another on macOS tasks) — so what: evaluation should consider not just success rate but real-device token/latency costs when comparing agents.
Who it's for and trade-offs
Great fit if you are a researcher or engineer exploring system-level design of autonomous computer agents, need reproducible trajectories from closed models for analysis, or want a starting harness/SDK to run agents on real macOS devices. Look elsewhere if you only need a polished end-user product or a trained model: OpenGCUA is a system release and dataset of trajectories, not a packaged, production-grade agent nor a trained open model. The harness is preliminary and macOS-focused; multi-OS, security, and privacy hardening remain future work.
Where it fits
OpenGCUA occupies the space between academic benchmarks and closed commercial demos: it opens the stack (design, harness, SDK, trajectories) to enable reproducible system research on agents that use both programmatic APIs and visual GUIs across devices.