Most computer-use agents treat screenshots as the canonical observation—and pay for it: pixels hide the true program state (files, DOM, application backends), creating ambiguous perceptions and expensive perception bottlenecks. StateAct reverses that assumption by making program state the primary grounding for action, memory, and verification, pushing the system bottleneck from perception toward reasoning.
Key Findings
- State-first orchestration: the main agent acts by executing and reasoning over program state (code-driven actions) rather than selecting pixel coordinates, with a dedicated GUI subagent reserved for 28 of 108 tasks and only 1.1% of main-agent steps. This reduces reliance on costly screenshot-based perception.
- Measured gains on OSWorld 2.0: when driving Claude Opus 4.8, StateAct raises binary success from 20.6% to 26.9% and partial success from 54.8% to 61.6%, while delivering ~9× lower cost per task than a screenshot-driven pipeline. A code-only variant (no GUI subagent) achieves 45.9% partial success, below the screenshot baseline, showing the hybrid state+GUI balance matters.
- Verification and robustness: an independent finish gate checks structural correctness (missing output, unsaved files, wrong paths), catching failures that pixel checks miss and enabling safer long-horizon plans.
- Context management: to stay reliable over hundreds of steps, the main agent delegates subgoals to fresh subagents, keeping each agent's context compact and reducing compounded reasoning errors.
How it works
StateAct is a code-first, multi-agent harness. The main agent inspects and modifies program state via executable code or APIs; the GUI subagent performs screenshot-and-click interactions only when required. A finish-gate verifier reads saved results to ensure structural correctness rather than relying on visual checks. The architecture emphasizes state-grounding: actions, memory, and verification are anchored to machine-readable state rather than pixel observations.
Who it's for and tradeoffs
Great fit if you build or evaluate agents that operate on desktop/web applications and can access program state or APIs—especially long-horizon workflows where verification and low-cost repeated interactions matter. Look elsewhere if you target closed GUIs with no programmatic state access, purely vision-only benchmarks, or extremely resource-constrained environments where integrating program-state access and verifiers is infeasible. State-grounding improves correctness and cost but shifts failure modes toward reasoning and state-modeling mistakes rather than perception errors.