AIAny
Icon for item

UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

Trains a foundation GUI agent using a closed-loop, environment-grounded data stack plus in-context multimodal demonstrations to automate long-horizon desktop workflows. Combines scalable task generation/verification, subtask-level demo guidance, and a 100-task OSWorkerBench benchmark to improve strict success and task progress.

Introduction

Long-horizon GUI automation fails in practice because training data are scarce, biased, and often omit tacit user conventions; that makes execution brittle even for strong base models. The paper's core insight is that coupling a scalable environment-grounded training pipeline with in-context, subtask-level demonstrations yields much higher reliability: demonstration guidance raises strict success and task progress substantially on long multi-step office tasks.

Key Findings
  • Strong empirical gains: the 27B agent variant achieves 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena, and on the new OSWorkerBench (100 long-horizon office tasks) it reaches 41.0% strict success and 76.9% progress—material improvements over its Qwen3.6-27B base (±17.7 and ±24.5 points).
  • Demonstration impact: providing one successful, closely related demonstration on a 33-task self-demo subset improves strict success from 17.2% to 35.4% and progress from 67.9% to 81.1%, showing in-context demos help bridge unstated conventions and tool-specific details.
  • System design: a closed-loop data engine automates task generation, environment construction, rollout, filtering, capability balancing, SFT and online RL across parallel environments; an in-context demonstration mechanism converts multimodal traces into reusable, subtask-level workflows and supports live replanning.
Who it's for & Tradeoffs

Great fit if you research or build multimodal GUI agents, need a reproducible benchmark for long-horizon desktop automation, or want to evaluate how demonstrations improve open-weight foundation agents. Look elsewhere if you need a lightweight on-device solution or prefer methods that avoid large synthetic rollout infrastructures. Expect substantial compute and engineering overhead: parallel environment orchestration, verifier tooling, and large-model SFT/RL training are central to the approach.

Where It Fits

Compared to prior GUI-agent work, this paper emphasizes a foundation-model stack + in-context demonstrations and a large, focused benchmark. It complements works that target memory-based experience reuse, few-shot demonstration parsing, or vision-language-action architectures by integrating scalable environment grounding, demonstration-guided execution, and an evaluation suite for long workflows.

Information

  • Websitearxiv.org
  • AuthorsZihan Ding, Longxu Dou, Qi Gao, Xiangwu Guo, Shengchao Hu, Zilong Huang, Zihang Jiang, Lei Ke, Mengcheng Lan, Weixian Lei …
  • Published date2026/08/16

Categories

More Items

Tests decision models as actors, trajectory judges, and reinforcement-learning teachers for long-horizon agents. It finds that a model can be a weak actor yet still improve policy learning by evaluating actions after seeing their consequences.

Lets AI agents read task-relevant evidence from external content instead of receiving entire documents by default. Intent tracking, adaptive readers, and an exposure budget reduce token use and latency while retaining a faithful full-view fallback.

Turns GTA V missions into automatically verifiable agent benchmarks and reinforcement-learning environments, with 150 navigation, interaction, and shooting tasks. Results expose persistent difficulty with dynamic control, while training on generated tasks improves some capabilities.