AIAny
Icon for item

Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

Designs and evaluates a foundation GUI agent that performs cross-platform GUI and CLI actions on real devices to complete long-horizon workflows. Emphasizes a unified action space, a large-scale real-device mobile runtime, an AutoResearch-style data flywheel, and online RL training across 10,000+ concurrent environments.

Introduction

Most GUI research focuses on simulated or narrow tasks; this paper argues real-world utility requires agents that operate reliably on real devices, interleave GUI and CLI actions, and sustain long-horizon stateful workflows. The core insight is that combining diverse sandboxing with a large-scale real-device runtime and an AutoResearch-style loop lets a single foundation agent move from lab benchmarks toward practical, cross-platform automation.

Key Findings
  • Real-device performance: achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily, showing strong mobile-use competence compared to prior agents. On desktop/browser tasks it records 79.5% on OSWorld-Verified, a 40.0% partial-progress score on OSWorld-v2, 73.6% on WebArena, and 81.5% on ScreenSpot-Pro for GUI grounding.
  • Architecture and training: uses a unified action space that interleaves GUI operations with CLI execution and supports batched multi-step actions per model turn; trains with online RL on trajectories exceeding 100 turns and scales rollouts with over 10,000 concurrent environments.
  • Data and iteration loop: an AutoResearch-style data flywheel has agents generate tasks/environments, diagnose failures, and plan iterations, enabling automated dataset and environment growth without fully manual annotation.
  • Comparative positioning: sets state-of-the-art on mobile benchmarks and is competitive on computer/browser tasks against frontier models referenced in the paper (Opus 4.8, Gemini 3.1 Pro, GPT-5.6 Sol).
Who It's For and Trade-offs

Great fit if you are evaluating foundation agents for real-device mobile automation or building cross-platform GUI+CLI workflows and can provision a device fleet and RL infrastructure. Look elsewhere if you need a lightweight, on-device-only agent or a solution for highly specialized GUIs without access to large-scale runtime and rollout capacity—the approach depends on substantial infrastructure (real-device runtime, parallel environments, and RL compute) and may require adaptation for proprietary or rapidly changing UI ecosystems.

Where It Fits

This work sits between benchmark-focused GUI agents and end-to-end automation platforms: it emphasizes research-forward iteration (data flywheel + RL) while demonstrating production-scale evaluation on real devices. Expect the most value for research teams and engineering groups that can run large-scale experiments and curate real-device fleets.

Information

  • Websitearxiv.org
  • AuthorsHanzhang Zhou, Panrong Tong, Xu Zhang, Quyu Kong, Chenglin Cai, Tianyu Xia, Gongjie Zhang, Jianan Zhang, Long Li, Long Chen
  • Published date2026/07/30

Categories

More Items

Converts text prompts into physically consistent videos by synthesizing executable Blender programs as a process-level chain-of-thought and using a dual-engine pipeline (deterministic simulation draft + draft-conditioned video editor). Ships with a VideoCoCo-3K draft–instruction–target dataset and shows substantial gains in physical-consistency benchmarks.

Presents Metis, a prototype memory foundation model that embeds a persistent native memory state into the backbone so historical experience is compressed and accessed via memory attention. Key features: forward-only, gradient-free online memory updates; memory-specific mid-training objectives; and a dual text/code memory design.

Evaluates whether vision-language models can make actionable decisions for a physical body by decoupling decision-making from low-level motor execution. Introduces HumanCLAW-Bench with 1,218 long-horizon egocentric episodes across 41 indoor scenes and diagnoses a lack of embodied self-awareness in current VLMs.