AIAny
Icon for item

TuringSpace: Grounding Spatial Intelligence in Action with Interactive CAPTCHAs

Tests multimodal models’ spatial reasoning through screen-based clicks and drags. Its programmatically generated CAPTCHA tasks also form a training environment, with fine-tuning gains reported across seven out-of-distribution benchmarks.

Introduction

A model can describe a spatial scene correctly and still fail when it has to act on it. That gap matters for computer-use agents: spatial understanding is not just recognizing what is on screen, but predicting how an action changes the scene and revising a mistaken interpretation from feedback.

Key Findings
  • The benchmark makes the observe–act–correct loop central: 125 CAPTCHA tasks across five spatial abilities require screen clicks and drags, rather than multiple-choice answers or text-only commands.
  • Humans solve 94.1% of tasks; the best reported model solves 74.1%, while the best open-weight model solves 32.0%. The gap suggests that spatial interaction remains difficult even for strong multimodal systems.
  • Most model failures are attributed to misreading screenshots. Subsequent reasoning often reinforces an incorrect interpretation instead of correcting it, pointing to perception and feedback use—not simply a lack of verbal reasoning—as key bottlenecks.
  • Fine-tuning Qwen3.5-9B on generated task trajectories raises its success rate from 1.6% to 33.1%. The reported gains extend to all seven out-of-distribution benchmarks, including a 9.9-point improvement on real-world indoor videos in VSI-Bench.
How the Benchmark Works

Each task is executable: it can generate new instances, check solutions, and provide an oracle trajectory. This makes the benchmark usable as a repeatable training gym rather than a fixed set of test examples. The design is useful for studying whether models can ground spatial beliefs in actual screen actions and feedback, although results on synthetic CAPTCHA tasks do not by themselves establish reliable performance in unrestricted computer use or physical environments.

Who It’s For—and the Trade-off

Great fit for researchers studying multimodal spatial reasoning, interactive agents, or training through action-based tasks. Look elsewhere if you need a general-purpose computer-use benchmark or a broad measure of visual perception: the task set is deliberately focused on spatial CAPTCHA interactions, and the reported results are benchmark-specific.

Information

  • Websitearxiv.org
  • OrganizationsUniversity of Pennsylvania, University of Minnesota, University of Southern California, MBZUAI, University of Wisconsin–Madison
  • AuthorsXuwei Ding, Skylar Zhai, Linxin Song, Jiarui Feng, Dongqi Fu, Huajun Xi, Junnan Li, Jingcheng Liang, Jieyu Zhao, Yinglong Xia …
  • Published date2026/11/19

More Items

Coordinates visual and audio forensic agents to locate and verify signs of AI-generated video. Its dual-timescale training improves both the investigation strategy and the specialist inspectors.

A rendered object mask can look complete even when the selected 3D Gaussians omit occluded parts. VODA uses local depth clusters to supervise those Gaussians directly, and Removal-3D evaluates whether their removal matches the object’s expected 3D removal.

Unifies text, images, audio, and video for Chinese-English multimodal understanding and reasoning, with fast instruct and explicit thinking modes. The paper reports leading open-source results on several omni-modal video benchmarks.