Active, iterative re-inspection is fundamental to human vision, but standard vision–language benchmarks implicitly reward one-shot descriptions. This paper makes that missing capability measurable: ActiveVision forces models to perform repeated visual perception across tasks and shows a large, persistent gap between human active observation and today's MLLMs.
Key Findings
- Benchmark design: ActiveVision comprises 17 tasks in 3 categories crafted to require sequential observations and information gathering rather than a single static caption or label.
- Empirical gap: The best evaluated model (GPT-5.5 at its highest reasoning tier) solves 10.6% of items and scores zero on 11 of 17 tasks; Claude Fable 5 solves 3.5%; three human participants average 96.1%.
- Code-generation limitation: Allowing models to write and run vision-processing code improves some cases but remains unreliable on realistic imagery, and detecting those failures itself requires active perception.
What This Means
Active visual observation is not an emergent capability in current frontier MLLMs at useful reliability levels. Benchmarks and training objectives that assume single-pass perception can overestimate model visual understanding. Closing the perception–reasoning loop likely requires architectural and training changes that explicitly support iterative inspection, verification, and tool-use monitoring.
Who Should Care & Trade-offs
Great fit if you design or evaluate multimodal systems where repeated inspection, verification, or question-driven vision is required—for example, robotic perception pipelines, visual QA with follow-up queries, and safety-critical inspection tasks. Look elsewhere if your use case is genuinely one-shot description (simple captioning or static labeling), since ActiveVision targets sequential perception and will be overly pessimistic for strictly single-pass benchmarks.