Spatial tasks often use locations, regions, and paths that are more naturally expressed by pointing, marking, or drawing than by discrete coordinates or text. This mismatch makes standard benchmarks favor text-output VLMs even when image-generation models can externalize spatial judgments directly in pixels. The paper argues that to fairly measure spatial cognition in generative-pixel systems, we must evaluate them in the modality they naturally use and provide a protocol to translate pixel answers into the existing metric space.
Key Findings
- ProVisE framework: defines protocol-constrained visual answer formats and an automated parser that converts pixel responses into structured predictions, so image-generation outputs can be scored by standard benchmarks.
- SpatialGen-Bench: a diagnostic suite of 470 samples across 14 spatial subtasks and four capability levels, enabling systematic comparison between image-generation models and text-output VLMs.
- Empirical result: image-generation models are competitive on tasks where spatial answers can be externalized as pixels, while text-output VLMs outperform on compositional spatial reasoning, revealing complementary strengths.
- Agentic builder: an automated tool to construct and validate task-specific visual protocols, lowering the manual effort to adapt external benchmarks for pixel-space evaluation.
Who it's for and tradeoffs
Great fit if you want a metric-compatible way to evaluate spatial judgments produced directly as images (e.g., pointing, marking, drawing) or to benchmark image-generation models on spatial reasoning tasks. Look elsewhere if your target is purely symbolic/compositional spatial reasoning or if you require zero-ambiguity textual answers; ProVisE still relies on carefully designed visual protocols and parsers, which can introduce engineering overhead and modality-specific failure modes.
How it works (brief)
The pipeline constrains image-generation outputs with protocol templates, renders them via image models, then runs a parser to extract structured labels (coordinates, regions, paths) compatible with original benchmark metrics. The Agentic builder automates protocol generation and validation, and the authors validate the approach on six external spatial benchmarks in addition to SpatialGen-Bench.