Frontier vision-language models often score well on driving-scene QA by exploiting 2D shortcuts (e.g., "lower in frame = closer") rather than reasoning about true 3D geometry. This dataset intentionally forces metric spatial judgments—distance bins, relative separation, left/right ordering, and heading—so benchmarks reflect geometric understanding instead of pixel heuristics.
What Sets It Apart
- Directly probes metric 3D concepts from a single RGB driving frame: tasks include absolute distance bins (0–20m, 20–50m, etc.), relative distance, pick-closer, left-right ordering, and heading by clock directions. This makes it different from generic VQA datasets that emphasize semantic recognition over metric reasoning.
- Paired numbered bounding boxes make questions explicitly reference objects, reducing ambiguity and allowing targeted error analysis (e.g., which category of spatial relation models fail on).
- Small, focused benchmark (under 1K samples) with high-quality driving images collected by PlusAI and annotated multiple-choice answers—designed for evaluation and targeted stress-testing rather than large-scale training.
Who It's For and Trade-offs
Great fit if you need a compact, interpretable test-suite to evaluate or debug a vision-language model's 3D spatial capabilities in driving contexts (distance estimation, lateral ordering, heading). Use it to surface failures that illusionarily high VQA accuracy can hide. Look elsewhere if you need large-scale training data, diverse geographic coverage, or non-driving indoor/outdoor scenes—the dataset is intentionally small and focused, so it is best used as a diagnostic benchmark rather than a primary training corpus.
