Why this matters Most vision–language spatial tasks have small, structured output spaces (a choice, a number in range, or a 2D point) yet are handled by autoregressive text decoders that pay decoding and parsing costs. MJev‑Spatial reframes spatial understanding as direct classification over supplied candidates, reusing pretrained query/key projections to score classes in one forward pass. That shift yields substantially lower latency and better benchmark performance while simplifying the readout pipeline.
Key Findings
- Classification-over-candidates: Treats each answer option, numeric bin, or image region as a class and scores them using the backbone’s pretrained attention projections, so inference requires no token decoding and returns class probabilities in one forward pass.
- Coarse-to-fine refinement: Uses multi-stage discretization for numeric and point prediction (up to two stages for numbers, three for points) with crop re-encoding between stages, so it achieves fine-grained outputs without evaluating enormous candidate sets at once.
- Training & throughput optimizations: Jointly supervises all refinement stages with teacher-forcing in a single forward pass and applies length-aware global batching plus cross-GPU load balancing, so training throughput increases significantly compared with naive batching.
- Empirical trade-offs: Built on Qwen3.5-4B and trained on 12.3M converted QA pairs, the approach achieves the highest average score across twelve spatial benchmarks and runs multiple× faster than autoregressive baselines, demonstrating a favorable accuracy–latency trade-off.
Who it's for and tradeoffs
Great fit if you need low-latency, structured spatial predictions (VQA-style spatial queries, distance estimation, or point placement) and prefer probabilistic, candidate-constrained outputs over free-form text. Look elsewhere if your application requires unconstrained natural-language explanations, continuous high-precision coordinates without discretization, or a different backbone/license; the method relies on discretizing outputs and on a pretrained multimodal backbone, and its accuracy depends on candidate/bin resolution choices.
Where it fits
Use MJev‑Spatial when model latency and robust structured readout matter (interactive robotics, embodied agents, real-time VQA). For tasks requiring explanatory text or open-ended multimodal generation, autoregressive VLMs remain more suitable.
Implementation notes (mechanism overview)
The core mechanism re-encodes candidate descriptions with the observation+question and computes class scores by reusing the final attention layer’s query and key projections—no extra prediction head. Numerical and point outputs are produced by selecting bins/regions and recursively refining selected areas. Conditional tree expansion supports multiple-point queries without retraining.
Overall insight: converting compact spatial outputs into classification decisions lets a pretrained multimodal backbone deliver faster, simpler, and often more accurate spatial judgments than coupling the prediction to text generation.