Precise, fast spatial grounding is a distinct perceptual requirement for closed-loop physical systems, and general-purpose vision backbones still struggle with tiny targets, clutter, and control-rate demands. GroundingPI reframes perception as language-conditioned generation of quantized visual primitives (points and boxes) and shows that a perception-native foundation improves transfer to action-heavy domains.
Key Findings
- A 4B-parameter grounding foundation that outputs quantized coordinate tokens unifying points and bounding boxes, enabling diverse tasks (grounding, referring, pointing, OCR, GUI/layout grounding).
- Outperforms 44 baselines on 34 grounding benchmarks (11 perceptual capabilities), averaging 73.68% versus GPT-6 Astra's 71.54%, demonstrating measurable perceptual gains at modest model scale.
- When used as a visual backbone, yields large downstream gains: up to 24.8% relative improvement on RoboTwin 2.0 OOD settings, improved sample efficiency on RoboCasa-GR1 (50% demos > baselines with 75%), and an average open-loop L2 error of 0.296 m on nuScenes.
- Training recipe combines multimodal + spatial pretraining, supervised fine-tuning, and RL with GRPO; ablations highlight dense grounding and OCR data as key contributors to transfer.
Who it's for and trade-offs
Great fit if you need perception optimized for real-world, action-oriented systems (robotics, autonomous driving) where precise spatial localization, low-latency grounding, and action-data efficiency matter. Look elsewhere if your primary goal is broad high-level reasoning or generative image/video synthesis rather than low-level spatial grounding; GroundingPI focuses on perceptual primitives and spatial accuracy rather than large-scale language reasoning.
Where it fits
Positions itself as a perception-first foundation model: smaller than some multimodal giants but tuned for spatial tasks and downstream action transfer. Useful as a visual backbone replacement when downstream tasks require precise, structured spatial outputs rather than free-form captions or dense semantic representations.
How it works (brief)
Architecturally couples a MoonViT-V2 visual encoder with a Qwen3-4B autoregressive language decoder and a learnable projector. Outputs share a vocabulary of semantic tokens, protocol markers, and 1,000 quantized coordinate tokens, enabling unified, language-conditioned structured generation of points and boxes. Training stages and task-specific localization/text-geometry rewards with GRPO refine spatial behaviors.