Fine-grained visual perception is often split between language models that describe images and specialist systems that mark exact pixels. The key idea here is to make both levels part of one prediction process: the model emits structured object references and coordinates, then uses an instance token to trigger dense pixel output. This links language-level queries to masks and alpha mattes without handing the task off to a separate large segmentation model.
Key Findings
- One interface covers object detection, referring localization, counting, pointing, and OCR alongside segmentation and matting. This makes it possible to ask about a target and receive the spatial representation suited to the task, from a point or box to a mask.
- Dedicated coordinate tokens encode normalized positions, while a lightweight pixel decoder produces dense outputs. The decoder has about 9.6 million parameters, keeping pixel prediction as an attached component rather than a second large vision model.
- The report describes 2B and 4B model variants built on Qwen3-VL, trained on roughly 100 million images through three SFT stages followed by mixed-task reinforcement learning. It reports evaluation across more than 30 benchmarks and includes rejection of queries for absent objects.
Method and Trade-offs
The model first predicts an object reference and its coordinates. For pixel-level tasks, an instance token supplies a latent representation to a decoder that combines the predicted box with multi-level visual features; matting adds refinement for soft boundaries. This design gives dense outputs spatial guidance from the same prediction that identifies the object. The trade-off is that fine alpha boundaries remain sensitive to resolution, occlusion, transparency, and prompt wording. The pixel decoder currently targets single-image perception rather than video.
Who It’s For
A good fit for research that needs a shared interface for visual grounding, OCR localization, object counting, referring segmentation, or image matting. Look elsewhere if the priority is video perception, highly reliable fine-edge matting under difficult imaging conditions, or a general-purpose conversational assistant rather than structured visual outputs.