A shared visual vocabulary could let perception and image synthesis use the same modeling machinery, much as language models reuse tokens across tasks. The central insight here is that discrete codes need not choose between semantic meaning and pixel detail: a frozen visual encoder anchors semantics, while lightweight modulation adds reconstruction cues before quantization.
Key Findings
- Grouped product quantization increases the number of representable code combinations while keeping the spatial sequence length fixed, making discrete representations more expressive without adding positions.
- A shared code stream supports reconstruction, classification, multimodal understanding, text-to-image generation, and editing. The paper reports competitive or stronger results than task-specific baselines across 12 understanding benchmarks, four generation benchmarks, and one editing benchmark.
- The dual-branch objective decodes the same quantized features into semantic representations and pixels, balancing high-level recognition with visual fidelity. Reported results include 79.47% ImageNet-1K linear-probe accuracy and an OCRBench score of 885 when paired with Qwen3-8B.
How the Representation Is Built
The tokenizer starts from a frozen pretrained vision encoder. A residual modulator combines features from multiple depths, then grouped codebooks quantize each spatial position. Separate semantic and pixel decoders train against the same discrete bottleneck in stages, reducing interference between the two objectives. The design makes the codes useful both as inputs to language models and as a route back to image generation.
Who Should Read It
A good fit for researchers exploring unified multimodal models, visual tokenizers, or discrete autoregressive generation. The main trade-off is that one representation must serve tasks with different goals, and the headline comparisons may use differing data, resolutions, prompts, and evaluation protocols. It is less directly useful if you only need a specialized model for one task or require a drop-in tokenizer without a broader modeling pipeline.