Why this matters Multimodal LLMs often struggle to assign probability specifically to tokens grounded in the visual input rather than generic, language-only completions. VCSD reframes that problem: instead of relying on an external teacher or privileged labels, it contrasts teacher outputs conditioned on the original image versus a content-erased control to highlight tokens whose likelihood is driven by instance-level visual content. That contrast becomes the training signal used to sharpen and distill a full-token distribution back into the student.
Key Findings
- Contrastive on-policy self-distillation: At each student-generated prefix, an EMA teacher computes two next-token distributions—one with the original image and one with a content-erased control. The token-wise log-probability difference identifies candidates specifically boosted by the image, which are then used to reweight and sharpen the teacher distribution before distillation.
- No external supervision or inference overhead: VCSD requires no external teacher model, privileged answers, extra visual-evidence annotations, reasoning traces, or additional inference-time cost for deployment.
- Empirical gains on multimodal LLMs: On the ViRL39K dataset, VCSD consistently improves Qwen3-VL benchmarks: for example, 2B model aggregate 62.27% → 67.04%, 4B 71.30% → 73.16%, 8B 72.51% → 76.26% (reported by the authors), showing the method scales across model sizes.
Who it's for and trade-offs
Great fit if you are fine-tuning or distilling multimodal LLMs and need a simple, data-driven way to strengthen visual grounding without collecting extra labels or maintaining a stronger external teacher. It is especially relevant when you can generate or obtain content-erased control images and run EMA-teacher-based training loops. Look elsewhere if you cannot produce reliable content-erased controls for your images, need methods that explicitly output interpretable visual evidence, or if minimizing training-time complexity (EMA teacher bookkeeping and additional forward passes during training) is a higher priority than avoiding external supervision.
How it works (brief)
The core mechanism is contrastive: by subtracting the teacher's log-probabilities on a control image from those on the original image, VCSD highlights token candidates whose probability gain is attributable to instance-level visual content. The teacher's original-image distribution is then sharpened within its plausible support using that contrast signal, and the resulting full-distribution target is distilled into the student via standard distributional distillation losses. The pipeline was evaluated on ViRL39K and applied to Qwen3-VL and Qwen3.5-family models in the paper.