Most privileged-view distillation mixes visual evidence with teacher priors, making it unclear which next-token corrections are truly supported by the image. VAD's core insight is to treat the change produced when adding or removing the visual evidence as a signed, intervention-derived proxy for the visual direction, and to reconstruct what to distill around the student distribution rather than naively matching the teacher.
Key Findings
- VAD computes a signed proxy u_t from counterfactual teacher queries (teacher with evidence present vs removed) using centered log-probability shifts, then projects the teacher correction onto that proxy to separate an intervention-aligned component and a proxy-unexplained residual.
- The method reconstructs a student-anchored target from the proxy-aligned component (raising supported candidates, suppressing refuted ones) while retaining a weak teacher-derived regularizer and splitting u_t into support/refutation branches for finer control.
- Evaluated on six fine-grained visual benchmarks at 4B and 9B student scales, VAD consistently outperforms direct privileged-view distillation and visual-advantage weighting, achieving Avg_6 = 78.32 (4B) and 79.93 (9B) and leading scale-matched alternatives by ~2–3 points.
- Gains are broad across tasks (detail localization, high-resolution perception, real-world recognition) and not driven by a single dataset; additional teacher views are required only during training, not at inference.
Who it's for & tradeoffs
Great fit if you train vision-language students with privileged-view teachers and need more precise visual-grounded corrections than source-mixed targets provide. It is most useful for fine-grained multimodal benchmarks and post-training distillation at scale. Look elsewhere if you cannot afford the extra training-time teacher queries (counterfactual views) or if simple advantage-weighting already meets your production constraints.
Where it fits
VAD is a post-training distillation technique positioned against Vision-OPD and decomposed OPD baselines; it prioritizes attributing which parts of a teacher correction are visually supported and reconstructing targets accordingly rather than only reweighting teacher signals.
Method detail (brief)
- For each student-generated prefix, query the fixed teacher with the relevant evidence present and removed; compute centered log-probability shifts to form u_t. 2) Project the original teacher correction onto the u_t proxy (regularized projection) to extract the intervention-aligned component and residual. 3) Reconstruct a target distribution anchored to the full-image student by shifting mass toward visually supported tokens and away from refuted ones; apply a light teacher regularizer. 4) Train the student on these reconstructed targets; at inference the student runs as usual with full images.