A model can hear a sound that is absent from the video—or miss speech because the visual scene suggests silence. The deeper issue is not always faulty decoding: evidence from one modality can be entangled with learned priors inside the fused representation. This work intervenes at that point, aiming to restore evidence the model underweights rather than merely editing its final answer.
Key Findings
- Four complementary views of a frozen model diagnose which modality the question depends on and which stream is competing with it. That makes the intervention conditional on the actual conflict, rather than applying the same correction to every answer.
- Orthogonal decomposition separates grounding evidence from competing priors in hidden states; targeted attention-head and residual-state rebalancing then suppresses misleading priors and restores ignored evidence before decoding. No model parameters are changed.
- With Qwen2.5-Omni-7B, the method raises accuracy by 12.9 points on CMM and 8.6 on AVHBench. It also reports gains of 0.67–2.25 points across four general omni-modal reasoning benchmarks, suggesting the correction does not only help targeted hallucination tests.
- The approach transfers to VideoLLaMA2-AV, with gains of 4.6 points on CMM and 3.9 on AVHBench. This supports cross-architecture applicability, though the evidence is limited to the two evaluated backbones.
How to Read the Contribution
The key distinction from output-level calibration is where the correction happens: OWP diagnoses a modality conflict, decomposes evidence and priors in the model’s representations, and rebalances those representations before generating an answer. This makes it a targeted inference-time intervention, not a new trained model or a general-purpose guarantee against hallucination.
Who It Suits—and the Trade-off
Great fit for researchers evaluating audio-visual grounding or exploring training-free interventions in frozen omni-modal models. Look elsewhere if you need a drop-in fix for arbitrary LLMs, text-only systems, or a solution that requires no additional inference work: the reported method depends on model-specific intervention points and was tested on two backbones.