AIAny
Icon for item

Video-Oasis: Rethinking Evaluation of Video Understanding

Provides a diagnostic suite that audits video-understanding benchmarks to find samples solvable without visual or temporal input, filters those shortcuts, and produces a distilled video-native testbed that reveals major capability gaps in current Video-LLMs.

Introduction

Most video-understanding benchmarks mix visual reasoning with linguistic shortcuts and dataset priors, so high scores can hide whether a model truly understands video. This paper argues that evaluation criteria matter more than yet another benchmark: by systematically auditing existing datasets the authors strip away non-visual shortcuts and expose the real, often-missed challenges for Video-LLMs.

Key Findings
  • A large portion of benchmark samples (reported ~55%) are solvable without using visual frames or temporal context — meaning many benchmarks overestimate visual understanding. This implies simple text/context priors can dominate reported performance.
  • After filtering shortcuts, the remaining video-native examples produce a substantial capability gap: state-of-the-art models perform only marginally above random guessing. This provides a clearer failure signal for future model work.
  • The distilled challenge set serves as a controlled testbed to probe which architectural or data-design choices improve robust video understanding, enabling more informative ablations than mixed-purpose benchmarks.
Who it's for and trade-offs

Great fit if you build or evaluate multimodal/video LLMs and need a diagnostic, shortcut-resistant testbed to measure true video reasoning. It helps researchers prioritize model changes that improve temporal and visual perception rather than exploit dataset artifacts. Look elsewhere if you only need broad, high-level leaderboard scores or a very large-scale task benchmark — Video-Oasis focuses on diagnostic clarity over throughput and coverage.

Where it fits

Use Video-Oasis as a filter or sanity-check before reporting improvements on broader video benchmarks. Treat it as complementary: a small-to-moderate curated suite that surfaces where a model's apparent strengths are actually dataset shortcuts.

Information

  • Websitearxiv.org
  • AuthorsGeuntaek Lim, Sungjune Park, Jaeyun Lee, Inwoong Lee, Taeoh Kim, Dongyoon Wee, Minho Shim, Yukyung Choi
  • Published date2026/03/31

More Items

Distills a diffusion teacher’s conditional image distribution by matching exponentially tilted student means to teacher posterior means. It supports flexible sampling budgets on the trained timestep grid while preserving generation diversity.

Introduces a multimodal lens for tracing how vision-language models represent visual concepts and selectively access them in response to questions. Counterfactual tests examine how those concepts affect later reasoning; the work does not claim models are conscious.

Tests multimodal models’ spatial reasoning through screen-based clicks and drags. Its programmatically generated CAPTCHA tasks also form a training environment, with fine-tuning gains reported across seven out-of-distribution benchmarks.