AIAny
Icon for item

Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

Reduces object-driven shortcut learning in zero-shot compositional action recognition by enforcing temporal verb cues and regularizing against frequent object-verb co-occurrence priors. Proposes RCORE with Co-occurrence Prior Regularization (treats frequent co-occurrences as hard negatives) and Temporal Order Regularization. Evaluated on Sth-com and EK100-com with improved compositional generalization.

Introduction

Current zero-shot compositional action recognition models often succeed by memorizing frequent object–verb co-occurrences rather than using temporal evidence for verbs; that shortcut kills generalization to novel verb-object pairs. This paper diagnoses that imbalance and proposes targeted regularizers so models rely less on object labels and more on temporal dynamics when predicting verbs.

Key Findings
  • Diagnostic analysis shows many existing methods overfit training co-occurrence patterns and underuse temporal verb cues — so they fail on unseen compositions. This quantifies a concrete failure mode to target.
  • RCORE reduces shortcut metrics and improves zero-shot compositional accuracy on two benchmarks (Sth-com, EK100-com) — so enforcing priors and temporal sensitivity yields measurable generalization gains.
  • Co-occurrence Prior Regularization (CPR) injects explicit supervision for unseen compositions and treats frequent co-occurrences as hard negatives — so the model learns to discount spurious object→verb shortcuts.
  • Temporal Order Regularization for Composition (TORC) encourages sensitivity to action ordering in video snippets — so verb representations become more temporally grounded rather than object-driven.
Who it's for and tradeoffs

Great fit if you research compositional generalization in video action recognition or need methods to reduce dataset co-occurrence biases. The approach is practical for benchmark-driven research and for systems where verb inference should depend on motion/ordering signals rather than static object labels.
Look elsewhere if you need a turnkey production system with minimal retraining: CPR/TORC introduce additional supervision and training constraints that may increase tuning complexity and computational cost. The methods target object-driven shortcut failure modes and may be less impactful when datasets already provide strong temporal cues or when composition vocabularies scale extremely large without reliable co-occurrence statistics.

Method overview
  • CPR: constructs explicit supervision signals for unseen compositions and imposes penalties so frequent object-verb pairs become hard negatives during training, reducing reliance on object labels alone.
  • TORC: applies regularization that rewards models for respecting temporal order information, forcing verb features to capture dynamic patterns across frames.

Overall, the paper reframes a common failure mode (object-driven shortcuts) into measurable diagnostics and supplies two complementary regularizers that shift models toward temporally grounded, more compositional verb recognition.

Information

  • Websitearxiv.org
  • AuthorsGeo Ahn, Inwoong Lee, Taeoh Kim, Minho Shim, Dongyoon Wee, Jinwoo Choi
  • Published date2026/01/22

More Items

Distills a diffusion teacher’s conditional image distribution by matching exponentially tilted student means to teacher posterior means. It supports flexible sampling budgets on the trained timestep grid while preserving generation diversity.

Introduces a multimodal lens for tracing how vision-language models represent visual concepts and selectively access them in response to questions. Counterfactual tests examine how those concepts affect later reasoning; the work does not claim models are conscious.

Tests multimodal models’ spatial reasoning through screen-based clicks and drags. Its programmatically generated CAPTCHA tasks also form a training environment, with fine-tuning gains reported across seven out-of-distribution benchmarks.