Current zero-shot compositional action recognition models often succeed by memorizing frequent object–verb co-occurrences rather than using temporal evidence for verbs; that shortcut kills generalization to novel verb-object pairs. This paper diagnoses that imbalance and proposes targeted regularizers so models rely less on object labels and more on temporal dynamics when predicting verbs.
Key Findings
- Diagnostic analysis shows many existing methods overfit training co-occurrence patterns and underuse temporal verb cues — so they fail on unseen compositions. This quantifies a concrete failure mode to target.
- RCORE reduces shortcut metrics and improves zero-shot compositional accuracy on two benchmarks (Sth-com, EK100-com) — so enforcing priors and temporal sensitivity yields measurable generalization gains.
- Co-occurrence Prior Regularization (CPR) injects explicit supervision for unseen compositions and treats frequent co-occurrences as hard negatives — so the model learns to discount spurious object→verb shortcuts.
- Temporal Order Regularization for Composition (TORC) encourages sensitivity to action ordering in video snippets — so verb representations become more temporally grounded rather than object-driven.
Who it's for and tradeoffs
Great fit if you research compositional generalization in video action recognition or need methods to reduce dataset co-occurrence biases. The approach is practical for benchmark-driven research and for systems where verb inference should depend on motion/ordering signals rather than static object labels.
Look elsewhere if you need a turnkey production system with minimal retraining: CPR/TORC introduce additional supervision and training constraints that may increase tuning complexity and computational cost. The methods target object-driven shortcut failure modes and may be less impactful when datasets already provide strong temporal cues or when composition vocabularies scale extremely large without reliable co-occurrence statistics.
Method overview
- CPR: constructs explicit supervision signals for unseen compositions and imposes penalties so frequent object-verb pairs become hard negatives during training, reducing reliance on object labels alone.
- TORC: applies regularization that rewards models for respecting temporal order information, forcing verb features to capture dynamic patterns across frames.
Overall, the paper reframes a common failure mode (object-driven shortcuts) into measurable diagnostics and supplies two complementary regularizers that shift models toward temporally grounded, more compositional verb recognition.