Most academic video corpora contain terse labels or short answers that are useful for benchmarks but poor for training instruction-following multimodal models. VideoChat3-Academic2M addresses that gap by converting existing academic captions and QA annotations into richer, evidence-grounded instruction responses that explicitly mention visible objects, actions, scenes, temporal cues, and supporting evidence — making them more suitable for training and evaluating video MLLMs.
What Sets It Apart
- Aggregates and re-annotates multiple academic caption/QA sources (e.g., LLaVA-Video, S‑MiT, Vript, StarQA, Sports‑QA, Perception‑Test), so you get cross-dataset coverage without re-downloading a new video collection.
- Evidence-grounded annotation enhancement pipeline: short answers, option-only labels, and concise captions are rewritten into full instruction-following responses, then filtered for consistency with original labels — this reduces hallucination risk while increasing response richness.
- Dataset distribution focuses on annotations only (JSONL files plus a mapping file linking back to original video paths). That keeps the repo lightweight and legally clean but requires users to resolve and host original videos from source datasets.
- Apache-2.0 license and English-only annotations simplify integration for research and fine-tuning in academic and open-source settings.
Who it's for & Tradeoffs
Great fit if you are training or benchmarking video-capable instruction-tuned models (video MLLMs) and need richer, evidence-linked human-style responses derived from established academic datasets. It is also useful for probing temporal reasoning and fine-grained motion understanding where terse labels are insufficient.
Look elsewhere if you need a packaged video corpus (original videos are not included), multilingual annotations, or highly curated human-video pairs collected under a single recording protocol — this release focuses on annotation enhancement across heterogeneous academic sources, not on unified video collection.
