Native audio‑visual dialogue forces models to treat synchronized video and the video's own audio as the single user query (no separate text or ASR). That setting is under-served by public datasets and hard to evaluate with keywords alone; this release supplies both an evaluation benchmark and a training reward so researchers can measure and optimize models that must jointly perceive audio and vision and reply in natural speech style.
What Sets It Apart
- A paired benchmark and training recipe: OmniVChat‑Bench (2,800 instances: 2,550 single‑turn + 250 multi‑turn) plus OmniVChat‑RL (a configurable reward combining the rubric with format, efficiency and style terms). The training reward is R(y)=r(y)+λ_fmt·f(y)+λ_eff·e(y)+λ_sty·s(y) with practical defaults.
- Fine‑grained, tiered rubrics (13,475 criteria) across five abilities (DSLP, MEA, MSA, AH, ER) and 17 subcategories; scoring is tier‑gated so prerequisite checks (tier 0) block credit for higher tiers.
- Media and tooling included: ~28 GB of 1080p mp4 media (the audio in each clip is the user's speech), JSONL data files (single_turn.jsonl, multi_turn.jsonl), a judge‑based scorer (eval/score.py), and the reward implementation (reward/reward.py).
- Practical diagnostics and configurability: format/style checks, absolute and group‑relative efficiency metrics, multi‑turn protocol details, and explicit judge configuration for reproducible evaluation.
- Special handling of embodiment assumptions: 200 MSA‑PLA instances carry two independent rubrics (non‑embodied vs embodied) so users must pick the rubric matching their system.
Who It's For and Tradeoffs
Great fit if you develop or evaluate multimodal/omni models that must answer from live audio+video (e.g., in-car assistants, embodied agents, multimodal assistants). It provides a reproducible judge-driven metric and a trainable reward. Tradeoffs: the media download is large (~28 GB), the license is CC‑BY‑NC‑ND (non‑commercial, no derivatives) limiting commercial use, many dialogues are synthesized (not human recordings), and correct benchmarking requires configuring a reliable LLM judge endpoint.
