AIAny
Icon for item

OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs

Benchmarks assistant-style, multi-turn interaction for omni-modal LLMs on real-time video by reverse-engineering Internet clips into guided multi-turn interactions. It provides predefined priors and segment-level constraints so models must follow exact routes while being evaluated on answer correctness, timing, visual-prompt handling, and context retention.

Introduction

Real-time multimodal assistants must do more than label frames: they must guide users through actions, manage multi-turn context, and decide when to speak. Static offline benchmarks miss this closed-loop aspect because a model's response changes subsequent user behavior; OmniAssistBench deliberately simulates that loop by deriving concrete user goals from real videos and enforcing predefined priors so evaluations remain comparable.

Key Findings
  • Dataset construction: Internet videos are reverse-engineered into multi-turn clips with explicit priors so a single user goal maps to a constrained route; building this required over 1000 expert person-hours. This design forces agents to plan and keep users on the same path rather than accept any valid-but-different solution.
  • Evaluation focus: measures go beyond QA accuracy to include multi-turn context continuity, response timing (including delayed responses until a target event), and handling of visual prompts such as hand gestures.
  • Model performance: proprietary Gemini-3-Pro scored 66.4/100 and open-source Qwen3-Omni-Instruct scored 51.2/100, showing that while models understand inputs, they commonly give incorrect or incomplete guidance, miss visual cues, and fail to maintain history across turns.
Who it's for and trade-offs

Great fit if you develop or evaluate real-time omni-modal assistants and need a strict, route-constrained benchmark that exposes planning, timing, and human-in-the-loop failure modes. Look elsewhere if you only need offline video QA, open-ended multi-solution evaluation, or lightweight synthetic tests — OmniAssistBench intentionally restricts acceptable interaction paths, which makes it more realistic for assistant evaluation but less permissive for alternative valid strategies.

Where it fits

OmniAssistBench complements offline video QA and proactive streaming benchmarks by emphasizing assistant behavior under constrained user goals and realistic interaction dynamics. Use it to stress-test turn-taking, temporal delay strategies, and multimodal instruction grounding that typical segmented QA datasets do not exercise.

Information

  • Websitearxiv.org
  • OrganizationsAffiliation: Project Leader & Corresponding AuthorProject Page: https://xianyunsun.github.io/OmniAssistBench/, Affiliation: Nankai University, Affiliation: University of Waterloo
  • AuthorsXianyun Sun, Chaoyou Fu, Zhengye Zhang, Feiyang Duan, Qingyuan Cao, Yonghui Niu, Sihang Yuan, Ge Zhang, Caifeng Shan
  • Published date2026/08/21

More Items

Hugging Face

Pairs Three.js scene prompts with curated reasoning traces, HTML/JavaScript outputs, and rendered references. Its family-held-out validation split helps test generalization beyond examples closely related to the training data.

Hugging Face

Provides Fongbe speech recordings with linguist-reviewed tonal transcriptions, alongside a separate set of validated recordings without transcripts. The two subsets support supervised and semi-supervised speech research.

Coordinates visual and audio forensic agents to locate and verify signs of AI-generated video. Its dual-timescale training improves both the investigation strategy and the specialist inspectors.