Real-time multimodal assistants must do more than label frames: they must guide users through actions, manage multi-turn context, and decide when to speak. Static offline benchmarks miss this closed-loop aspect because a model's response changes subsequent user behavior; OmniAssistBench deliberately simulates that loop by deriving concrete user goals from real videos and enforcing predefined priors so evaluations remain comparable.
Key Findings
- Dataset construction: Internet videos are reverse-engineered into multi-turn clips with explicit priors so a single user goal maps to a constrained route; building this required over 1000 expert person-hours. This design forces agents to plan and keep users on the same path rather than accept any valid-but-different solution.
- Evaluation focus: measures go beyond QA accuracy to include multi-turn context continuity, response timing (including delayed responses until a target event), and handling of visual prompts such as hand gestures.
- Model performance: proprietary Gemini-3-Pro scored 66.4/100 and open-source Qwen3-Omni-Instruct scored 51.2/100, showing that while models understand inputs, they commonly give incorrect or incomplete guidance, miss visual cues, and fail to maintain history across turns.
Who it's for and trade-offs
Great fit if you develop or evaluate real-time omni-modal assistants and need a strict, route-constrained benchmark that exposes planning, timing, and human-in-the-loop failure modes. Look elsewhere if you only need offline video QA, open-ended multi-solution evaluation, or lightweight synthetic tests — OmniAssistBench intentionally restricts acceptable interaction paths, which makes it more realistic for assistant evaluation but less permissive for alternative valid strategies.
Where it fits
OmniAssistBench complements offline video QA and proactive streaming benchmarks by emphasizing assistant behavior under constrained user goals and realistic interaction dynamics. Use it to stress-test turn-taking, temporal delay strategies, and multimodal instruction grounding that typical segmented QA datasets do not exercise.
