Gander attacks a practical gap in multimodal agents: most systems either process offline video or operate turn‑by‑turn, which breaks natural interruptions, proactive feedback, and continuous monitoring. The core insight is to separate low‑latency conversational control from heavyweight reasoning so a single system can both respond immediately to streaming inputs and execute complex agentic workflows.
Key Findings
- Cerebellum–Brain architecture: a streaming Thinker‑Talker "Cerebellum" handles chunk‑level low‑latency perception and dialog act decisions, while a separate "Brain" handles higher‑latency, complex reasoning and tool orchestration — this decoupling preserves responsiveness without discarding long‑horizon reasoning.
- Native full‑duplex streaming interface: flattening inputs/outputs into ordered token chunks lets the model detect multimodal triggers, accept user interruptions, and proactively emit intermediate feedback; in practice this reduces response latency compared with turn‑based baselines.
- Evaluation across four axes: conversational quality, omni‑modal understanding, interactive robustness (interruptions, backchannel, noise), and agentic intelligence (tool calls, multi‑step tasks). Human evaluations show competitive spoken dialogue fluency while exposing gaps in streaming mathematical reasoning and long‑step tool plans.
- Release and reproducibility: models, datasets, and code are provided to encourage follow‑up work on streaming benchmarks and agent orchestration.
Who it's for and tradeoffs
Great fit if you need a research‑grade reference for continuous multimodal interaction—especially scenarios with live video/audio, user interruptions, or agents that must interleave conversation with tool use. Look elsewhere if your primary goal is lightweight deployment on constrained devices (the architecture and training are resource‑intensive) or purely offline batch video understanding where low‑latency turn taking is irrelevant. Expect remaining limitations in robust long‑horizon reasoning under full‑duplex constraints and in extreme multi‑party audio environments.