A live assistant has to decide not only what to say, but whether to speak while new evidence is still arriving. Treating that timing decision as part of the model—not as an external controller—is the central idea here, and it makes interruption, silence, and turn-taking first-class parts of audiovisual interaction.
Key Findings
The design separates Intent, which reasons autoregressively about the evolving exchange, from Realization, which generates synchronized speech and video. Both operate on a shared chunked timeline with causal attention, so a response cannot use future frames or audio. Dual rollout distillation then trains the modules using their own generated histories, addressing the mismatch between recorded training examples and live interaction. In a 30-case human evaluation, 168 of 180 checks passed; responsiveness was the weakest measure, passing 22 of 30 cases.
Who It’s For—and the Trade-off
A useful reference for researchers building streaming multimodal agents, especially where interruption and audiovisual turn-taking matter. It is not yet evidence of robust long-session interaction: the evaluation is small, content responsiveness still misses some user intent, and the report describes the release as a preliminary system with broader testing still needed.