A model can give a plausible instruction and still fail the person relying on it. The key insight here is to judge assistance by what changes in the world—not by whether the assistant’s words sound convincing. That makes long-running guidance, where each instruction depends on what happened before, measurable rather than anecdotal.
Key Findings
- The assistant sees only the person’s first-person view and dialogue; a separate human agent follows its guidance, while the simulator executes actions and checks the final state. This separates giving advice from carrying it out.
- InteractiveBench covers 120 multi-turn tasks across six categories, including guidance, planning, accessibility, decision-making, prevention, and finding. Because success requires every goal state to hold, partial progress cannot masquerade as task completion.
- Tests of 12 MLLMs expose a gap between isolated judgments and sustained support: the strongest reported overall pass rate is 41.7%, while the best results on long-horizon guidance and planning are 20% and 25%. Errors in perception and memory can cascade through later instructions.
- The state-based setup makes failures diagnosable and could support training assistants against outcomes, not just text quality. It also reveals the cost of inefficient guidance: models may direct people through substantially more actions than an oracle assistant.
Who It’s For—and the Trade-off
A good fit for researchers studying egocentric assistants, interactive MLLM evaluation, or human-centered embodied AI who need reproducible, outcome-based tests. It is less suitable for drawing conclusions about real users or unconstrained physical environments: simulated tasks and a fixed model acting as the human do not capture the full variability of people and the physical world.