A robot can remember the right instruction and still fail at the first grasp; it can also execute a practiced motion perfectly while acting on the wrong plan. The key insight is that reasoning and action scores alone do not reveal whether a system can connect them through feedback from the physical world.
Key Findings
- The benchmark separates Mind (decisions based on earlier observations), Act (precise, timely execution), and Mind2Act (chained operations whose next step depends on the physical outcome). Comparing these views helps distinguish a missing capability from a failure to coordinate capabilities.
- Its 30 simulated tasks cover five application domains, with three difficulty levels and Clean and Random scene settings, for 180 configurations. This makes it possible to examine how task demands and changes in clutter, materials, and lighting affect performance.
- The gap between isolated and combined performance is stark: the strongest evaluated agent reached 52.7% on Mind and 39.0% on Act, but only 12.0% on Mind2Act. Success in either component does not guarantee that a system can close the loop.
- A diagnosis harness traces failed episodes to reasoning, execution, or their composition. That turns a headline score into a more actionable view of where a system breaks down.
Method and Fit
The benchmark is useful for researchers comparing multimodal agents and end-to-end vision-language-action or world-action policies, especially when the question is whether a system can use action outcomes to revise its next decision. It is less suited to claims about real-world robot reliability on its own: the tasks are simulated, and results depend on the benchmark’s task design and evaluation setup. Treat it as a diagnostic complement to real-robot testing, not a substitute.