RoboDojo tackles a practical gap: benchmarks either scale in simulation or expose real-world challenges, but rarely unify both with reproducible protocols. By combining heterogeneous parallel simulation with a standardized, remotely accessible real-world evaluation system, the paper makes it possible to compare generalist manipulation policies on the same set of capabilities and deployment constraints.
Key Findings
- Multi-domain coverage: The benchmark pairs 42 simulation tasks with 18 real-world tasks to span complementary manipulation capabilities. This means evaluations capture both scalable metric-driven comparison in sim and the deployment variability of physical systems.
- Five evaluation dimensions: Generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following are measured explicitly—so evaluators can pinpoint which aspects of policy performance drive failures in transfer or deployment.
- Reproducible real-world eval: RoboDojo-RealEval offers standardized hardware, scene reset procedures, evaluation protocol, and remote cloud access, enabling repeatable physical tests without each group rebuilding hardware setups.
- Integration and benchmarking: XPolicyLab lets policies be integrated once and run across sim and real with minimal adaptation; the authors integrated 30 policies and published a public leaderboard to surface current performance gaps.
Who It's For and Trade-offs
Great fit if you need a systematic, comparable evaluation between sim and real for manipulation research—particularly for teams benchmarking generalist policies across multiple capabilities or studying sim-to-real gaps. The benchmark accelerates comparative analysis and reproducibility by providing task suites, standardized real-world rigs, and an evaluation pipeline.
Look elsewhere if your focus is on highly specialized hardware, non-manipulation robotics (e.g., locomotion-only), or ultra-low-latency on-device deployment: RoboDojo’s real-world arm is standardized and queue-based cloud access may not match bespoke lab setups. Also, while the suite is broad, adding new hardware or radically different task modalities will require extra engineering to adopt the provided RealEval rig.
Overall, RoboDojo is positioned as a practical middle ground for researchers who want both scalable simulation studies and reproducible physical validation without rebuilding end-to-end infrastructure.
