AIAny
Icon for item

RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

Evaluates generalist robot manipulation policies across simulation and real-world settings using 42 sim tasks and 18 real tasks; measures generalization, memory, precision, long-horizon execution and open-vocabulary instruction following, and provides a cloud-accessible real-world evaluation system with XPolicyLab integration and a public leaderboard.

Introduction

RoboDojo tackles a practical gap: benchmarks either scale in simulation or expose real-world challenges, but rarely unify both with reproducible protocols. By combining heterogeneous parallel simulation with a standardized, remotely accessible real-world evaluation system, the paper makes it possible to compare generalist manipulation policies on the same set of capabilities and deployment constraints.

Key Findings
  • Multi-domain coverage: The benchmark pairs 42 simulation tasks with 18 real-world tasks to span complementary manipulation capabilities. This means evaluations capture both scalable metric-driven comparison in sim and the deployment variability of physical systems.
  • Five evaluation dimensions: Generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following are measured explicitly—so evaluators can pinpoint which aspects of policy performance drive failures in transfer or deployment.
  • Reproducible real-world eval: RoboDojo-RealEval offers standardized hardware, scene reset procedures, evaluation protocol, and remote cloud access, enabling repeatable physical tests without each group rebuilding hardware setups.
  • Integration and benchmarking: XPolicyLab lets policies be integrated once and run across sim and real with minimal adaptation; the authors integrated 30 policies and published a public leaderboard to surface current performance gaps.
Who It's For and Trade-offs

Great fit if you need a systematic, comparable evaluation between sim and real for manipulation research—particularly for teams benchmarking generalist policies across multiple capabilities or studying sim-to-real gaps. The benchmark accelerates comparative analysis and reproducibility by providing task suites, standardized real-world rigs, and an evaluation pipeline.

Look elsewhere if your focus is on highly specialized hardware, non-manipulation robotics (e.g., locomotion-only), or ultra-low-latency on-device deployment: RoboDojo’s real-world arm is standardized and queue-based cloud access may not match bespoke lab setups. Also, while the suite is broad, adding new hardware or radically different task modalities will require extra engineering to adopt the provided RealEval rig.

Overall, RoboDojo is positioned as a practical middle ground for researchers who want both scalable simulation studies and reproducible physical validation without rebuilding end-to-end infrastructure.

Information

  • Websitearxiv.org
  • AuthorsTianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang, Kailun Su, Haoran Lu, Weijie Wan, Baijun Chen, Songling Liu, Haowen Yan …
  • Published date2026/07/05

Categories

More Items

AI Others2026

Diagnoses whether robotic manipulation failures stem from history-based reasoning, precise physical execution, or their closed-loop coordination. It spans 30 tasks, three difficulty levels, and two scene settings.

Hugging Face

Provides first-person video of skilled manual work with synchronized motion-sensor data for robot learning and dexterous manipulation research. The first release spans 10,124 hours from 12,227 contributors, with task-level annotations.

Hugging Face
AI Model2026

Simulates how a scene changes in response to text-described actions, optionally conditioned on a starting image, and generates videos up to 20 seconds long. It pairs language-model reasoning about events with a 29B diffusion model for visual prediction.