RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies

Abstract: Generalist robot manipulation policies have advanced rapidly, yet existing benchmarks remain limited in systematically evaluating their capabilities. Many rely on simple, short-horizon, or skill-narrow tasks with limited capability coverage, and are often conducted only in simulation or only in the real world. Simulation enables scalable feedback but misses physical deployment challenges, while real-world evaluation is costly, time-consuming, and difficult to reproduce. We introduce RoboDojo, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. RoboDojo includes 42 simulation tasks and 18 real-world tasks covering diverse and complementary manipulation capabilities. The simulation benchmark evaluates five dimensions: generalization, memory, precision, long-horizon execution, and open-vocabulary instruction following, while the real-world benchmark exposes policies to challenging physical-world deployment conditions. RoboDojo supports scalable evaluation through heterogeneous parallel simulation in Isaac Sim and provides RoboDojo-RealEval, a reproducible real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and deployment interface. Together with XPolicyLab, policies can be integrated once and evaluated across simulation and real-world settings with minimal adaptation. We integrate 30 policies into XPolicyLab and evaluate them on RoboDojo, establishing a public leaderboard and systematic analysis of current policy performance. The website is available at http://robodojo-benchmark.com/.

• The paper introduces RoboDojo, a unified benchmark that integrates 42 simulation tasks and 18 real-world tasks to evaluate generalist robot manipulation policies.
• The benchmark reveals that current policies achieve as low as 8.8% success in simulation and 12.8% in real-world settings, sharply contrasting with expert human performance.
• The study emphasizes standardized testing, reproducibility, and the need for algorithmic advances in multimodal grounding, memory integration, and contact-aware control.

RoboDojo: A Unified Sim-and-Real Benchmark for Generalist Robot Manipulation Policies

Introduction

The paper presents RoboDojo, a unified benchmark designed for the comprehensive evaluation of generalist robot manipulation policies across both simulated and real-world settings (2607.04434). RoboDojo addresses critical challenges in the evaluation of large-scale, embodied robot learning models by integrating capability-driven simulation tasks with standardized, reproducible real-world testing. The benchmark is coupled with XPolicyLab, an infrastructure enabling seamless policy development, deployment, and cross-domain evaluation. Through extensive task coverage, rigorous evaluation protocols, and remote-access real hardware, RoboDojo enables systematic diagnosis of policy weaknesses, provides insights into current algorithmic limits, and supports the progression toward robust, general-purpose manipulation.

Figure 1: RoboDojo system overview, showing the unified simulation and real-world pipeline and associated infrastructure.

Benchmark Structure and System Design

Unified Sim-and-Real Evaluation Paradigm

RoboDojo unifies evaluation in simulation and reality via a common policy interface and testing protocol. The simulation benchmark supports scalable, heterogeneous parallel evaluation on NVIDIA Isaac Sim, encompassing 42 tasks grouped across five capability axes: Generalization, Memory, Long-Horizon, Precision, and Open. The real-world benchmark comprises 18 complex manipulation tasks executed on three distinct collaborative bimanual robot platforms. This dual-setup enables direct comparison of policy robustness, generalization, and real-world deployability under controlled, reproducible conditions.

Figure 2: Task overview highlighting 42 simulation and 18 real-world tasks grouped by capability dimension.

Task and Skill Diversity

Tasks are designed to challenge different aspects of embodied intelligence:

• Generalization: Robustness to visual and environmental variation.
• Memory: Long-context reasoning and non-Markovian decision making.
• Long-Horizon: Sequential, multi-stage manipulation workflows.
• Precision: Contact-rich, spatially and temporally sensitive tasks.
• Open: Unseen, language-specified task recomposition.

The suite covers 24 unique manipulation skill primitives, ranging from grasping and tool use to folding and insertion. This extensive coverage supports multidimensional competency analysis.

Figure 3: Visualization of skill diversity across tasks and embodiment configurations.

Real-World Evaluation: RoboDojo-RealEval

RoboDojo-RealEval standardizes real-world testing through meticulously specified hardware layouts, scene reset protocols, camera and lighting geometry, and an online cloud-based deployment interface. Evaluations are controlled for reproducibility: all trials are scored by independent human raters, and all required components (code, checkpoints, layouts, configuration) are open-sourced for verified leaderboard entries.

Figure 4: Technical schematic of the RoboDojo-RealEval setup, supporting remote and on-premises policy deployment with reproducible hardware and scene configuration.

Heterogeneous Parallel Simulation and Asset Library

Simulation efficiency and scene diversity are achieved via heterogeneous parallelism: concurrent environments instantiate distinct object geometries, layouts, and capabilities within the same simulation process. RoboDojo's asset library includes rigid, articulated, and deformable objects, with physically validated digital twins ensuring realism and scalability.

Figure 5: Examples of rigid, articulated, and deformable assets enabling diverse, contact-rich simulation environments.

Domain Randomization and Robustness

Domain randomization is integral to the generalization assessment in simulation: RoboDojo randomizes visual backgrounds, lighting, object appearance, clutter, and layout at both data collection and evaluation time to systematically stress overfitting and highlight distributional brittleness.

Figure 6: Domain randomization modes producing a broad spectrum of visually and physically distinct task instantiations.

Experimental Results and Diagnostic Insights

Simulation Performance

The simulation leaderboard integrates 30 state-of-the-art policies, evaluating each across fine-grained capability axes. Notably, no policy achieved above 9% average success rate across all tasks; the best-performing method (Hy-Embodied-0.5-VLA) attains 8.8% average success—substantially inferior to expert human teleoperation at 76% average success. This substantial gap remains true even in structurally simpler or highly-imitated settings, underscoring a vast unsolved space for embodied generalization.

Major findings include:

• Catastrophic performance collapse under domain and scene randomization—even top models show >90% drop in performance when transitioning from standard to randomized settings.
• Better spatial representation (e.g., in Spatial Forcing) partially enhances robustness, but the overall improvement is minor in absolute terms.
• Long-horizon task performance is relatively better than open or memory-demanding tasks, yet failure to consistently complete multi-step requirements reveals insufficient skill compositionality.
• Precision tasks remain a critical bottleneck; dominant failures are caused by poor state-conditioned correction, contact-awareness, and insufficient low-level action smoothness.
• Explicit memory-enhanced models demonstrate limited success; remembering context is insufficient when not tightly integrated with actionable policy outputs.
• Semantically open tasks are essentially unsolved—current policies do not robustly interpret and execute tasks specified by language alone, especially those not present in the demonstration set.

Real-World Performance

Translating simulation improvements to the physical world is only partially successful. The best model ($\pi_{0.5}$) yields an average real-world success rate of just 12.8%, again vastly below human expert performance. Task scores are somewhat higher, indicating partial sub-goal competency, but successful completion remains rare.

Empirically, leaderboard alignment between sim and real is only partial; several models with competitive simulated scores underperform dramatically when exposed to real hardware. Real-world rollouts expose additional issues such as action jitter, contact instability, and unsafe behaviors not observable in simulation.

Figure 7: Representative frames from challenging real-world tasks, emphasizing cross-embodiment deployment and contact-driven scenarios.

Technical Contributions

Integrated Evaluation Loop and Remote Access

RoboDojo is designed for rapid policy iteration: policies can be integrated once, tested across all simulated and real-world tasks with minimal adaptation, and compared via a unified leaderboard. Remote deployment is enabled via a standardized client-server communication protocol (WebSocket + MessagePack), minimizing friction between policy providers and the evaluation hardware.

Reproducibility and Anti-Gaming

Strict reproducibility is enforced via hidden verification layouts, mandatory multi-seed reporting, and artifact release requirements for leaderboard entries. These safeguards disincentivize task-specific overfitting, hand-tuning, and metric gaming—chronic issues in prior embodied AI benchmarks.

Evaluation Efficiency

Heterogeneous parallel simulation provides up to 2x speedup over non-parallel baselines under both zero-action and policy-inference rollouts without sacrificing scene variety. RoboDojo-RealEval completes more than 180 real-world trials in under 3.5 hours, demonstrating practical usability for iterative research.

Implications and Future Directions

RoboDojo demonstrates that despite recent advances, current generalist manipulation policies are severely limited in balanced capability, open-endedness, memory, precision, and cross-domain deployability. Policy performance in simulation does not yet guarantee safe, robust real-world operation. Fundamental algorithmic advances are needed:

• More robust multimodal grounding and open-vocabulary skill acquisition.
• Explicitly compositional and memory-conditioned architectures.
• Contact-aware, closed-loop low-level control tightly integrated with semantic planning.
• Policy learning objectives and evaluation strategies prioritizing real-world safety, recovery, and correction.

The benchmark is deliberately extensible—future releases will address dexterous hands, whole-body humanoid manipulation, tactile and mobile manipulation, and additional robot embodiments.

Figure 8: RoboDojo roadmap, illustrating planned extensions to dexterous, humanoid, tactile, and mobile domains.

Conclusion

RoboDojo establishes a rigorous, unified, and extensible platform for diagnosing, comparing, and advancing generalist robot manipulation policies. By bridging systematic simulation with reproducible, scalable real-world testing, RoboDojo provides both a revealing diagnosis of current policy limitations and a practical infrastructure for driving the next generation of embodied intelligence research. Significant open challenges remain—especially in cross-domain generalization, closed-loop control, and language-conditioned skill compositionality. RoboDojo will continue to evolve in scope and rigor, serving as a core testbed for real progress in robust embodied AI.