RoboDojo: Unified Sim-and-Real Benchmark

• RoboDojo is a unified sim-and-real benchmark that evaluates generalist robot manipulation policies across 42 simulation tasks and 18 real-world tasks.
• It assesses five key capability dimensions—Generalization, Memory, Long-Horizon, Precision, and Open—to reveal nuanced failure modes and performance gaps.
• The framework leverages heterogeneous parallel simulation and standardized physical setups to ensure reproducibility, comprehensive evaluation, and public leaderboard comparison.

Searching arXiv for the specified papers to ground the article and citations. Searching arXiv for "RoboDojo unified sim-and-real benchmark generalist robot manipulation policies". RoboDojo is “a unified sim-and-real benchmark for efficient, comprehensive, and reproducible evaluation of generalist robot manipulation policies,” combining “42 simulation tasks and 18 real-world tasks,” “heterogeneous parallel simulation,” “RoboDojo-RealEval,” “XPolicyLab,” and “a continuously updated leaderboard” (Chen et al., 5 Jul 2026). It was introduced to address two stated deficiencies in prior evaluation practice: capability-incomplete simulation benchmarks and expensive, hard-to-reproduce real-world testing. The benchmark evaluates five simulation dimensions—Generalization, Memory, Precision, Long-Horizon, and Open—and complements them with standardized physical-world deployment across three embodiments. Despite the name similarity, RoboDojo is distinct from Dojo, a differentiable physics engine for robotics that prioritizes stable simulation, accurate contact physics, and differentiability with respect to states, actions, and system parameters (Howell et al., 2022).

1. Motivation, scope, and benchmark philosophy

RoboDojo was created against the claim that “current evaluation protocols remain insufficient.” The stated critique is that many existing benchmarks rely on “relatively simple or short-horizon tasks” and vary mainly “objects, layouts, or language expressions,” which “often preserve similar underlying manipulation patterns.” On this account, prior protocols under-diagnose generalization, memory, precision, open semantics, long-horizon composition, and bimanual coordination. A second limitation is the separation of simulation and real-world evaluation: simulation is “efficient and scalable,” but cannot capture “contact-rich dynamics, actuation errors, perception noise,” whereas real evaluation is “costly, time-consuming, and difficult to reproduce without standardized hardware setups, scene reset procedures, evaluation protocols, and deployment interfaces” (Chen et al., 5 Jul 2026).

The benchmark therefore combines a shared policy interface and evaluation pipeline across simulation and real-world testing. In simulation, the design emphasis is capability coverage rather than simple scene variation. In the real world, the emphasis is standardized deployment, repeatability, and remote access. The result is a benchmark intended not only to rank policies, but also to expose failure modes that remain obscured when evaluation is restricted either to homogeneous simulation or to ad hoc physical testing.

A common misconception is to interpret RoboDojo as a direct sim-to-real transfer benchmark with matched task pairs. The benchmark explicitly “does not construct one-to-one aligned task pairs between simulation and the real world”; the aim is “complementary deployment stress, not matched sim-to-real transfer.” This choice matters because the real-world suite is designed to probe deployment behavior under physical constraints rather than to serve as a strict counterpart of each simulated task.

2. Simulation task suite and capability dimensions

The simulation benchmark contains 42 tasks on the ARX X5 bimanual platform, with arm bases separated by 0.6 m. These tasks are organized into five capability dimensions: Generalization, Memory, Long-Horizon, Precision, and Open. Each dimension is defined operationally in terms of the failure modes it is supposed to reveal. Generalization stresses robustness under unseen backgrounds, lighting, clutter, objects, and layouts; Memory targets partial observability and non-Markov decision-making; Long-Horizon targets multi-step sequential execution, progress maintenance, stage transitions, and error recovery; Precision targets fine-grained spatial alignment, trajectory smoothness, and stable contact control; and Open targets open-vocabulary instruction following, semantic grounding, and skill recombination for unseen task specifications (Chen et al., 5 Jul 2026).

Dimension	Tasks	Capability target
Generalization (12)	stack_bowls; push_T; pack_objects_into_box; fold_clothes; hang_mugs; sweep_blocks; pour_liquid_into_cup; make_toast; arrange_largest_number; sort_nesting_dolls_by_size; store_laptop_and_headphones; stack_blocks	scene-level robustness; visual-spatial grounding amid domain randomization
Memory (6)	cover_blocks; match_and_pick_from_conveyor; swap_T; press_by_number; imitate_sorting_sequence; swap_blocks	long-context observation modeling and recall of hidden state
Long-Horizon (8)	put_bottles_into_dustbin; play_tic_tac_toe; classify_objects; fill_pen_holder; fill_egg_holder; organize_table; play_stacking_toy; make_kong	infer task structure, select subtasks, switch stages at the right time; manage large workspaces and handovers
Precision (8)	fasten_screws; insert_tubes; plug_in_charger; pour_balls_into_vase; play_Xylophone; deposit_coin; insert_key; build_tower	accurate metric localization and smooth, contact-aware motion under tight tolerances
Open (8)	align_blocks; solve_equation; stack_blocks_by_language; general_pickup; classify_objects_by_language; pick_from_conveyor_by_image; store_tools_in_toolbox; pour_by_language	interpret novel instructions, identify objects by semantics or images, recombine learned skills

The Generalization split is specifically designed to exceed cosmetic perturbation. The benchmark description highlights unseen backgrounds, lighting, clutter with up to 25 distractors versus 10 in RoboTwin 2.0, objects, and layouts. Memory tasks require policies to act on earlier observations that are no longer directly visible, such as remembering block colors under covers or recalling an order demonstrated by another arm. Long-Horizon tasks often require two-arm handovers and explicit subtask sequencing. Precision tasks emphasize contact-sensitive alignment, such as insertion and slotting. Open tasks are “eval-only” and are intended to test transfer rather than imitation of matched demonstrations.

The simulation benchmark is also framed in terms of skill diversity. “Representative simulation tasks cover 24 manipulation skills,” including grasp, place, push/pull, stack, insert, open/close, fold, align, tool use, and contact-sensitive operations. This formulation positions task design around manipulation requirements rather than around narrow object categories alone.

3. Simulation platform, assets, and data generation

RoboDojo “builds a configurable simulation platform based on NVIDIA Isaac Sim and Isaac Lab,” reusing MagicSim’s “modular manager architecture, configuration-driven scene construction pipeline, and object simulation backend.” Tasks are instantiated from “modular YAML specifications” that define assets, layouts, randomization ranges, and success conditions. At reset, the platform “samples task-relevant objects, object poses, articulation states, clutter layouts, lighting conditions, and background textures with deterministic seed control,” supporting “diverse yet reproducible scene instances.” The asset library includes physically grounded rigid, articulated, and deformable assets annotated with “object categories, language descriptions, placement regions, success-checking annotations, and manipulation affordances,” and deformables include selections from ClothesNet (Chen et al., 5 Jul 2026).

A central systems feature is heterogeneous parallel simulation. “Multiple environments are stepped under a shared vectorized interface, while each environment maintains an independently sampled scene configuration… different object categories, geometry, distractors, articulation, and layouts.” This departs from homogeneous cloned environments and is presented as critical for generalist stress testing. On 8×RTX 4090 GPUs, heterogeneous parallel simulation achieved 77.4 interactions/s for zero action versus 40.0 non-hetero, a 1.94× speedup, and 64.0 interactions/s with $\pi0.5$ inference versus 39.2 non-hetero, a 1.63× speedup.

Data collection proceeds in two modes. One uses automated trajectory synthesis via reusable skills—grasp, place, handover, insert, open/close, stack, and push_up—with cuRobo v2 motion planning. The other uses VR-based teleoperation, mapping controller 6D deltas to end-effector motion. For a single 6-DoF ARX X5 arm, spatial motion solving latency can reach approximately 3 ms per planning step. Simulation training data contain synchronized RGB-D observations from a head-mounted camera and two wrist cameras at 640×480, robot states and next-frame actions covering end-effector poses, gripper states, and joint positions, at 25 Hz. The released simulation training set comprises 35 task directories, 3,500 trajectories, and 1,859,602 frames, corresponding to 20.66 hours at 25 Hz; Open tasks are excluded from training. An additional 100 auxiliary DLC trajectories under strong domain randomization are provided for visual augmentation.

4. RoboDojo-RealEval and standardized physical evaluation

The real-world component contains 18 tasks across three embodiments: ARX X5, Piper, and Piper X. The task suite is described as “a compact yet challenging physical test suite” because of “hardware cost, execution time, human reset effort, and safety considerations.” Target capabilities include long-horizon sequencing, precise insertion and alignment, memory-conditioned execution, and generalization across embodiments and camera or kinematics differences (Chen et al., 5 Jul 2026).

Embodiment	Tasks	Notes
ARX X5 (6)	insert_tubes; make_bread; make_food; pack_and_pour_fruit; cover_blocks; store_in_safe	bimanual collaborative embodiment
Piper X (6)	disassemble_LEGO; cap_pen; hang_mugs; classify_objects; sweep_blocks; pack_objects_into_backpack	bimanual collaborative embodiment
Piper (6)	fill_pen_holder; put_objects_into_basket; stack_and_cover_blocks; stack_bowls; stand_up_bottles; insert_charger	bimanual collaborative embodiment

RoboDojo-RealEval standardizes the hardware geometry, cameras, lighting, reset procedures, scoring, safety, and cloud-based interface. The hardware specification includes a fixed-height white table measuring 1.2 m × 1.2 m, an external frame of 1.5 m × 1.5 m × 2.1 m, black curtains controlling illumination, “three linear LED light sources with fixed mounts,” fixed robot and camera mounts, a touchscreen interface, and an integrated workstation box. The camera configuration uses “two Gemini 305 wrist cameras” and “one Gemini 335L head camera.”

Reset and replay are integral to reproducibility. The target layout image is overlaid “with transparency on the live observation stream” to restore initial conditions, and a 5-object scene reset averaged 14 s in testing. For each task, evaluation uses 10 trials with pre-collected layouts replayed each time. Maximum execution horizon is determined by the benchmark’s horizon rule, and unsafe behavior triggers safety termination. End-effector actions are planned with Pink to joint motions.

Scoring is not automated by a single geometric threshold. Each trial is “recorded and independently scored by three evaluators under a double-blind protocol,” with scoring that “accounts for both final task success and intermediate sub-step completion.” Evaluation videos and scores are released for transparency, together with an appeal mechanism. The platform supports local and cloud-based evaluation via a standardized communication protocol, and end-to-end real evaluation for 18 tasks took 202 minutes, approximately 3.4 hours for 180 trials, with a $\pi0.5$ policy server on an RTX 4090 LAN; this corresponds to an average of 11.2 minutes per task for 10 trials.

5. XPolicyLab, interfaces, and evaluation metrics

XPolicyLab is the integration layer that “integrates 30 representative robot policy models” into a shared framework while preserving their internal architectures. Policies expose update_obs, get_action, and reset, together with batched variants update_obs_batch and get_action_batch, so that the same model can be evaluated in heterogeneous parallel simulation and in real deployment without code changes. Observations may include language, multi-view RGB(-D), and robot states, and the shared pose convention is [x, y, z, q_w, q_x, q_y, q_z]. Communication uses a lightweight client-server protocol based on WebSocket and MessagePack (Chen et al., 5 Jul 2026).

The simulation evaluation protocol uses 50 episodes per task; for Generalization tasks, this is split into 25 standard and 25 random episodes. The reported metrics are success rate, defined as binary task completion, and average score, defined as partial progress. The overall performance is not computed by averaging across all tasks directly. Instead, “The overall performance is computed as the mean across the five capability dimensions,” so that larger task groups do not dominate the aggregate. If $M_d$ denotes the dimension-average success rate or score, the benchmark uses

$$M_{\mathrm{overall}} = \frac{1}{5}\sum_{d \in \{\text{Gen, Prec, Long, Mem, Open}\}} M_d.$$

For Generalization, the reported relative drop is

$$d = \frac{S_{\mathrm{std}} - S_{\mathrm{rand}}}{S_{\mathrm{std}}}.$$

The evaluation horizon is determined from demonstration lengths. If $L$ is the demonstration trajectory length and $P_{90}(L)$ denotes the 90th percentile across demonstrations, then

$H = \alpha \cdot P_{90}(L),$

with $\alpha = 1.2$ generally and $\alpha = 1.5$ “for short tasks generated by automated trajectory synthesis.” The benchmark explicitly does not provide position or rotation error thresholds, instruction-grounding scores, or aggregate weighted scores beyond success rate, score, the Generalization drop, and the dimension-averaged overall metric.

6. Leaderboards and empirical findings

RoboDojo evaluates 30 policies in simulation and 10 policies in the real world, with a public leaderboard at https://robodojo-benchmark.com/LeaderBoard. As of July 3, 2026, the simulation leaderboard reports Hy-Embodied-0.5-VLA as best overall at 13.07 score and 8.80% success, leading Long-Horizon at 25.74 / 14.92% and Memory at 13.37 / 12.11%. Spatial Forcing leads Generalization at 14.12 / 9.33%, X-VLA leads Precision at 18.32 / 12.00%, and $\pi0.5$0 leads Open at 1.98 / 1.67%. Human teleoperation is reported at 80.42 score and 76.03% success, indicating a large gap despite the tasks being feasible (Chen et al., 5 Jul 2026).

The benchmark reports that scene randomization causes broad performance collapse. Even strong policies show large relative drops under the Generalization random split; for example, Hy-Embodied has standard score 21.98 versus random 1.57, corresponding to a 92.9% drop under the benchmark’s formula. Spatial Forcing retains better random performance than its base $\pi0.5$1, which is highlighted in the reported comparison. These results make robustness under task-preserving scene variation a major discriminant rather than a secondary diagnostic.

The real-world leaderboard reports $\pi0.5$2 as best overall at 22.9 score and 12.8% success across 18 tasks and three embodiments, while human teleoperation achieves 100/100 across all embodiments. The accompanying analysis emphasizes that “partial progress rarely translates into task completion” and that deployment exposes execution instability, action jitter, contact issues, and safety-critical behaviors that are not fully captured by aggregate metrics alone. The benchmark therefore treats videos and per-task analysis as essential complements to summary numbers.

7. Reproducibility, governance, benchmark positioning, and future extensions

RoboDojo publishes a website at https://RoboDojo-Benchmark.com, simulation benchmark code at https://github.com/RoboDojo-Benchmark/RoboDojo, XPolicyLab at https://github.com/XPolicyLab/XPolicyLab, and documentation at http://robodojo-benchmark.com/doc. Verified leaderboard entries must satisfy five requirements: evaluation through the official online system; simulation results over three seeds and real-world results across all three embodiments; hidden-layout verification; release of the evaluated checkpoint, training or deployment code, configurations, and instructions via XPolicyLab at publication; and release of evaluation videos. Leaderboard governance is by AI MMLab Club, and “no commercial company” participates in official evaluation (Chen et al., 5 Jul 2026).

The benchmark also reports stability statistics. Cross-GPU simulation stability shows small standard deviations of at most 1.1 percentage points in success rate and at most 1.07 in score per dimension, with at most 0.5 percentage points and 0.49 overall across three RTX 4090s. Real-world repeated runs exhibit low overall standard deviation of at most 1.3 percentage points in success rate and at most 1.2 in score, although contact-rich tasks show higher per-task variance.

In comparison with prior benchmarks, RoboDojo is positioned against sim-only suites such as RLBench, Meta-World, ManiSkill, CALVIN, LIBERO, and BEHAVIOR, and real-only suites such as ManipulationNet, RoboChallenge, and RoboArena. Its stated distinctive aspects are sim-and-real unification, five explicit capability dimensions, heterogeneous parallel simulation, representative coverage of 24 manipulation skills, and a reproducible real-world platform with fixed hardware, cameras, lighting, layout replay, double-blind scoring, safety protocols, remote evaluation, and verified governance.

The benchmark also states several limitations. The real-world suite remains compact because of “hardware cost, execution time, human reset effort, and safety considerations.” It does not build one-to-one matched task pairs between simulation and real deployment. Current coverage remains limited to three bimanual platforms and RGB camera setups. Planned extensions include “dexterous hand manipulation, humanoid whole-body manipulation, tactile manipulation, and mobile manipulation,” together with broader multi-embodiment support. These stated limitations and extensions place RoboDojo less as a finished endpoint than as an evolving evaluation infrastructure for generalist manipulation.