RoboDojo-RealEval: Real-World Robotic Benchmark

• RoboDojo-RealEval is a standardized real-world evaluation platform featuring remote access and fixed hardware setups for consistent robot manipulation testing.
• It integrates simulation and real-world protocols via XPolicyLab, enabling evaluation of 18 diverse tasks across three collaborative bimanual robot embodiments.
• Key features include a reproducible reset system, detailed human scoring, and strict safety protocols to address complex challenges in physical deployment.

Searching arXiv for the cited benchmark papers to ground the article in current records. RoboDojo-RealEval is the standardized, remote-access real-world evaluation platform in RoboDojo that complements the simulation benchmark with reproducible, deployment-focused testing across three collaborative bimanual robot embodiments. It was introduced as part of RoboDojo, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies, and is designed to address the cost, complexity, and reproducibility problems of physical evaluation by standardizing hardware, workspace, lighting, camera mounts, reset procedures, scoring, and deployment interfaces (Chen et al., 5 Jul 2026). Within the broader RoboDojo framework, RealEval serves as the physical-world counterpart to capability-oriented simulation, providing direct evidence of deployed performance under standardized physical conditions while remaining accessible through remote cloud-based evaluation and a unified policy interface via XPolicyLab (Chen et al., 5 Jul 2026).

1. Position within the RoboDojo benchmark

RoboDojo was introduced to evaluate generalist robot manipulation policies across both simulation and real-world settings, motivated by the observation that many existing benchmarks are limited to simple, short-horizon, or skill-narrow tasks and are often conducted only in simulation or only in the real world (Chen et al., 5 Jul 2026). In this design, RoboDojo includes 42 simulation tasks and 18 real-world tasks covering diverse and complementary manipulation capabilities, while RoboDojo-RealEval specifically denotes the real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and deployment interface (Chen et al., 5 Jul 2026).

The rationale for RealEval is explicitly deployment-oriented. Simulation is described as efficient for capability-oriented diagnosis and rapid model iteration, but insufficient for fully capturing contact-rich dynamics, actuation errors, perception noise, workspace constraints, temporal variations, and safety-critical behaviors that occur during physical deployment (Chen et al., 5 Jul 2026). RealEval therefore does not attempt to replace simulation; instead, it provides standardized physical testing under controlled conditions and a unified policy interface so that policies integrated once can be run both in sim and on real robots with minimal adaptation through XPolicyLab (Chen et al., 5 Jul 2026).

A frequent misconception is that a real-world benchmark of this kind should be a direct physical replay of the simulation benchmark. RoboDojo explicitly rejects that assumption: the real-world tasks are not one-to-one matched to the 42 simulation tasks, and RealEval is instead designed to complement simulation with physical deployment challenges (Chen et al., 5 Jul 2026). This separation makes the real benchmark a distinct diagnostic instrument rather than a simple transfer test.

2. Physical platform, workspace standardization, and sensing

RoboDojo-RealEval is built around a fixed and explicitly standardized physical platform. The workspace consists of a fixed-height white evaluation table with a 1.2 m × 1.2 m workspace, a replaceable white table cover for consistent visual background and easy replacement, and an external frame enclosing the workspace with dimensions 1.5 m × 1.5 m × 2.1 m (Chen et al., 5 Jul 2026). Black curtains are installed on the left, right, front, and top sides to suppress external illumination, and three linear LED light sources with fixed mounts provide stable lighting over the workspace (Chen et al., 5 Jul 2026). The platform also includes an integrated workstation box and a touchscreen interface (Chen et al., 5 Jul 2026).

The robot side comprises three collaborative bimanual platforms: ARX X5, Piper, and Piper X (Chen et al., 5 Jul 2026). The paper characterizes this as a multi-embodiment design that assesses cross-kinematics robustness, and notes that the platform includes a bottom connector to fix the relative distance and pose of the two arm bases (Chen et al., 5 Jul 2026). RealEval is therefore not a single-robot benchmark, but a standardized multi-embodiment evaluation system.

The sensing setup is also fixed. Each platform uses two Gemini 305 wrist cameras, one per wrist, and one Gemini 335L head camera (Chen et al., 5 Jul 2026). The head camera is mounted using a custom steel tube and a 3D-printed mount to fix camera pose, and the fixed mounts are described as reducing incidental variation (Chen et al., 5 Jul 2026). Video capture is at 640 × 480 resolution, while demonstrations and evaluation videos are recorded and synchronized at 25 Hz for dataset collection and scoring (Chen et al., 5 Jul 2026). In the unified observation format used for both simulation and real deployment, observations include RGB images from one head camera and two wrist cameras, robot states including joint positions, end-effector poses, gripper states, and TCP, and language instructions accompanying tasks and demonstrations (Chen et al., 5 Jul 2026). The pose convention is represented as $[x, y, z, q_w, q_x, q_y, q_z]$ (Chen et al., 5 Jul 2026).

This level of standardization is central to cross-site reproducibility. The paper specifies identical hardware across RealEval sites, including the same table, enclosure, black curtains, LED lights, fixed mounts for arm bases and cameras, the same head-camera mounting design, and identical Gemini 305 and Gemini 335L cameras (Chen et al., 5 Jul 2026). Fixed geometry and layout replay are used to ensure consistent initial conditions across trials, policies, and sites (Chen et al., 5 Jul 2026).

3. Reset system, safety mechanisms, and cloud access

A defining feature of RoboDojo-RealEval is its scene reset system. The platform uses a touchscreen-based web interface for layout replay, policy execution, and emergency stop (Chen et al., 5 Jul 2026). During reset, the web interface displays a transparent overlay of the target layout atop the live camera feed, and evaluators reposition objects until the live scene matches the overlay before starting policy execution via touchscreen (Chen et al., 5 Jul 2026). The paper reports a concrete timing figure: with five data collectors, restoring a scene with five objects took 14 seconds on average (Chen et al., 5 Jul 2026).

Public evaluation layouts are pre-collected and replayed via overlay before each trial, while hidden verification layouts are used to reduce overfitting and detect gaming (Chen et al., 5 Jul 2026). Reference layouts are shared, and hidden verification layouts supplement fairness checks (Chen et al., 5 Jul 2026). This reset mechanism is not merely ergonomic; it is the main reproducibility device for restoring initial object layouts under fixed hardware and lighting.

Safety is handled through both interface-level and human-supervised procedures. Emergency stop disables policy control immediately and returns the robot slowly to a safe reset state (Chen et al., 5 Jul 2026). Evaluators may also manually terminate trials if unsafe behavior is observed, including hitting the table, colliding with the frame, or self-collision (Chen et al., 5 Jul 2026). If reset cannot be matched due to incidental disturbances, evaluators adjust and verify alignment via the overlay before starting (Chen et al., 5 Jul 2026).

Remote access is provided through the RoboDojo online evaluation system. Policies can be evaluated without releasing checkpoints during private iterations, with policy servers connecting remotely via a standardized protocol (Chen et al., 5 Jul 2026). Official leaderboard entries require verified evaluation through the online system and later artifact release via XPolicyLab (Chen et al., 5 Jul 2026). Governance is maintained by AI MMLab Club and jointly operated by global academic partners, with no commercial involvement in official evaluation (Chen et al., 5 Jul 2026). Anti-gaming measures include hidden verification layouts, required reporting of simulation means and standard deviations over three seeds, evaluation across all three embodiments in the real world, public release of evaluation videos and scores, and an appeal mechanism for scoring errors (Chen et al., 5 Jul 2026).

A plausible implication is that RealEval treats reproducibility as a socio-technical protocol rather than only a software property: hardware identity, layout replay, cloud governance, and hidden verification are all part of the benchmark definition, not peripheral implementation details.

4. Deployment interface and integration through XPolicyLab

RoboDojo-RealEval is closely tied to XPolicyLab, which provides the unified deployment interface for both simulation and real-world evaluation (Chen et al., 5 Jul 2026). Communication uses WebSocket for persistent bidirectional communication and MessagePack for compact binary serialization of observations and actions, including images and robot states (Chen et al., 5 Jul 2026). The minimal required policy-server methods are update_obs(o_t) or update_obs_batch for batched simulation, get_action() returning an action chunk $a_{t:t+K}$ or get_action_batch for batched inputs, and reset() for per-episode reset (Chen et al., 5 Jul 2026).

The action space in RealEval is expressed in end-effector control and gripper commands. When policies output end-effector targets, Pink, described as Python inverse kinematics based on Pinocchio, is used to convert end-effector targets to robot motions in real robot deployment (Chen et al., 5 Jul 2026). Action chunks are supported to reduce per-step communication overhead (Chen et al., 5 Jul 2026). Dataset capture is fixed at 25 Hz, but the paper does not specify controller loop frequencies for RealEval beyond stating that execution rate is governed by the client-server loop and the Pink planner (Chen et al., 5 Jul 2026).

XPolicyLab also standardizes package organization and experiment parameters. The standardized package layout includes install.sh, process_data.sh, train.sh, eval.sh, model.py, deploy.py, and deploy.yml, while unified experiment parameters include dataset, task, checkpoint, robot config, action type, and seed (Chen et al., 5 Jul 2026). One-time integration is emphasized: policies integrated into XPolicyLab once can be evaluated in both simulation and RealEval with the same interface and minimal policy-side adaptation (Chen et al., 5 Jul 2026).

This interface design places RoboDojo-RealEval in a distinctive position relative to prior evaluation systems. BenchBot provides a unified sim-real Python API built around task-defined observations and actions, ROS-based integration, and containerized or native submissions (Talbot et al., 2020). RoboChallenge instead exposes a low-level asynchronous remote robot API in which users keep inference on their side, send capture requests, and push time-stamped actions into a FIFO queue (Yakefu et al., 20 Oct 2025). RoboDojo-RealEval shares the remote evaluation orientation of RoboChallenge while retaining a unified sim-and-real integration path through XPolicyLab, and it differs from BenchBot in being explicitly centered on standardized real-world manipulation evaluation across three collaborative bimanual embodiments rather than on a broader software suite spanning photorealistic simulation and real robots (Chen et al., 5 Jul 2026, Talbot et al., 2020, Yakefu et al., 20 Oct 2025).

5. Evaluation protocol, task suite, and scoring

RoboDojo-RealEval evaluates 18 real-world tasks, each executed with 10 trials per policy (Chen et al., 5 Jul 2026). Before each trial, a pre-collected evaluation layout is replayed via overlay to match initial conditions (Chen et al., 5 Jul 2026). The maximum execution horizon for each task is derived from demonstrations by multiplying the 90th percentile of demonstration trajectory length by 1.5, and termination occurs at the step limit (Chen et al., 5 Jul 2026). Manual stop applies for unsafe actions, while emergency stop returns the robot to a safe reset state (Chen et al., 5 Jul 2026).

Scoring is video-based and human-judged under controlled protocols. Three independent evaluators score each video under double-blind conditions; large discrepancies are filtered; and the final score is the average of the three (Chen et al., 5 Jul 2026). Scoring captures both final success and intermediate sub-steps, and all evaluation videos are released for leaderboard entries (Chen et al., 5 Jul 2026). The reported metrics are success rate, defined as binary completion, and mean score, defined as graded partial progress (Chen et al., 5 Jul 2026). The paper does not provide a formal LaTeX aggregator for real-world metrics, although it notes that aggregate performance is reported across embodiments and tasks on the real leaderboard (Chen et al., 5 Jul 2026).

The 18 tasks are divided by embodiment. For ARX X5, the tasks are insert_tubes, make_bread, make_food, pack_and_pour_fruit, cover_blocks, and store_in_safe (Chen et al., 5 Jul 2026). For Piper, the tasks are fill_pen_holder, put_objects_into_basket, stack_and_cover_blocks, stack_bowls, stand_up_bottles, and insert_charger (Chen et al., 5 Jul 2026). For Piper X, the tasks are disassemble_LEGO, cap_pen, hang_mugs, classify_objects, sweep_blocks, and pack_objects_into_backpack (Chen et al., 5 Jul 2026). The task descriptions include objectives such as inserting test tubes into rack slots, placing bread slices into a toaster, covering and uncovering colored blocks in a specified order, inserting a charger plug into a power strip, capping a pen by aligning pen and cap, and sweeping blocks with coordinated use of a broom and dustpan (Chen et al., 5 Jul 2026).

The benchmark is intended to cover memory, generalization, precision, and long-horizon behavior, but the paper states that dimension-specific scoring formulas are not defined for the real-world split (Chen et al., 5 Jul 2026). Formal success thresholds, such as numerical tolerance values, are likewise not specified; instead, scoring captures full success and partial sub-step completions (Chen et al., 5 Jul 2026).

A common misunderstanding is that RealEval uses fully automated scoring. In fact, the benchmark relies on automatic video collection and upload to the cloud, but the final trial score is determined through three independent human evaluators under a double-blind protocol (Chen et al., 5 Jul 2026). The design therefore combines automation in data capture and governance with human judgment in task outcome assessment.

6. Data, reproducibility, results, and relation to adjacent benchmarks

RoboDojo-RealEval includes a real-world training dataset for policies that use demonstrations. The paper reports 1,800 trajectories across 18 tasks and 3 embodiments, totaling 1,611,841 frames and 17.91 hours of data at 25 Hz (Chen et al., 5 Jul 2026). The per-embodiment breakdown is ARX X5 with 665,071 frames, 7.39 h, and 600 trajectories; Piper with 539,737 frames, 6.00 h, and 600 trajectories; and Piper X with 407,033 frames, 4.52 h, and 600 trajectories (Chen et al., 5 Jul 2026). Modalities include synchronized RGB from the head and two wrist cameras, robot states, and language annotations (Chen et al., 5 Jul 2026).

The paper reports repeated-run stability statistics for the real-world evaluation. Overall standard deviation in success rate is given as 1.0 percentage point for $\pi_{0.5}$, 1.2 percentage points for GalaxeaVLA, and 1.3 percentage points for InternVLA-A1; overall standard deviation in score is 0.5 for $\pi_{0.5}$, 1.0 for GalaxeaVLA, and 1.2 for InternVLA-A1 (Chen et al., 5 Jul 2026). It also reports an efficiency figure for real-world evaluation: using an RTX 4090 policy server within a local-area network, total time for all 18 tasks × 10 trials was 202.0 minutes, corresponding to approximately 3.4 hours for 180 trials, with an average of 11.2 minutes per task for 10 trials (Chen et al., 5 Jul 2026).

On the RealEval leaderboard, 10 representative policies were evaluated across 18 tasks with 10 trials per task, for 180 trials per policy (Chen et al., 5 Jul 2026). The best overall reported performance is $\pi_{0.5}$ with overall average score 22.9 and success rate 12.8% (Chen et al., 5 Jul 2026). Humans using teleoperation achieve 100% success and 100 score across all tasks and embodiments (Chen et al., 5 Jul 2026). The paper notes that partial progress often does not translate to full completion, with higher scores than success rates indicating sub-step execution without final success; it also reports deployment-specific issues including action jitter, oscillatory motions, contact instability, repeated ineffective actions, and occasional safety-critical behaviors requiring manual stops (Chen et al., 5 Jul 2026).

In relation to preceding systems, BenchBot can be read as an important antecedent in unifying simulated and real evaluation through a consistent sensorimotor abstraction, YAML-defined tasks and environments, automated platform setup, logging, and task-specific evaluation modules (Talbot et al., 2020). Its emphasis, however, is on a software suite for benchmarking robotics research across photorealistic 3D simulation and real robots, with current focus on semantic scene understanding and extensibility toward navigation, exploration, mapping, and manipulation (Talbot et al., 2020). RoboChallenge, by contrast, operationalizes large-scale real-robot evaluation for embodied policies and Vision-Language-Action models through remote low-level APIs, Visual Task Reproduction for reset, stage-based grading, and a fleet of 10 robots across 4 platform types, centered on the Table30 benchmark of 30 table-centric tasks (Yakefu et al., 20 Oct 2025). RoboDojo-RealEval overlaps with RoboChallenge in its commitment to remote access, standardized reset, public logs or videos, and large-scale real-robot evaluation, but differs in its integration into a unified sim-and-real benchmark, its XPolicyLab deployment pathway, and its standardized three-embodiment bimanual platform design (Chen et al., 5 Jul 2026, Yakefu et al., 20 Oct 2025).

The main limitations stated for RoboDojo-RealEval are structural rather than incidental. RealEval tasks are not one-to-one aligned with simulation tasks; controller loop frequencies are not specified; formal success tolerances are not numerically defined; and the platform exposes physical deployment challenges that can reorder rankings relative to simulation because of perception noise, calibration errors, actuation latency, controller dynamics, and accumulated drift (Chen et al., 5 Jul 2026). This suggests that RoboDojo-RealEval is intended less as a final arbiter of general robotic competence than as a standardized physical stress test for deployed manipulation policies under controlled but nontrivial real-world conditions.