---
title: 'RoboDojo-RealEval: Real-World Robotic Benchmark'
url: https://www.emergentmind.com/topics/robodojo-realeval
type: topic
---

# RoboDojo-RealEval: Real-World Robotic Benchmark

Searching arXiv for the cited benchmark papers to ground the article in current records.
RoboDojo-RealEval is the standardized, remote-access real-world evaluation platform in RoboDojo that complements the simulation benchmark with reproducible, deployment-focused testing across three collaborative bimanual robot embodiments. It was introduced as part of RoboDojo, a unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies, and is designed to address the cost, complexity, and reproducibility problems of physical evaluation by standardizing hardware, workspace, lighting, camera mounts, reset procedures, scoring, and deployment interfaces [2607.04434]. Within the broader RoboDojo framework, RealEval serves as the physical-world counterpart to capability-oriented simulation, providing direct evidence of deployed performance under standardized physical conditions while remaining accessible through remote cloud-based evaluation and a unified policy interface via XPolicyLab [2607.04434].

## 1. Position within the RoboDojo benchmark

RoboDojo was introduced to evaluate generalist robot manipulation policies across both simulation and real-world settings, motivated by the observation that many existing benchmarks are limited to simple, short-horizon, or skill-narrow tasks and are often conducted only in simulation or only in the real world [2607.04434]. In this design, RoboDojo includes 42 simulation tasks and 18 real-world tasks covering diverse and complementary manipulation capabilities, while RoboDojo-RealEval specifically denotes the real-world evaluation system with remote cloud access, standardized hardware, scene reset, evaluation protocol, and deployment interface [2607.04434].

The rationale for RealEval is explicitly deployment-oriented. Simulation is described as efficient for capability-oriented diagnosis and rapid model iteration, but insufficient for fully capturing contact-rich dynamics, actuation errors, perception noise, workspace constraints, temporal variations, and safety-critical behaviors that occur during physical deployment [2607.04434]. RealEval therefore does not attempt to replace simulation; instead, it provides standardized physical testing under controlled conditions and a unified policy interface so that policies integrated once can be run both in sim and on real robots with minimal adaptation through XPolicyLab [2607.04434].

A frequent misconception is that a real-world benchmark of this kind should be a direct physical replay of the simulation benchmark. RoboDojo explicitly rejects that assumption: the real-world tasks are not one-to-one matched to the 42 simulation tasks, and RealEval is instead designed to complement simulation with physical deployment challenges [2607.04434]. This separation makes the real benchmark a distinct diagnostic instrument rather than a simple transfer test.

## 2. Physical platform, workspace standardization, and sensing

RoboDojo-RealEval is built around a fixed and explicitly standardized physical platform. The workspace consists of a fixed-height white evaluation table with a 1.2 m × 1.2 m workspace, a replaceable white table cover for consistent visual background and easy replacement, and an external frame enclosing the workspace with dimensions 1.5 m × 1.5 m × 2.1 m [2607.04434]. Black curtains are installed on the left, right, front, and top sides to suppress external illumination, and three linear LED light sources with fixed mounts provide stable lighting over the workspace [2607.04434]. The platform also includes an integrated workstation box and a touchscreen interface [2607.04434].

The robot side comprises three collaborative bimanual platforms: ARX X5, Piper, and Piper X [2607.04434]. The paper characterizes this as a multi-embodiment design that assesses cross-kinematics robustness, and notes that the platform includes a bottom connector to fix the relative distance and pose of the two arm bases [2607.04434]. RealEval is therefore not a single-robot benchmark, but a standardized multi-embodiment evaluation system.

The sensing setup is also fixed. Each platform uses two Gemini 305 wrist cameras, one per wrist, and one Gemini 335L head camera [2607.04434]. The head camera is mounted using a custom steel tube and a 3D-printed mount to fix camera pose, and the fixed mounts are described as reducing incidental variation [2607.04434]. Video capture is at 640 × 480 resolution, while demonstrations and evaluation videos are recorded and synchronized at 25 Hz for dataset collection and scoring [2607.04434]. In the unified observation format used for both simulation and real deployment, observations include RGB images from one head camera and two wrist cameras, robot states including joint positions, end-effector poses, gripper states, and TCP, and language instructions accompanying tasks and demonstrations [2607.04434]. The pose convention is represented as $[x, y, z, q_w, q_x, q_y, q_z]$ [2607.04434].

This level of standardization is central to cross-site reproducibility. The paper specifies identical hardware across RealEval sites, including the same table, enclosure, black curtains, LED lights, fixed mounts for arm bases and cameras, the same head-camera mounting design, and identical Gemini 305 and Gemini 335L cameras [2607.04434]. Fixed geometry and layout replay are used to ensure consistent initial conditions across trials, policies, and sites [2607.04434].

## 3. Reset system, safety mechanisms, and cloud access

A defining feature of RoboDojo-RealEval is its scene reset system. The platform uses a touchscreen-based web interface for layout replay, policy execution, and emergency stop [2607.04434]. During reset, the web interface displays a transparent overlay of the target layout atop the live camera feed, and evaluators reposition objects until the live scene matches the overlay before starting policy execution via touchscreen [2607.04434]. The paper reports a concrete timing figure: with five data collectors, restoring a scene with five objects took 14 seconds on average [2607.04434].

Public evaluation layouts are pre-collected and replayed via overlay before each trial, while hidden verification layouts are used to reduce overfitting and detect gaming [2607.04434]. Reference layouts are shared, and hidden verification layouts supplement fairness checks [2607.04434]. This reset mechanism is not merely ergonomic; it is the main reproducibility device for restoring initial object layouts under fixed hardware and lighting.

Safety is handled through both interface-level and human-supervised procedures. Emergency stop disables policy control immediately and returns the robot slowly to a safe reset state [2607.04434]. Evaluators may also manually terminate trials if unsafe behavior is observed, including hitting the table, colliding with the frame, or self-collision [2607.04434]. If reset cannot be matched due to incidental disturbances, evaluators adjust and verify alignment via the overlay before starting [2607.04434].

Remote access is provided through the RoboDojo online evaluation system. Policies can be evaluated without releasing checkpoints during private iterations, with policy servers connecting remotely via a standardized protocol [2607.04434]. Official leaderboard entries require verified evaluation through the online system and later artifact release via XPolicyLab [2607.04434]. Governance is maintained by AI MMLab Club and jointly operated by global academic partners, with no commercial involvement in official evaluation [2607.04434]. Anti-gaming measures include hidden verification layouts, required reporting of simulation means and standard deviations over three seeds, evaluation across all three embodiments in the real world, public release of evaluation videos and scores, and an appeal mechanism for scoring errors [2607.04434].

A plausible implication is that RealEval treats reproducibility as a socio-technical protocol rather than only a software property: hardware identity, layout replay, cloud governance, and hidden verification are all part of the benchmark definition, not peripheral implementation details.

## 4. Deployment interface and integration through XPolicyLab

RoboDojo-RealEval is closely tied to XPolicyLab, which provides the unified deployment interface for both simulation and real-world evaluation [2607.04434]. Communication uses WebSocket for persistent bidirectional communication and MessagePack for compact binary serialization of observations and actions, including images and robot states [2607.04434]. The minimal required policy-server methods are `update_obs(o_t)` or `update_obs_batch` for batched simulation, `get_action()` returning an action chunk $a_{t:t+K}$ or `get_action_batch` for batched inputs, and `reset()` for per-episode reset [2607.04434].

The action space in RealEval is expressed in end-effector control and gripper commands. When policies output end-effector targets, Pink, described as Python inverse kinematics based on Pinocchio, is used to convert end-effector targets to robot motions in real robot deployment [2607.04434]. Action chunks are supported to reduce per-step communication overhead [2607.04434]. Dataset capture is fixed at 25 Hz, but the paper does not specify controller loop frequencies for RealEval beyond stating that execution rate is governed by the client-server loop and the Pink planner [2607.04434].

XPolicyLab also standardizes package organization and experiment parameters. The standardized package layout includes `install.sh`, `process_data.sh`, `train.sh`, `eval.sh`, `model.py`, `deploy.py`, and `deploy.yml`, while unified experiment parameters include dataset, task, checkpoint, robot config, action type, and seed [2607.04434]. One-time integration is emphasized: policies integrated into XPolicyLab once can be evaluated in both simulation and RealEval with the same interface and minimal policy-side adaptation [2607.04434].

This interface design places RoboDojo-RealEval in a distinctive position relative to prior evaluation systems. BenchBot provides a unified sim-real Python API built around task-defined observations and actions, ROS-based integration, and containerized or native submissions [2008.00635]. RoboChallenge instead exposes a low-level asynchronous remote robot API in which users keep inference on their side, send capture requests, and push time-stamped actions into a FIFO queue [2510.17950]. RoboDojo-RealEval shares the remote evaluation orientation of RoboChallenge while retaining a unified sim-and-real integration path through XPolicyLab, and it differs from BenchBot in being explicitly centered on standardized real-world manipulation evaluation across three collaborative bimanual embodiments rather than on a broader software suite spanning photorealistic simulation and real robots [2607.04434][2008.00635][2510.17950].

## 5. Evaluation protocol, task suite, and scoring

RoboDojo-RealEval evaluates 18 real-world tasks, each executed with 10 trials per policy [2607.04434]. Before each trial, a pre-collected evaluation layout is replayed via overlay to match initial conditions [2607.04434]. The maximum execution horizon for each task is derived from demonstrations by multiplying the 90th percentile of demonstration trajectory length by 1.5, and termination occurs at the step limit [2607.04434]. Manual stop applies for unsafe actions, while emergency stop returns the robot to a safe reset state [2607.04434].

Scoring is video-based and human-judged under controlled protocols. Three independent evaluators score each video under double-blind conditions; large discrepancies are filtered; and the final score is the average of the three [2607.04434]. Scoring captures both final success and intermediate sub-steps, and all evaluation videos are released for leaderboard entries [2607.04434]. The reported metrics are success rate, defined as binary completion, and mean score, defined as graded partial progress [2607.04434]. The paper does not provide a formal LaTeX aggregator for real-world metrics, although it notes that aggregate performance is reported across embodiments and tasks on the real leaderboard [2607.04434].

The 18 tasks are divided by embodiment. For ARX X5, the tasks are `insert_tubes`, `make_bread`, `make_food`, `pack_and_pour_fruit`, `cover_blocks`, and `store_in_safe` [2607.04434]. For Piper, the tasks are `fill_pen_holder`, `put_objects_into_basket`, `stack_and_cover_blocks`, `stack_bowls`, `stand_up_bottles`, and `insert_charger` [2607.04434]. For Piper X, the tasks are `disassemble_LEGO`, `cap_pen`, `hang_mugs`, `classify_objects`, `sweep_blocks`, and `pack_objects_into_backpack` [2607.04434]. The task descriptions include objectives such as inserting test tubes into rack slots, placing bread slices into a toaster, covering and uncovering colored blocks in a specified order, inserting a charger plug into a power strip, capping a pen by aligning pen and cap, and sweeping blocks with coordinated use of a broom and dustpan [2607.04434].

The benchmark is intended to cover memory, generalization, precision, and long-horizon behavior, but the paper states that dimension-specific scoring formulas are not defined for the real-world split [2607.04434]. Formal success thresholds, such as numerical tolerance values, are likewise not specified; instead, scoring captures full success and partial sub-step completions [2607.04434].

A common misunderstanding is that RealEval uses fully automated scoring. In fact, the benchmark relies on automatic video collection and upload to the cloud, but the final trial score is determined through three independent human evaluators under a double-blind protocol [2607.04434]. The design therefore combines automation in data capture and governance with human judgment in task outcome assessment.

## 6. Data, reproducibility, results, and relation to adjacent benchmarks

RoboDojo-RealEval includes a real-world training dataset for policies that use demonstrations. The paper reports 1,800 trajectories across 18 tasks and 3 embodiments, totaling 1,611,841 frames and 17.91 hours of data at 25 Hz [2607.04434]. The per-embodiment breakdown is ARX X5 with 665,071 frames, 7.39 h, and 600 trajectories; Piper with 539,737 frames, 6.00 h, and 600 trajectories; and Piper X with 407,033 frames, 4.52 h, and 600 trajectories [2607.04434]. Modalities include synchronized RGB from the head and two wrist cameras, robot states, and language annotations [2607.04434].

The paper reports repeated-run stability statistics for the real-world evaluation. Overall standard deviation in success rate is given as 1.0 percentage point for $\pi_{0.5}$, 1.2 percentage points for GalaxeaVLA, and 1.3 percentage points for InternVLA-A1; overall standard deviation in score is 0.5 for $\pi_{0.5}$, 1.0 for GalaxeaVLA, and 1.2 for InternVLA-A1 [2607.04434]. It also reports an efficiency figure for real-world evaluation: using an RTX 4090 policy server within a local-area network, total time for all 18 tasks × 10 trials was 202.0 minutes, corresponding to approximately 3.4 hours for 180 trials, with an average of 11.2 minutes per task for 10 trials [2607.04434].

On the RealEval leaderboard, 10 representative policies were evaluated across 18 tasks with 10 trials per task, for 180 trials per policy [2607.04434]. The best overall reported performance is $\pi_{0.5}$ with overall average score 22.9 and success rate 12.8% [2607.04434]. Humans using teleoperation achieve 100% success and 100 score across all tasks and embodiments [2607.04434]. The paper notes that partial progress often does not translate to full completion, with higher scores than success rates indicating sub-step execution without final success; it also reports deployment-specific issues including action jitter, oscillatory motions, contact instability, repeated ineffective actions, and occasional safety-critical behaviors requiring manual stops [2607.04434].

In relation to preceding systems, BenchBot can be read as an important antecedent in unifying simulated and real evaluation through a consistent sensorimotor abstraction, YAML-defined tasks and environments, automated platform setup, logging, and task-specific evaluation modules [2008.00635]. Its emphasis, however, is on a software suite for benchmarking robotics research across photorealistic 3D simulation and real robots, with current focus on semantic scene understanding and extensibility toward navigation, exploration, mapping, and manipulation [2008.00635]. RoboChallenge, by contrast, operationalizes large-scale real-robot evaluation for embodied policies and Vision-Language-Action models through remote low-level APIs, Visual Task Reproduction for reset, stage-based grading, and a fleet of 10 robots across 4 platform types, centered on the Table30 benchmark of 30 table-centric tasks [2510.17950]. RoboDojo-RealEval overlaps with RoboChallenge in its commitment to remote access, standardized reset, public logs or videos, and large-scale real-robot evaluation, but differs in its integration into a unified sim-and-real benchmark, its XPolicyLab deployment pathway, and its standardized three-embodiment bimanual platform design [2607.04434][2510.17950].

The main limitations stated for RoboDojo-RealEval are structural rather than incidental. RealEval tasks are not one-to-one aligned with simulation tasks; controller loop frequencies are not specified; formal success tolerances are not numerically defined; and the platform exposes physical deployment challenges that can reorder rankings relative to simulation because of perception noise, calibration errors, actuation latency, controller dynamics, and accumulated drift [2607.04434]. This suggests that RoboDojo-RealEval is intended less as a final arbiter of general robotic competence than as a standardized physical stress test for deployed manipulation policies under controlled but nontrivial real-world conditions.

Source: https://www.emergentmind.com/topics/robodojo-realeval