RoboChallenge: Real-Robot Benchmark

• RoboChallenge is a large-scale real-robot evaluation benchmark that systematically assesses vision-language-action models for embodied manipulation.
• It integrates multi-robot hardware with synchronized multi-view RGB-D sensors to enable reproducible, remote experiments across diverse manipulator architectures.
• The platform drives progress in sim-to-real transfer, zero-shot generalization, and long-horizon planning through standardized tasks and comprehensive metrics.

RoboChallenge is a large-scale real-robot evaluation benchmark designed to systematically assess the performance, generalization, and robustness of vision-language-action (VLA) models for embodied manipulation. Originating from the need for reproducible and scalable assessment of state-of-the-art robotic control algorithms, RoboChallenge serves as an automated, cloud-based evaluation infrastructure that hosts both the benchmark suite (“Table30”) and an online platform for remote, standard-compliant experiments on industrial robot hardware. This benchmark has catalyzed progress in generalist VLA models, structured comparisons, and analysis of progress in long-horizon embodied manipulation, sim-to-real transfer, and scalable model evaluation (Yakefu et al., 20 Oct 2025).

1. Benchmark Design and System Architecture

RoboChallenge provides a tightly integrated, end-to-end system encompassing multi-robot hardware, real-time scheduling, remote APIs, and a comprehensive data-logging and evaluation backend. The platform comprises a heterogeneous fleet of 10 online robots across four manipulator architectures: UR5 (6-DOF), Franka Panda (7-DOF), Cobot Magic Aloha (dual-arm), and ARX-5. Each cell is instrumented with synchronized overhead, side, and wrist-mounted Intel RealSense RGB-D cameras, providing time-stamped multi-view RGB-D and proprioceptive observations. Action interfaces allow for either joint-space commands or Cartesian end-effector deltas with associated time durations, executed asynchronously via HTTP/gRPC APIs (Yakefu et al., 20 Oct 2025).

A central scheduling server orchestrates job submissions, allocates robot resources, and dispatches jobs to available cells, modeled as an optimization problem for minimizing batch completion time. A distributed experiment runner manages initialization, resource allocation, results archiving, and health monitoring, enabling seamless parallel evaluation and rapid fault recovery. All experiment data, including raw sensor streams, joint states, action logs, and high-resolution video, are archived for offline analysis. Strict version and calibration protocols ensure reproducibility across releases (Yakefu et al., 20 Oct 2025).

2. The Table30 Suite: Tasks, Protocols, and Metrics

The Table30 suite consists of 30 tabletop manipulation tasks, encompassing a diverse range of settings—pick-and-place, stacking, articulated-object actions (e.g., drawers, switches), sorting and search, tool usage, and compositionally challenging instructions (e.g., “stack color blocks,” “wipe the table”). Tasks are specified solely via natural-language descriptions and require robust grounding, perception, and sequential planning under realistic noise and environmental variations. Each task is defined by a set of sequential subgoals; trial success is binary, with partial progress recorded through progress scores (Yakefu et al., 20 Oct 2025, Ye et al., 13 Apr 2026).

Formally, for $N$ independent rollouts, the success rate is

$$S = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{\text{task}_i\ \text{succeeds}\}$$

and the progress score is

$$\overline P = \frac1N \sum_{i=1}^N P_i, \qquad P_i = \sum_{j=1}^M p_j\,\mathbf{1}\{\text{stage }j\text{ completed}\} - 0.5\, r_i$$

where $p_j$ is the subgoal weight and $r_i$ the number of retries (capped at 4). Calibration, rigid initial state reproduction, random seed control, and tracked hardware configurations ensure statistical soundness and reproducibility (Yakefu et al., 20 Oct 2025, Ye et al., 13 Apr 2026, Zhang et al., 7 Apr 2026).

3. Model Evaluation, Leaderboards, and Baseline Results

RoboChallenge supports the evaluation of both specialist and generalist VLA models. Researchers fine-tune or adapt their models locally using provided demonstration data (up to 1 000 episodes per task), then integrate their policy via a standard skeleton client for automated deployment and result retrieval. The online leaderboard displays per-task and aggregate metrics, confidence intervals, replay videos, and allows fair, blinded comparisons.

Empirical comparative results demonstrate key trends:

Model	Avg. Success Rate (%)	Avg. Progress Score
$\pi_{0.5}$ (Task-spec.)	43.7	62.2
$\pi_{0}$ (Task-spec.)	28.3	47.6
DM0 (Task-spec., 2B)	62.0	--
DM0 (Generalist, 2B)	37.3	49.08
CogACT	11.7	21.8
$\pi_{0.5}$ Multi	17.7	31.3
A1	29.0	--
StarVLA-$\alpha$ (Zero-shot)	33.6	54.5

Results indicate a pronounced gap between fine-tuned, VLA-native architectures (notably DM0) and older pipeline-based models, as well as strong zero-shot transfer capacity for minimalist but high-capacity VLA baselines (e.g., StarVLA-$\alpha$). Task-level analysis reveals that deformable-object and temporally extended tasks are the most challenging and unsolved by current methods (Yakefu et al., 20 Oct 2025, Yu et al., 16 Feb 2026, Ye et al., 13 Apr 2026).

4. Methodological Innovations and Baseline Architectures

RoboChallenge has served as the proving ground for major advances in VLA model design. Initial pipelines (e.g., $$S = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{\text{task}_i\ \text{succeeds}\}$$0, $$S = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{\text{task}_i\ \text{succeeds}\}$$1) employed per-task finetuning atop pretrained vision-language backbones, flow-matching action heads, and limited multi-modal fusion. Subsequent state-of-the-art models, such as DM0, GigaBrain-0.5M*, and StarVLA-$$S = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{\text{task}_i\ \text{succeeds}\}$$2, exhibit the following technical features:

• Embodied-native pretraining: Large-scale pretraining on mixed web, vision-language, embodied logs, and simulated robot data to ground semantic and physical priors at the model core (Yu et al., 16 Feb 2026).
• Hybrid gradient strategies: Detaching action-expert gradients for embodied (robot) data to preserve VLM semantic generality and enable robust knowledge composition (Yu et al., 16 Feb 2026).
• Hierarchical/Spatial Chain-of-Thought: Structured intermediate objectives—subtasks, spatial bounding, predicted trajectories—inject inductive bias for sequential skill acquisition (Yu et al., 16 Feb 2026).
• Efficient inference: Adaptive inference schemes, early-exit policies across transformer layers, and truncated/warm-started flow matching for inference speed-up without accuracy loss (Zhang et al., 7 Apr 2026).
• Minimal decoherence via unified policy design: StarVLA-$$S = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{\text{task}_i\ \text{succeeds}\}$$3 demonstrates that a clean vision-language regression head, stripped of specialized benchmark engineering, achieves highly competitive transfer (Ye et al., 13 Apr 2026).

The current empirical upper bound for single-task specialist success on Table30 is 62.0% (DM0), with generalist models achieving up to 37.3%—a multi-fold increase over early generalist settings (Yu et al., 16 Feb 2026).

5. Scalability, Reproducibility, and Evaluation Protocols

The RoboChallenge infrastructure is engineered for concurrent, high-throughput, and statistically robust evaluation. Key aspects include:

• Full automation: Users are shielded from hardware idiosyncrasies, with transparent initialization, reset, and calibration at each rollout (Yakefu et al., 20 Oct 2025).
• Distributed scheduling and resource pooling: Centralized job queueing (RabbitMQ or similar), with multi-cell parallelization and heartbeat monitoring for fault tolerance.
• Versioning/distribution control: Strict management of code, demonstration data (using DVC), and hardware configuration YAMLs, supporting reproducibility and post-hoc auditability.
• Confidence estimation: Wald intervals, two-sample statistical tests, and controlled experimental comparisons.
• Comprehensive artifact archiving: All sensorimotor data, videos, and run metadata are retained—enabling in-depth post-analysis and training data re-use (Yakefu et al., 20 Oct 2025).

A plausible implication is that this reproducibility standard sets a reference for future embodied AI and RL benchmarks.

6. Impact on Embodied AI, Open Challenges, and Future Directions

RoboChallenge has precipitated rapid advancement in general-purpose VLA modeling and provided a critical anchor for progress evaluation. Key outcomes include empirical demonstration of true multi-task and zero-shot generalization, identification of persistent performance gaps (e.g., temporally extended, soft-body, and compositional tasks), and the emergence of shared methodological best practices such as spatial scaffolding, world-model augmentation, and data-centric robustness.

Open technical challenges derived from community analysis are:

• Handling soft-body and deformable-object manipulation, which remains largely unsolved even for large models (Yakefu et al., 20 Oct 2025, Yu et al., 16 Feb 2026).
• Long-horizon temporal reasoning and explicit memory, with near-zero scores on tasks requiring sequential planning or recovery (Yakefu et al., 20 Oct 2025).
• Progress-aware metrics that better capture near-success and partial skill execution, supplementing binary success rates (Chen et al., 29 Jun 2025).
• Sim-to-real transfer and robust generalization, with ongoing work towards domain-randomized pipelines, data augmentation, and minimal pipeline specialization (Ye et al., 13 Apr 2026, Team et al., 12 Feb 2026).
• Efficient evaluation at scale, with continuing development of action-chunking, early-exit inference, and hardware abstraction layers to support hundreds of concurrent experiments (Yakefu et al., 20 Oct 2025, Zhang et al., 7 Apr 2026).

RoboChallenge’s standardization, scale, and reproducibility position it as a central reference for the embodied AI and robotic manipulation community, shaping both algorithmic development and empirical validation frameworks (Yakefu et al., 20 Oct 2025, Yu et al., 16 Feb 2026, Ye et al., 13 Apr 2026).