RoboDojo Score: An Overview

• RoboDojo Score is a continuous performance metric that evaluates partial progress and successes, scoring across five dimensions—Generalization Precision, Long-Horizon, Memory, and Open instruction following—rather than just binary task completions (success rate).
• The evaluation ensures robustness with varied episodes and task types; techniques such as spatial tolerances, complex end scenarios, smooth motion, alignment, insertion, and interaction manipulation are stressed during testing, for instance, stacking blocks and solving equations.
• The hierarchical structure of the RoboDojo Score aggregates results by averaging task-level scores within each dimension, then averaging across dimensions, rather than pooling all episodes together—Generalization, for example, requires robustness against many environmental changes, highlighting different complexities than precision tasks.

RoboDojo Score is the aggregate manipulation-performance metric used by RoboDojo, a unified simulation-and-real-world benchmark for generalist robot manipulation policies. It is distinct from success rate: success rate measures binary completion of the complete task, whereas Score is a graded process metric that assigns partial credit for intermediate progress. In the principal simulation track, task results are aggregated within five capability dimensions—Generalization, Precision, Long-Horizon, Memory, and Open instruction following—and the five dimension-level values are averaged with equal weight. The benchmark does not define a single scalar that combines simulation and real-world results, and the exact task-level partial-credit mapping is not fully specified in the original RoboDojo paper (Chen et al., 5 Jul 2026).

1. Definition and distinction from success rate

RoboDojo reports two related metrics:

• Score: a continuous process-performance metric that captures partial task progress.
• Success rate (SR): the percentage of episodes satisfying the environment’s complete-task success criterion.

A failed episode contributes zero to the binary success indicator, but it can contribute a positive Score if the policy completes intermediate subtasks. Consequently, Score may substantially exceed SR. For example, the reported simulation result for Hy-Embodied-0.5-VLA is 13.07 Score / 8.80% SR, while the result for $\pi_{0.5}$ is 11.41 Score / 6.91% SR (Chen et al., 5 Jul 2026).

Subsequent RoboDojo evaluations describe Score as the mean process reward multiplied by 100. If $r_i$ is the normalized process reward for episode $i$, the task-level form is:

$$\operatorname{Score}=100\cdot\frac{1}{N}\sum_{i=1}^{N}r_i.$$

The normalized process reward is generally represented on a $0$–$1$ scale before multiplication by 100. Equivalently, if the environment already reports process rewards on a $0$–$100 scale, the reported task value is their arithmetic mean. SR is computed independently from a Boolean full-success predicate:$$\operatorname{SR}=100\cdot\frac{\#\{\text{episodes satisfying the full-success predicate}\}}{N}.$$</p> <p>This distinction is explicit in evaluations of GPT-6 <a href="https://www.emergentmind.com/topics/astra" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Astra</a> and RoboDoctor, where process reward and binary success are separately reported (<a href="/papers/2609.24170" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Zhang et al., 21 Sep 2026</a>, <a href="/papers/2610.35604" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Yu et al., 26 Oct 2026</a>).</p> <p>The Score should therefore not be interpreted as a success percentage, a human-normalized score, or a probability of complete task completion. A value of 28.97, for example, does not mean that 28.97<h2 class='paper-heading' id='hierarchical-aggregation-in-simulation'>2. Hierarchical aggregation in simulation</h2> <p>RoboDojo-Sim aggregates results hierarchically rather than by pooling every episode into one flat mean. The procedure is:</p> <ol> <li>Compute a process Score and SR for each task.</li> <li>Average task-level results within each capability dimension.</li> <li>Average the five dimension-level results with equal weight.</li> </ol> <p>For a dimension$$d$containing task set$r_i$0, the task-balanced dimension Score is:

$r_i$1

The corresponding dimension-level SR is aggregated analogously. The overall simulation metric is:

$r_i$2

with an equivalent expression for overall SR.

The five simulation dimensions contain different numbers of tasks:

Dimension	Tasks
Generalization	12
Precision	8
Long-Horizon	8
Memory	6
Open	8
Total	42

Equal dimension weighting means that Generalization contributes 20% of the overall metric despite containing 12 tasks, while Memory also contributes 20% despite containing six tasks. The official aggregation is therefore not a flat episode-weighted mean over the 42 tasks. This distinction is central to interpreting leaderboard rankings (Chen et al., 5 Jul 2026).

Generalization results include standard and randomized conditions. The randomized setting changes backgrounds, lighting, clutter, layouts, and task-relevant objects. Later evaluations state that Generalization task cells average standard and randomized conditions before the overall five-dimension aggregation (Zhang et al., 21 Sep 2026).

3. Dimensions and evaluation protocols

RoboDojo-Sim evaluates five capability dimensions.

Generalization contains 12 tasks involving unseen backgrounds, lighting variation, clutter, object variation, and scene or layout changes. Standard episodes measure in-distribution performance, whereas randomized episodes test robustness to environmental changes.

Precision contains eight tasks requiring tight spatial tolerances, smooth motion, stable contacts, alignment, insertion, or other fine-grained interactions. Examples include fasten_screws, insert_tubes, plug_in_charger, deposit_coin, insert_key, and build_tower.

Long-Horizon contains eight tasks requiring multiple dependent actions, including put_bottles_into_dustbin, play_tic_tac_toe, classify_objects, organize_table, and make_kong. Score is particularly informative for this dimension because a policy can complete early stages while failing before the final state.

Memory contains six tasks requiring information persistence across observations, including cover_blocks, match_and_pick_from_conveyor, swap_T, press_by_number, imitate_sorting_sequence, and swap_blocks.

Open instruction following contains eight evaluation-only tasks involving open-semantic grounding and recombination of familiar skills under new language or task specifications. Examples include align_blocks, solve_equation, stack_blocks_by_language, general_pickup, and pour_by_language. The Open dimension is excluded from the simulation training set (Chen et al., 5 Jul 2026).

The original official simulation protocol uses 50 episodes per task for a single seed, with most leaderboard policies evaluated using three seeds. Thus, a three-seed evaluation corresponds to 150 trials per task, although dimension-level aggregation is described as averaging the seed-level results. Generalization episodes are divided into standard and randomized subsets. Real-world evaluations use 10 trials per task across 18 tasks, producing 180 trials per policy (Chen et al., 5 Jul 2026).

Some later studies use different protocols. GPT-6 Astra was evaluated on all 42 tasks with 50 episodes per task and one evaluation seed, producing 2,100 trials. DeepSeek-Flash used 10 episodes per task, producing 420 trials, so its results have greater sampling variability and are not directly equivalent in statistical precision to the official 50-episode evaluations (Zhang et al., 21 Sep 2026). Auriga used three seeds and 50 cases per task, yielding 6,300 cases, while uni-robot-agent reports a principal evaluation with three random seeds and two layouts, yielding 252 episodes (Wang et al., 31 Oct 2026, Yuan et al., 1 Nov 2026).

4. Per-episode scoring, normalization, and unresolved specification

The benchmark’s process Score is based on a native episode score that permits partial credit. Subsequent papers characterize this native score as normalized to $r_i$3 and multiplied by 100 for reporting. A task-level Score is therefore the mean of the valid episode-level native scores on the reported scale.

The original RoboDojo paper, however, does not fully specify the raw-to-score mapping. It does not state:

• the exact function converting subtask completion into a numerical value;
• whether subtasks are equally weighted;
• whether scoring is object-, step-, milestone-, or subtask-based;
• whether partial failures receive linear credit;
• whether task types use different normalization;
• whether scores are clipped or rescaled;
• how safety termination affects partial credit.

A plausible interpretation is that the native process reward reflects the number or weight of completed required subtasks, but this is not an author-stated universal formula in the original paper. The benchmark code or task documentation is required to reproduce the exact episode-level mapping.

The score scale appears approximately 0–100. Reported human references include a human simulation Score of 80.42 and a human real-world Score of 100.0. RoboDojo does not define a human-normalized metric such as:

$r_i$4

Accordingly, 13.07 should not be described as 13.07% of human performance. Dividing 13.07 by the reported human simulation Score of 80.42 would produce an external comparison of approximately 16.2%, but that calculation is not the published RoboDojo normalization (Chen et al., 5 Jul 2026).

RoboDojo also does not define a difficulty-weighted score, a random-policy-normalized score, or a combined simulation-and-real-world Score. There is no published quantity of the form:

$r_i$5

Simulation and real-world results are reported in separate tracks.

5. Real-world scoring and evaluation integrity

RoboDojo-RealEval uses concepts analogous to simulation Score and SR but does not use exactly the same evaluation protocol. The real-world benchmark includes six tasks for each of three embodiments:

• ARX X5: cover_blocks, make_bread, make_food, pack_and_pour_fruit, store_in_safe, and insert_tubes;
• Piper: stack_and_cover_blocks, fill_pen_holder, put_objects_into_basket, insert_charger, stack_bowls, and stand_up_bottles;
• Piper X: classify_objects, disassemble_LEGO, hang_mugs, pack_objects_into_backpack, sweep_blocks, and cap_pen.

Each task is evaluated over 10 trials. Three evaluators independently score each video, accounting for final success and intermediate substep completion. The final score for trial $r_i$6 of task $r_i$7 is the average of the three evaluator scores:

$r_i$8

The per-task real-world Score is then the mean over 10 trials, while SR is the percentage of trials judged complete successes. Overall results are averaged across the 18 tasks, equivalently across the three embodiments because each embodiment contributes six tasks (Chen et al., 5 Jul 2026).

Real-world evaluations also involve evaluator disagreement filtering: samples with large evaluator discrepancies are filtered before averaging, although the paper does not specify the discrepancy threshold or exact filtering rule. Unsafe behavior can lead to manual termination, including table impacts, collisions with the external frame, self-collision, or behavior that could damage the platform. The paper does not specify a separate numerical penalty for early safety termination.

For simulation, episodes terminate at task-specific horizons based on demonstration trajectory lengths. The stated horizon is $r_i$9 times the 90th percentile of trajectory length, with a larger multiplier of $i$0 for short automatically synthesized tasks. Real-world horizons use $i$1 times the 90th percentile of demonstration length. Failed or incomplete trials can retain positive partial Score but receive zero binary success.

Official submissions are subject to hidden-layout verification. A submission whose hidden-verification SR differs significantly from the public-set SR is excluded from the verified leaderboard, although the numerical threshold is not provided. Submissions lacking the evaluated checkpoint, code, configurations, and reproducibility instructions are assigned to a separate closed-source, unverified track (Chen et al., 5 Jul 2026).

6. Reported scores and interpretation

Reported RoboDojo Scores vary substantially across policies and harnesses. Representative simulation results from the original benchmark include:

Policy	Score	SR
Hy-Embodied-0.5-VLA	13.07	8.80%
Spatial Forcing	12.38	8.04%
$i$2	11.41	6.91%
X-VLA	10.13	6.52%
Human Expert	80.42	76.03%

GPT-6 Astra later achieved 28.97 Score / 22.48% SR over the official 2,100-trial simulation protocol, ranking above the public policies in the reported comparison. Its dimension Scores were 33.36 for Generalization, 12.65 for Precision, 21.45 for Long-Horizon, 43.04 for Memory, and 34.36 for Open, whose unweighted mean is 28.97 (Zhang et al., 21 Sep 2026).

Astra’s profile was strongly polarized. It performed comparatively well on semantic understanding, open-ended instructions, and several memory or generalization tasks, while performing poorly on precision, dynamic control, contact-rich manipulation, collision avoidance, and complex bimanual coordination. Its SR was 60.0% on push_T, 60.0% on arrange_largest_number, 50.0% on align_blocks, and 40.0% on solve_equation, but 0.0% on make_kong, 2.0% on build_tower, and 0.0% on insert_tubes.

The interaction harness can substantially affect RoboDojo Score. With GPT-6 Astra, Auriga reported 64.00 Score / 57.74% SR over 6,300 official cases, compared with 28.97 / 22.48% for the RoboProbe reference interface. Its dimension results were particularly strong on Memory, with 93.40 Score and 92.99% SR, while Precision and Long-Horizon remained more difficult (Wang et al., 31 Oct 2026).

RoboDoctor reported a Refinement Score of 57.10 and native SR of 49.44% on a curated archive of 54 RoboDojo-Sim task entries. Its dimension Scores were 56.98 for Generalization, 47.60 for Precision, 52.92 for Long-Horizon, 77.50 for Memory, and 50.52 for Open instruction following. These results were obtained through hierarchical aggregation over valid selected episodes, with 24 Generalization entries, eight Precision entries, eight Long-Horizon entries, six Memory entries, and eight Open entries (Yu et al., 26 Oct 2026).

The open-weight Qwen3.8-Flash-Next with uni-robot-agent achieved 31.39 Score / 25.00% SR in a three-seed, two-layout evaluation. Its strongest dimensions were Memory at 54.72 Score and Open-vocabulary instruction following at 46.67, while Precision was 10.21 and Long-Horizon was 23.75. The result exceeded the published GPT-6 Astra–RoboProbe result of 28.97, but GPT-6 Astra achieved 51.90 Score when evaluated with the same uni-robot-agent harness, demonstrating that model–harness interactions materially influence the metric (Yuan et al., 1 Nov 2026).

These results establish that a high RoboDojo Score indicates broad average process completion across dimension-balanced manipulation capabilities, not uniform reliability across all tasks. A large gap between Score and SR indicates frequent partial but incomplete behavior. Dimension-level reporting is therefore essential: policies can lead on Generalization, Memory, Precision, Long-Horizon, or Open instruction following without dominating the benchmark as a whole.

The metric also has important comparability limitations. Studies differ in the number of episodes, seeds, layouts, harnesses, available observations, task information, controller interfaces, and outcome-selection procedures. Some evaluations report no confidence intervals, standard deviations, standard errors, or significance tests. RoboDoctor uses a curated archive with selected valid outcomes, while DeepSeek-Flash uses only 10 episodes per task. Published policy comparisons may also be taken from different leaderboard snapshots rather than synchronized reruns.

RoboDojo Score is consequently best understood as a dimension-balanced, 0–100 process-completion metric whose binary counterpart is SR. Its formal aggregation is comparatively clear—episode outcomes are averaged within tasks, tasks within dimensions, and the five dimensions equally overall—but the exact native partial-credit function, cross-modal normalization, difficulty weighting, and unified simulation–real-world score are not defined as universal quantities in the benchmark literature.