---
title: GM-100 Benchmark for Embodied AI
url: https://www.emergentmind.com/topics/gm-100-benchmark
type: topic
---

# GM-100 Benchmark for Embodied AI

Searching arXiv for the GM-100 paper and the specific baseline/platform papers mentioned.
Searching arXiv for "The Great March 100: 100 Detail-oriented Tasks for Evaluating Embodied AI Agents".
GM-100, short for Great March 100, is a benchmark for embodied AI and robot learning that comprises 100 carefully curated, detail-oriented tasks designed to evaluate robotic agents across a broad spectrum of human–object interactions, including both high-frequency primitives and long-tail behaviors. It is presented as the first step toward a “Robot Learning Olympics,” with the explicit goals of providing a unified and systematic task design standard, revealing nuanced differences among embodied AI methods, and encouraging diversity and complexity in dataset creation and model evaluation rather than concentrating evaluation on a small set of common tasks [2601.11421].

## 1. Definition, objectives, and benchmark scope

GM-100 is defined as a benchmark of 100 tasks intended to comprehensively evaluate the capabilities of robotic agents in real-world manipulation settings. Its task set spans interactions from common primitives such as “pick and place” to rarer behaviors such as threading small fruits on skewers, striking a small ball into a goal, and folding along a precise crease [2601.11421]. The benchmark is explicitly motivated by the concern that prior datasets and task designs often lack systematic consideration and principles, making it difficult to determine whether evaluations on a few common tasks genuinely reflect the differentiated performance of methods proposed by different teams and tested under different task distributions [2601.11421].

The benchmark’s stated objectives are threefold: to cover a broad spectrum of human–object interactions, to establish a unified and systematic task design standard, and to promote diversity and complexity in both dataset construction and model evaluation. The paper characterizes this as an attempt to avoid the “one-trick pony” effect associated with earlier benchmarks focused on a narrow subset of manipulation skills [2601.11421].

A central feature of GM-100 is its emphasis on long-tail behaviors. Roughly 40–50% of the task set consists of low-frequency primitives, which distinguishes the benchmark from evaluation suites dominated by frequent verbs such as “pick,” “hold,” and “place” [2601.11421]. This suggests that GM-100 is intended not merely as a scaling-up of task count, but as a redistribution of evaluation mass toward underrepresented interaction types.

## 2. Task taxonomy and design methodology

The 100 tasks are grouped into ten semantic categories, each corresponding to a cluster of related human–object interaction primitives. The category structure is as follows [2601.11421]:

| Category | Task count |
|---|---:|
| Pick & Place (General manipulation) | 20 |
| Tool Use (knife, screwdriver, hammer, etc.) | 15 |
| Structural Manipulation (folding, bending) | 10 |
| Deformable-Object Manipulation (clay, dough) | 10 |
| On/Off Controls (buttons, switches) | 10 |
| Sorting & Sequencing | 8 |
| Assembly & Threading | 10 |
| Environment Interaction (drawers, lids) | 7 |
| Cleaning & Organization (tossing, sorting) | 10 |
| Multi-step Complex Tasks (subtask chains) | 10 |

This taxonomy covers both single-step and multi-step tasks and includes object classes ranging from rigid solids to deformables, articulated objects, and tools. The benchmark therefore treats manipulation not as a homogeneous setting but as a structured space with qualitatively different affordance profiles and control requirements [2601.11421].

The task design procedure is described as a systematic pipeline with five components. First, prior task lists, including Agibot, Open X-Embodiment, and $\pi_{0.5}$, were collected, deduplicated, and analyzed via verb-frequency distributions to expose concentration around high-frequency actions [2601.11421]. Second, the benchmark used human–object interaction primitives derived from HAKE, PaStaNet, and OCL to identify a superset of actions covering both common and rare interactions [2601.11421]. Third, Qwen3 was used for optionally guided LLM generation to expand action–object pairs into concrete task descriptions after verb-sense disambiguation and object enumeration [2601.11421]. Fourth, a hybrid filtering stage combined LLM-based executability scoring with human expert review to ensure physical feasibility on current hardware, clear and objective success criteria, and the avoidance of excessive safety or teleoperation complexity [2601.11421]. Fifth, task instantiation added detailed interaction criteria, such as “apply sawing motion” or “hold pressed for 1–2 s,” along with object purchase links and template demonstration videos [2601.11421].

Methodologically, this design process places GM-100 between a purely data-driven benchmark and a purely hand-authored one. A plausible implication is that the benchmark is attempting to preserve coverage and diversity while retaining operational rigor in task definition.

## 3. Platforms, trajectories, and data collection protocol

GM-100 includes a trajectory dataset collected on two robotic platforms: Agilex Cobot Magic and Dobot Xtrainer [2601.11421]. Cobot Magic is described as using a Mobile-Aloha-style base with a forward-reaching arm and a head-mounted RGB camera, whereas Xtrainer uses an Aloha-style base with an inward-folding arm and a top-down RGB camera [2601.11421]. Both platforms employ low-cost, compliant grippers and teleoperation interfaces based on Mobile ALOHA [2601.11421].

The dataset contains more than 13,000 demonstrations distributed across the 100 tasks, with 100 training trajectories and 30 test trajectories per task [2601.11421]. Trajectory lengths vary by task, averaging between 20 and 200 time steps. The object inventory includes approximately 200 unique purchasable items spanning solids, deformables, articulated objects, and tools [2601.11421]. The action space includes gripper open/close, Cartesian position deltas, orientation adjustments, and force-sensitive compliance [2601.11421].

The collection protocol includes controlled perturbation of initial conditions. Each of the 100 training demonstrations for a task uses randomized initial positions and orientations within a bounded range, and the test set mirrors the same distribution to preserve comparability [2601.11421]. This is significant because it ensures that evaluation does not collapse into replay under a single canonical scene arrangement. Instead, benchmark performance reflects at least limited robustness to variation in scene initialization.

The benchmark is therefore neither purely static nor fully open-ended. It occupies an intermediate regime in which held-out test initializations are distribution-matched to randomized training setups. This suggests that GM-100 is designed primarily to measure generalization within bounded operational variability rather than unconstrained transfer across radically different embodiments or environments.

## 4. Evaluation metrics and experimental protocol

GM-100 uses three principal classes of evaluation metric: Success Rate (SR), Partial Success Rate (PSR), and action prediction error [2601.11421]. Success Rate is defined as

$$
\mathrm{SR} = \frac{N_{\mathrm{success}}}{N_{\mathrm{total}}} \times 100\%.
$$

A trajectory is counted as successful if it satisfies the task’s terminal criteria within a fixed time budget [2601.11421]. This makes SR the strictest task-level metric in the benchmark.

For tasks composed of $S$ subtasks, Partial Success Rate is defined as

$$
\mathrm{PSR}
=
\frac{1}{N_{\mathrm{total}}}
\sum_{i=1}^{N_{\mathrm{total}}}
\frac{s_i}{S}
\times 100\%,
$$

where $s_i$ denotes the number of subtasks correctly executed in demonstration $i$ [2601.11421]. PSR is intended to capture graded competence on multi-stage tasks for which terminal success may be sparse even when partial progress is meaningful.

Action prediction error is reported using both mean squared error and $\ell_1$ loss over a fixed overlapping prediction window across all models:

$$
\mathrm{MSE}
=
\frac{1}{N}\sum_{t=1}^{N}\|\hat a_t-a_t\|_2^2,
$$

$$
\mathrm{L1}
=
\frac{1}{N}\sum_{t=1}^{N}\|\hat a_t-a_t\|_1.
$$

The train/test split is fixed at 100 demonstrations for training and 30 held-out initializations for testing per task, and each model executes the 30 test initializations once per task [2601.11421]. Results are aggregated by reporting average SR, PSR, and action error across all 100 tasks, with task-level breakdowns and heatmaps also provided [2601.11421].

The coexistence of SR and PSR is notable. A frequent misconception in robotics benchmarking is that binary success alone is sufficient for comparing policies across heterogeneous tasks. GM-100 rejects that premise by explicitly incorporating partial-completion accounting for subtask-structured manipulation. This is especially relevant for long-horizon tasks, where binary success can be too sparse to expose meaningful differences among policies.

## 5. Baseline models and empirical findings

The benchmark evaluates three baseline models: DP, $\pi_{0}$, and $\pi_{0.5}$ [2601.11421]. DP is Diffusion Policy based on Chi et al., finetuned from scratch for each task using the LeRobot framework [2601.11421]. $\pi_{0}$ is a Vision-Language-Action flow model finetuned on 100 trajectories per task using OpenPi, and $\pi_{0.5}$ is described as the successor to $\pi_{0}$ with architectural refinements for open-world generalization [2601.11421]. The paper additionally notes that GR00T and other VLA models are in progress and will be released on the public leaderboard [2601.11421].

On Dobot Xtrainer, for Tasks 0001–0010, the reported averages are PSR of 7.0 for DP, 32.1 for $\pi_{0}$, and 53.9 for $\pi_{0.5}$, alongside SR of 1.6 for DP, 4.4 for $\pi_{0}$, and 24.9 for $\pi_{0.5}$ [2601.11421]. Individual tasks show similarly wide separation. For example, Task 0001 reports PSR values of 2.5, 45.8, and 72.5 for DP, $\pi_{0}$, and $\pi_{0.5}$ respectively, with corresponding SR values of 0.0, 13.3, and 36.7 [2601.11421]. Task 0010 reports PSR values of 0.0, 10.0, and 36.7 and SR values of 0.0, 10.0, and 36.7 for the same models [2601.11421].

The benchmark reports several performance trends. $\pi_{0.5}$ consistently outperforms both DP and $\pi_{0}$, often by a factor of 2–3$\times$ in SR and PSR [2601.11421]. DP struggles on multi-step tasks and fine-force primitives, with average SR below 2% [2601.11421]. $\pi_{0}$ demonstrates moderate competence on single-step tasks but underperforms on long-tail interactions such as threading and folding [2601.11421]. Heatmaps on Cobot Magic show the same qualitative pattern, with tool-use and deformable-object tasks remaining especially difficult; even $\pi_{0.5}$ remains below 30% PSR on these categories [2601.11421].

These findings position GM-100 as a discriminative benchmark rather than a saturated one. The reported performance levels are sufficiently low and sufficiently separated to expose meaningful differences among current VLA models.

## 6. Benchmark difficulty, limitations, and proposed extensions

GM-100 identifies three principal challenge areas. First, long-tail and multi-step tasks remain difficult, particularly tasks involving precise force control, such as folding cardboard and stamping, or sequential primitives, such as threading hawthorns; these yield very low SR, below 5% for DP and $\pi_{0}$ [2601.11421]. Second, the benchmark observes under-sampled modes: even with 100 demonstrations per task, some subgoals remain rare, which induces overfitting to dominant strategies [2601.11421]. Third, hardware constraints—including compliance, limited end-effector dexterity, and camera-induced occlusions—create noise and failure modes that do not appear in synthetic or open-loop settings [2601.11421].

These limitations are important for interpreting benchmark scores. Poor performance cannot automatically be attributed solely to policy architecture, because embodiment constraints and sensing geometry materially shape the operational difficulty of the tasks. Conversely, the benchmark’s insistence on real hardware prevents the simplification of manipulation evaluation into purely kinematic trajectory matching.

The paper outlines several future extensions: integrating tactile sensing, force-torque sensing, and depth cameras; constructing more complex task compositions by chaining GM-100 tasks into longer pipelines under the label GM-X; developing cross-platform policy-distillation pipelines from Cobot Magic to Xtrainer and beyond; building a photorealistic standardized simulation proxy; and extending evaluation beyond SR and PSR to include completion time $T_{\mathrm{complete}}$, path length $L=\sum_t \|\Delta p_t\|$, and safety-violation rates [2601.11421].

Taken together, these proposed directions indicate that GM-100 is intended as a foundation rather than a closed benchmark. A plausible implication is that its long-term role is to anchor a family of interoperable evaluations spanning real-world execution, simulation, richer sensing, and cross-platform transfer.

## 7. Position within embodied AI evaluation

GM-100 is presented in direct response to the proliferation of robot learning and imitation learning datasets whose task designs, according to the paper, often lack systematic principles [2601.11421]. Its distinguishing characteristic is not simply the number 100, but the combination of systematic corpus analysis, primitive-based expansion, hybrid feasibility filtering, real-world trajectory collection on multiple platforms, and evaluation procedures that separate terminal success from partial completion [2601.11421].

Within embodied AI evaluation, the benchmark therefore serves two roles. First, it is an empirical testbed for current VLA and imitation-learning systems under diverse manipulation conditions. Second, it is a proposal for task-design methodology: the benchmark treats task coverage, long-tail behavior inclusion, and explicit interaction criteria as first-class design variables rather than incidental byproducts of dataset collection [2601.11421].

A common misunderstanding would be to reduce GM-100 to a harder pick-and-place suite. The benchmark’s taxonomy and reported results do not support that interpretation. Tool use, deformable-object manipulation, structural manipulation, threading, environment interaction, and multi-step complex tasks are integral components of the benchmark and contribute directly to its difficulty profile [2601.11421]. For researchers concerned with benchmark validity, this broader distribution of task types is the central reason GM-100 is positioned as a more diagnostic evaluation of embodied competence.

Source: https://www.emergentmind.com/topics/gm-100-benchmark