GM-100 Benchmark for Embodied AI

• GM-100 is a comprehensive benchmark comprising 100 curated tasks designed to evaluate robotic agents on both common and rare human–object interactions.
• It employs a systematic task taxonomy and a hybrid design methodology that combines data-driven analysis with expert validation to ensure task clarity and feasibility.
• Baseline evaluations on real robotic platforms reveal significant performance differences among models, highlighting challenges in multi-step and force-critical manipulations.

Searching arXiv for the GM-100 paper and the specific baseline/platform papers mentioned. Searching arXiv for "The Great March 100: 100 Detail-oriented Tasks for Evaluating Embodied AI Agents". GM-100, short for Great March 100, is a benchmark for embodied AI and robot learning that comprises 100 carefully curated, detail-oriented tasks designed to evaluate robotic agents across a broad spectrum of human–object interactions, including both high-frequency primitives and long-tail behaviors. It is presented as the first step toward a “Robot Learning Olympics,” with the explicit goals of providing a unified and systematic task design standard, revealing nuanced differences among embodied AI methods, and encouraging diversity and complexity in dataset creation and model evaluation rather than concentrating evaluation on a small set of common tasks (Wang et al., 16 Jan 2026).

1. Definition, objectives, and benchmark scope

GM-100 is defined as a benchmark of 100 tasks intended to comprehensively evaluate the capabilities of robotic agents in real-world manipulation settings. Its task set spans interactions from common primitives such as “pick and place” to rarer behaviors such as threading small fruits on skewers, striking a small ball into a goal, and folding along a precise crease (Wang et al., 16 Jan 2026). The benchmark is explicitly motivated by the concern that prior datasets and task designs often lack systematic consideration and principles, making it difficult to determine whether evaluations on a few common tasks genuinely reflect the differentiated performance of methods proposed by different teams and tested under different task distributions (Wang et al., 16 Jan 2026).

The benchmark’s stated objectives are threefold: to cover a broad spectrum of human–object interactions, to establish a unified and systematic task design standard, and to promote diversity and complexity in both dataset construction and model evaluation. The paper characterizes this as an attempt to avoid the “one-trick pony” effect associated with earlier benchmarks focused on a narrow subset of manipulation skills (Wang et al., 16 Jan 2026).

A central feature of GM-100 is its emphasis on long-tail behaviors. Roughly 40–50% of the task set consists of low-frequency primitives, which distinguishes the benchmark from evaluation suites dominated by frequent verbs such as “pick,” “hold,” and “place” (Wang et al., 16 Jan 2026). This suggests that GM-100 is intended not merely as a scaling-up of task count, but as a redistribution of evaluation mass toward underrepresented interaction types.

2. Task taxonomy and design methodology

The 100 tasks are grouped into ten semantic categories, each corresponding to a cluster of related human–object interaction primitives. The category structure is as follows (Wang et al., 16 Jan 2026):

Category	Task count
Pick & Place (General manipulation)	20
Tool Use (knife, screwdriver, hammer, etc.)	15
Structural Manipulation (folding, bending)	10
Deformable-Object Manipulation (clay, dough)	10
On/Off Controls (buttons, switches)	10
Sorting & Sequencing	8
Assembly & Threading	10
Environment Interaction (drawers, lids)	7
Cleaning & Organization (tossing, sorting)	10
Multi-step Complex Tasks (subtask chains)	10

This taxonomy covers both single-step and multi-step tasks and includes object classes ranging from rigid solids to deformables, articulated objects, and tools. The benchmark therefore treats manipulation not as a homogeneous setting but as a structured space with qualitatively different affordance profiles and control requirements (Wang et al., 16 Jan 2026).

The task design procedure is described as a systematic pipeline with five components. First, prior task lists, including Agibot, Open X-Embodiment, and $\pi_{0.5}$, were collected, deduplicated, and analyzed via verb-frequency distributions to expose concentration around high-frequency actions (Wang et al., 16 Jan 2026). Second, the benchmark used human–object interaction primitives derived from HAKE, PaStaNet, and OCL to identify a superset of actions covering both common and rare interactions (Wang et al., 16 Jan 2026). Third, Qwen3 was used for optionally guided LLM generation to expand action–object pairs into concrete task descriptions after verb-sense disambiguation and object enumeration (Wang et al., 16 Jan 2026). Fourth, a hybrid filtering stage combined LLM-based executability scoring with human expert review to ensure physical feasibility on current hardware, clear and objective success criteria, and the avoidance of excessive safety or teleoperation complexity (Wang et al., 16 Jan 2026). Fifth, task instantiation added detailed interaction criteria, such as “apply sawing motion” or “hold pressed for 1–2 s,” along with object purchase links and template demonstration videos (Wang et al., 16 Jan 2026).

Methodologically, this design process places GM-100 between a purely data-driven benchmark and a purely hand-authored one. A plausible implication is that the benchmark is attempting to preserve coverage and diversity while retaining operational rigor in task definition.

3. Platforms, trajectories, and data collection protocol

GM-100 includes a trajectory dataset collected on two robotic platforms: Agilex Cobot Magic and Dobot Xtrainer (Wang et al., 16 Jan 2026). Cobot Magic is described as using a Mobile-Aloha-style base with a forward-reaching arm and a head-mounted RGB camera, whereas Xtrainer uses an Aloha-style base with an inward-folding arm and a top-down RGB camera (Wang et al., 16 Jan 2026). Both platforms employ low-cost, compliant grippers and teleoperation interfaces based on Mobile ALOHA (Wang et al., 16 Jan 2026).

The dataset contains more than 13,000 demonstrations distributed across the 100 tasks, with 100 training trajectories and 30 test trajectories per task (Wang et al., 16 Jan 2026). Trajectory lengths vary by task, averaging between 20 and 200 time steps. The object inventory includes approximately 200 unique purchasable items spanning solids, deformables, articulated objects, and tools (Wang et al., 16 Jan 2026). The action space includes gripper open/close, Cartesian position deltas, orientation adjustments, and force-sensitive compliance (Wang et al., 16 Jan 2026).

The collection protocol includes controlled perturbation of initial conditions. Each of the 100 training demonstrations for a task uses randomized initial positions and orientations within a bounded range, and the test set mirrors the same distribution to preserve comparability (Wang et al., 16 Jan 2026). This is significant because it ensures that evaluation does not collapse into replay under a single canonical scene arrangement. Instead, benchmark performance reflects at least limited robustness to variation in scene initialization.

The benchmark is therefore neither purely static nor fully open-ended. It occupies an intermediate regime in which held-out test initializations are distribution-matched to randomized training setups. This suggests that GM-100 is designed primarily to measure generalization within bounded operational variability rather than unconstrained transfer across radically different embodiments or environments.

4. Evaluation metrics and experimental protocol

GM-100 uses three principal classes of evaluation metric: Success Rate (SR), Partial Success Rate (PSR), and action prediction error (Wang et al., 16 Jan 2026). Success Rate is defined as

$$\mathrm{SR} = \frac{N_{\mathrm{success}}}{N_{\mathrm{total}}} \times 100\%.$$

A trajectory is counted as successful if it satisfies the task’s terminal criteria within a fixed time budget (Wang et al., 16 Jan 2026). This makes SR the strictest task-level metric in the benchmark.

For tasks composed of $S$ subtasks, Partial Success Rate is defined as

$$\mathrm{PSR} = \frac{1}{N_{\mathrm{total}}} \sum_{i=1}^{N_{\mathrm{total}}} \frac{s_i}{S} \times 100\%,$$

where $s_i$ denotes the number of subtasks correctly executed in demonstration $i$ (Wang et al., 16 Jan 2026). PSR is intended to capture graded competence on multi-stage tasks for which terminal success may be sparse even when partial progress is meaningful.

Action prediction error is reported using both mean squared error and $\ell_1$ loss over a fixed overlapping prediction window across all models:

$$\mathrm{MSE} = \frac{1}{N}\sum_{t=1}^{N}\|\hat a_t-a_t\|_2^2,$$

$$\mathrm{L1} = \frac{1}{N}\sum_{t=1}^{N}\|\hat a_t-a_t\|_1.$$

The train/test split is fixed at 100 demonstrations for training and 30 held-out initializations for testing per task, and each model executes the 30 test initializations once per task (Wang et al., 16 Jan 2026). Results are aggregated by reporting average SR, PSR, and action error across all 100 tasks, with task-level breakdowns and heatmaps also provided (Wang et al., 16 Jan 2026).

The coexistence of SR and PSR is notable. A frequent misconception in robotics benchmarking is that binary success alone is sufficient for comparing policies across heterogeneous tasks. GM-100 rejects that premise by explicitly incorporating partial-completion accounting for subtask-structured manipulation. This is especially relevant for long-horizon tasks, where binary success can be too sparse to expose meaningful differences among policies.

5. Baseline models and empirical findings

The benchmark evaluates three baseline models: DP, $\pi_{0}$, and $$\mathrm{SR} = \frac{N_{\mathrm{success}}}{N_{\mathrm{total}}} \times 100\%.$$0 (Wang et al., 16 Jan 2026). DP is Diffusion Policy based on Chi et al., finetuned from scratch for each task using the LeRobot framework (Wang et al., 16 Jan 2026). $$\mathrm{SR} = \frac{N_{\mathrm{success}}}{N_{\mathrm{total}}} \times 100\%.$$1 is a Vision-Language-Action flow model finetuned on 100 trajectories per task using OpenPi, and $$\mathrm{SR} = \frac{N_{\mathrm{success}}}{N_{\mathrm{total}}} \times 100\%.$$2 is described as the successor to $$\mathrm{SR} = \frac{N_{\mathrm{success}}}{N_{\mathrm{total}}} \times 100\%.$$3 with architectural refinements for open-world generalization (Wang et al., 16 Jan 2026). The paper additionally notes that GR00T and other VLA models are in progress and will be released on the public leaderboard (Wang et al., 16 Jan 2026).

On Dobot Xtrainer, for Tasks 0001–0010, the reported averages are PSR of 7.0 for DP, 32.1 for $$\mathrm{SR} = \frac{N_{\mathrm{success}}}{N_{\mathrm{total}}} \times 100\%.$$4, and 53.9 for $$\mathrm{SR} = \frac{N_{\mathrm{success}}}{N_{\mathrm{total}}} \times 100\%.$$5, alongside SR of 1.6 for DP, 4.4 for $$\mathrm{SR} = \frac{N_{\mathrm{success}}}{N_{\mathrm{total}}} \times 100\%.$$6, and 24.9 for $$\mathrm{SR} = \frac{N_{\mathrm{success}}}{N_{\mathrm{total}}} \times 100\%.$$7 (Wang et al., 16 Jan 2026). Individual tasks show similarly wide separation. For example, Task 0001 reports PSR values of 2.5, 45.8, and 72.5 for DP, $$\mathrm{SR} = \frac{N_{\mathrm{success}}}{N_{\mathrm{total}}} \times 100\%.$$8, and $$\mathrm{SR} = \frac{N_{\mathrm{success}}}{N_{\mathrm{total}}} \times 100\%.$$9 respectively, with corresponding SR values of 0.0, 13.3, and 36.7 (Wang et al., 16 Jan 2026). Task 0010 reports PSR values of 0.0, 10.0, and 36.7 and SR values of 0.0, 10.0, and 36.7 for the same models (Wang et al., 16 Jan 2026).

The benchmark reports several performance trends. $S$0 consistently outperforms both DP and $S$1, often by a factor of 2–3$S$2 in SR and PSR (Wang et al., 16 Jan 2026). DP struggles on multi-step tasks and fine-force primitives, with average SR below 2% (Wang et al., 16 Jan 2026). $S$3 demonstrates moderate competence on single-step tasks but underperforms on long-tail interactions such as threading and folding (Wang et al., 16 Jan 2026). Heatmaps on Cobot Magic show the same qualitative pattern, with tool-use and deformable-object tasks remaining especially difficult; even $S$4 remains below 30% PSR on these categories (Wang et al., 16 Jan 2026).

These findings position GM-100 as a discriminative benchmark rather than a saturated one. The reported performance levels are sufficiently low and sufficiently separated to expose meaningful differences among current VLA models.

6. Benchmark difficulty, limitations, and proposed extensions

GM-100 identifies three principal challenge areas. First, long-tail and multi-step tasks remain difficult, particularly tasks involving precise force control, such as folding cardboard and stamping, or sequential primitives, such as threading hawthorns; these yield very low SR, below 5% for DP and $S$5 (Wang et al., 16 Jan 2026). Second, the benchmark observes under-sampled modes: even with 100 demonstrations per task, some subgoals remain rare, which induces overfitting to dominant strategies (Wang et al., 16 Jan 2026). Third, hardware constraints—including compliance, limited end-effector dexterity, and camera-induced occlusions—create noise and failure modes that do not appear in synthetic or open-loop settings (Wang et al., 16 Jan 2026).

These limitations are important for interpreting benchmark scores. Poor performance cannot automatically be attributed solely to policy architecture, because embodiment constraints and sensing geometry materially shape the operational difficulty of the tasks. Conversely, the benchmark’s insistence on real hardware prevents the simplification of manipulation evaluation into purely kinematic trajectory matching.

The paper outlines several future extensions: integrating tactile sensing, force-torque sensing, and depth cameras; constructing more complex task compositions by chaining GM-100 tasks into longer pipelines under the label GM-X; developing cross-platform policy-distillation pipelines from Cobot Magic to Xtrainer and beyond; building a photorealistic standardized simulation proxy; and extending evaluation beyond SR and PSR to include completion time $S$6, path length $S$7, and safety-violation rates (Wang et al., 16 Jan 2026).

Taken together, these proposed directions indicate that GM-100 is intended as a foundation rather than a closed benchmark. A plausible implication is that its long-term role is to anchor a family of interoperable evaluations spanning real-world execution, simulation, richer sensing, and cross-platform transfer.

7. Position within embodied AI evaluation

GM-100 is presented in direct response to the proliferation of robot learning and imitation learning datasets whose task designs, according to the paper, often lack systematic principles (Wang et al., 16 Jan 2026). Its distinguishing characteristic is not simply the number 100, but the combination of systematic corpus analysis, primitive-based expansion, hybrid feasibility filtering, real-world trajectory collection on multiple platforms, and evaluation procedures that separate terminal success from partial completion (Wang et al., 16 Jan 2026).

Within embodied AI evaluation, the benchmark therefore serves two roles. First, it is an empirical testbed for current VLA and imitation-learning systems under diverse manipulation conditions. Second, it is a proposal for task-design methodology: the benchmark treats task coverage, long-tail behavior inclusion, and explicit interaction criteria as first-class design variables rather than incidental byproducts of dataset collection (Wang et al., 16 Jan 2026).

A common misunderstanding would be to reduce GM-100 to a harder pick-and-place suite. The benchmark’s taxonomy and reported results do not support that interpretation. Tool use, deformable-object manipulation, structural manipulation, threading, environment interaction, and multi-step complex tasks are integral components of the benchmark and contribute directly to its difficulty profile (Wang et al., 16 Jan 2026). For researchers concerned with benchmark validity, this broader distribution of task types is the central reason GM-100 is positioned as a more diagnostic evaluation of embodied competence.