---
title: 'MotionBenchMaker: Benchmarking for Robotics'
url: https://www.emergentmind.com/topics/motionbenchmaker
type: topic
---

# MotionBenchMaker: Benchmarking for Robotics

MotionBenchMaker is an open-source tool for generating benchmarking datasets for realistic robot manipulation problems and for benchmarking motion planning algorithms on those datasets. Introduced by Chamzas et al. in 2021, it was motivated by the observation that planner evaluation is often performed on ad-hoc problem sets that are time-consuming to construct, prone to bias, and difficult to compare directly against prior work. MotionBenchMaker therefore couples procedural dataset generation with benchmarking workflows, and it ships with 40 prefabricated datasets spanning 5 robot platforms and 8 environments as a common evaluation substrate for motion planning research [2112.06402].

## 1. Conceptual scope and benchmarking role

MotionBenchMaker was designed to address a methodological problem in robot motion planning: new planners are continuously proposed, but their empirical evaluation is frequently fragmented across custom scenes, custom query sets, and incompatible reporting pipelines. The 2021 system is explicitly described as an extensible, easy-to-use tool that allows users both to generate datasets and to benchmark them by comparing motion planning algorithms [2112.06402]. In that sense, MotionBenchMaker is not merely a static corpus of start–goal pairs, and it is not itself a planner; it is a toolchain for procedural benchmark construction and comparative evaluation.

Its benchmarking role also situates it within a broader robotics evaluation ecosystem. MotionBenchMaker can emit CSV or JSON logs per problem, a SQL database for PlannerArena compatibility, and a plain-text `.log` summarizing per-planner, per-dataset metrics [2112.06402]. Earlier work on benchmarking infrastructure for motion planning defined an OMPL-centered framework with extensible plain-text logs, SQLite3 ingestion, and interactive visualization through Planner Arena [1412.6673]. MotionBenchMaker’s explicit SQL/PlannerArena compatibility indicates operational continuity with that infrastructure, while adding procedural generation of manipulation datasets rather than only execution and visualization of planner runs.

A recurrent source of confusion in the literature is the phrase “the MotionBenchMaker dataset.” In many later papers, that phrase refers specifically to the released benchmark suite associated with the 2021 tool, rather than to the generator alone. This usage is consistent with the original release, which combines generation modules, manifest files, robot and scene descriptions, and prefabricated datasets into a reusable benchmarking package [2112.06402].

## 2. Internal architecture and data-generation pipeline

MotionBenchMaker is organized as four core C++ modules: `SceneSampler`, `OctomapGenerator` (optional), `ProblemGenerator`, and `Setup` [2112.06402]. The data flow begins with three user-provided inputs: a nominal scene description, a robot description, and a set of manipulation-query templates. The nominal scene may be specified using meshes or primitives, or via a URDF; the robot is provided through a URDF together with any sensor offsets; and the query templates encode object-centric grasp or place offsets.

`SceneSampler` perturbs the nominal scene according to a YAML configuration of sampling parameters and produces sampled scenes $\{S_i\}$. For rigid objects, the pose of object $j$ is sampled either from a Gaussian
$$
p_j(x_j)=\mathcal{N}(\mu_j,\Sigma_j)
$$
or from a uniform box
$$
x_j \sim \mathrm{Uniform}([a_j,b_j]).
$$
For articulated objects modeled as kinematic chains, such as a drawer, MotionBenchMaker performs URDF sampling in joint space,
$$
q_{\text{object}} \sim \mathrm{Uniform}(q_{\min},q_{\max}),
$$
while rejecting self-colliding configurations [2112.06402].

`OctomapGenerator`, when enabled, simulates an RGB-D camera for each sampled scene, producing point clouds $P_i$ and octrees $O_i$. This permits parallel benchmark releases in both geometric and sensed representations. `ProblemGenerator` then combines the scene, the robot, and the query templates, solving collision-aware inverse kinematics to obtain start/goal joint pairs $(q_{\text{start}}, q_{\text{goal}})$. The stored queries are accepted only if valid IK solutions exist in free space. The corresponding optimization problem is written in the 2021 description as
$$
q_{\text{start}}^{(k)}=\arg\min_q \|F(q)-S_{\text{start}}^{(k)}\|^2
$$
subject to $q \in [q_{\min},q_{\max}]$ and $C_{\text{free}}(q)$, with an analogous construction for the goal template [2112.06402].

At the dataset level, the procedural generation algorithm constructs $N$ scenes and then selects $M$ planning problems per scene. Scene diversity arises from global versus local pose perturbations, multi-query templates per object, optional robot-base variation, and the combination of rigid and articulated sampling. The `Setup` module finally packages scenes, point clouds or octomaps, requests, and a manifest file `conf.yaml` into a self-contained dataset [2112.06402].

## 3. Robot, environment, and file abstractions

MotionBenchMaker’s environment model supports both flat collections of collision objects and articulated URDF scenes. Collision geometry is taken directly from meshes in the scene or from the `collision` tags in URDF files. Robots are loaded from standard URDFs, and their kinematic solvers are provided via KDL or a user-specified IK plugin. Collision checking is performed through FCL, and the `ProblemGenerator` enforces that sampled start and goal states lie in $C_{\text{free}}$ [2112.06402].

The released benchmark suite contains 40 datasets defined as 5 robot platforms crossed with 8 environments, with each dataset containing 100 problems [2112.06402]. The five robot platforms are Fetch (7–8 DoF), Franka Panda (7 DoF), UR5 (6 DoF), Baxter (14 DoF, dual-arm), and ShadowHand on KUKA arm (31 DoF). The eight environments are shelf picking with cylindrical objects, box picking from open box, table-top grasping on a cluttered table, cabinet-drawer opening, narrow-passage “tunnel,” multi-object rearrangement on table, bimanual shelf access, and dexterous hand manipulations. Because each environment has both a geometric model and a sensed octomap version, the release yields 80 distinct datasets if geometric and sensed variants are counted separately [2112.06402].

The file structure is correspondingly explicit. A typical dataset directory contains a manifest `conf.yaml`, per-scene files under `scenes/`, optional sensed data under `pointclouds/` and `octomaps/`, request files with start/goal joint vectors under `requests/`, and robot descriptions under `robots/`. After benchmarking, the same setup can produce `results.log`, `results.csv`, or `results.db` [2112.06402]. This explicit packaging supports reproducibility at the level of scenes, robot models, planning requests, and planner outputs.

## 4. Benchmark execution, metrics, and statistical reporting

Benchmark execution is organized around repeated planner runs over the generated requests. In the 2021 API example, a `Setup` object loads a dataset from `conf.yaml`, retrieves the robot, creates a planner such as `BiEST`, iterates over sampled scenes and requests, accumulates queries in an `Experiment`, and dumps an OMPL-compatible result log through `OMPLPlanDataSetOutputter` [2112.06402]. The same paper also describes a CLI-style workflow in which a dataset is first generated and then benchmarked with one or more planners.

MotionBenchMaker’s reported metrics are standard but explicitly formalized. Success rate is
$$
S_r=\frac{N_{\text{success}}}{N_{\text{trials}}}.
$$
If $T_{i,t}$ denotes the planning time for trial $t$ on problem $i$, with timeout $T_{\max}$, then the average time per problem is
$$
\bar{T}_i=\frac{1}{N_{\text{trials}}}\sum_{t=1}^{N_{\text{trials}}} T_{i,t},
$$
and the aggregate over all problems is
$$
\bar{T}=\frac{1}{N_{\text{problems}}}\sum_{i=1}^{N_{\text{problems}}}\bar{T}_i.
$$
Joint-space path length is defined as
$$
L(\tau)=\sum_{k=0}^{K-1}\|q_{k+1}-q_k\|_2,
$$
while clearance cost is
$$
C(\tau)=\min_{k,p\in \text{robot}} d\bigl(p(q_k),\text{Environment}\bigr).
$$
For optimizing planners, MotionBenchMaker tracks the best cost $J^*(t)$ found by time $t$, and it aggregates results with medians, means, and 95% confidence intervals [2112.06402].

These outputs align naturally with the earlier OMPL benchmarking infrastructure, which defined run-level and progress-level logging, a SQLite3 schema, and visualization through overall performance plots, progress plots, and regression plots in Planner Arena [1412.6673]. MotionBenchMaker’s SQL export for PlannerArena compatibility therefore provides not only raw planner outcomes but also immediate compatibility with established statistical visualizations such as ECDFs, box plots, and convergence curves.

## 5. Empirical findings on evaluation methodology

A central empirical claim of MotionBenchMaker is methodological rather than purely algorithmic: small or poorly chosen benchmark samples can produce unstable or incorrect conclusions about planner quality. The paper’s “Wrong Hypothesis” demonstration used UR5 shelf-picking to compare BiEST with RRT at two `range` settings across 100 problems and 20 seeds. Two different adversarial orderings of the same 100 problems yielded opposite conclusions when only 5–10 samples were used, whereas the full 100-problem aggregate correctly revealed that the RRT variants outperformed BiEST [2112.06402]. The implication is that planner rankings can be highly sensitive to benchmark subsampling.

The same paper reports a comparison among optimizing planners on Panda box picking, using normalized joint-path length and a 180 s timeout. Single-instance runs produced contradictory best-planner rankings among RRT*, BIT*, and AIT*, while aggregation over 100 problems stabilized the conclusion [2112.06402]. This finding reinforces the argument that planner evaluation should be dataset-based rather than anecdotal.

In a broader planner challenge covering 12 of the 40 datasets, with 4 environments and 3 robots, the evaluated planners were BiEST, RRT, and BKPIECE, each swept over `range` values from 0 to 7 in steps of 0.25. The reported findings were that no single planner dominates all tasks, the sensed octomap versions incur only a modest overhead while remaining challenging, and each planner has a distinct “sweet-spot” range parameter per robot/environment [2112.06402]. MotionBenchMaker’s procedural construction was therefore presented not only as a convenience tool but as an instrument for reducing evaluation bias and exposing parameter sensitivity.

The 2021 release also codified several operational recommendations: run a small feasibility pass with a high-timeout planner to filter impossible queries, tune sampling variances so that problems span “easy,” “medium,” and “hard,” and use at least 50 problems per dataset to avoid statistical anomalies [2112.06402]. These recommendations formalize benchmark curation as part of the experimental method.

## 6. Later reuse, extensions, and neighboring systems

Subsequent work has used the MotionBenchMaker dataset as a standardized stress test for new planning algorithms. The pRRTC paper evaluates GPU-parallel RRT-Connect on MotionBenchMaker with three manipulators—Panda (7 DoF), Fetch (8 DoF), and Baxter (14 DoF)—over reaching tasks in seven environments drawn from table-top, bookshelf, bin, drawer, and cage geometries, with collision-checking resolutions up to 512 subdivisions per unit step. Reported outcomes include a Fetch average solve time of 4.23 ms versus 13.85 ms for VAMP-RRTC, a 38× smaller solution-time variance on Fetch, path-cost improvements from 21.29 to 13.95 on Fetch and from 17.49 to 11.22 on Baxter, and up to 6× speedup at 512 collision-check resolution on Fetch-Cage, while maintaining 100% solve rates across all robots [2503.06757].

AORRTC uses benchmark problems from the MotionBenchMaker repository for Panda and Fetch, with 100 pre-defined start/goal pairs in seven environments: `bookshelf_small`, `bookshelf_tall`, `bookshelf_thin`, `table_pick`, `table_under_pick`, `box`, and `cage`. In the VAMP implementation, AORRTC solves 100% of all seven Panda environments, with median initial times from 0.3 ms up to 1.0 ms, and it solves 100% of the more-constrained Fetch bookshelf and table tasks together with 98% of Fetch cage. The same study reports that VAMP AORRTC is 3–5× faster than OMPL AORRTC on identical problems, attributing much of the difference to vectorized collision checking [2505.10542].

MotionBenchMaker has also been used outside classical planner-comparison studies. The VaPr paper evaluates variable-precision tensors for motion generation on 800 Franka Panda planning problems across 8 MotionBenchMaker environments—`bookshelf_small_panda`, `bookshelf_tall_panda`, `bookshelf_thin_panda`, `box_panda`, `box_panda_flipped`, `cage_panda`, `table_pick_panda`, and `table_under_pick_panda`. It reports that several large tensors can be reduced to 4–5 bits, that memory footprint is reduced by 3.5×–4.4× on average for the selected tensors, and that overall motion-generation speedups of approximately 6.3% on Jetson Orin and 6.3% on RTX 2080 Ti rise to 9.9% and 17.7% with converter hardware, without loss of the baseline 100% success rate across all 800 problems [2310.07854].

A distinct but conceptually related development is M3BenchMaker, introduced in the M3Bench benchmark for whole-body motion generation in mobile manipulation. M3BenchMaker is described as an automatic data generation tool that produces whole-body motion trajectories from high-level task instructions using basic scene and robot information. Its pipeline comprises a `Task Builder`, `Conditional Scene Sampler`, `Goal Configuration Generator`, and `VKC Problem Generator`, and it was used to produce 30,000 valid pick/place demonstrations spanning 119 household scenes and 32 object types [2410.06678]. Although M3BenchMaker is a separate system rather than MotionBenchMaker itself, a plausible implication is that the procedural benchmark-generation paradigm inaugurated for manipulator planning has been extended toward embodied whole-body mobile manipulation in 3D scenes.

Source: https://www.emergentmind.com/topics/motionbenchmaker