MotionBenchMaker: Benchmarking for Robotics

• MotionBenchMaker is an open-source tool for generating realistic robot manipulation benchmarking datasets and evaluating motion planning algorithms.
• It couples procedural dataset generation with standardized evaluation workflows to ensure reproducibility and direct comparisons across multiple robot platforms and environments.
• The system integrates modules for scene sampling, query generation, and statistical reporting, reducing evaluation bias and enabling robust planner comparisons.

MotionBenchMaker is an open-source tool for generating benchmarking datasets for realistic robot manipulation problems and for benchmarking motion planning algorithms on those datasets. Introduced by Chamzas et al. in 2021, it was motivated by the observation that planner evaluation is often performed on ad-hoc problem sets that are time-consuming to construct, prone to bias, and difficult to compare directly against prior work. MotionBenchMaker therefore couples procedural dataset generation with benchmarking workflows, and it ships with 40 prefabricated datasets spanning 5 robot platforms and 8 environments as a common evaluation substrate for motion planning research (Chamzas et al., 2021).

1. Conceptual scope and benchmarking role

MotionBenchMaker was designed to address a methodological problem in robot motion planning: new planners are continuously proposed, but their empirical evaluation is frequently fragmented across custom scenes, custom query sets, and incompatible reporting pipelines. The 2021 system is explicitly described as an extensible, easy-to-use tool that allows users both to generate datasets and to benchmark them by comparing motion planning algorithms (Chamzas et al., 2021). In that sense, MotionBenchMaker is not merely a static corpus of start–goal pairs, and it is not itself a planner; it is a toolchain for procedural benchmark construction and comparative evaluation.

Its benchmarking role also situates it within a broader robotics evaluation ecosystem. MotionBenchMaker can emit CSV or JSON logs per problem, a SQL database for PlannerArena compatibility, and a plain-text .log summarizing per-planner, per-dataset metrics (Chamzas et al., 2021). Earlier work on benchmarking infrastructure for motion planning defined an OMPL-centered framework with extensible plain-text logs, SQLite3 ingestion, and interactive visualization through Planner Arena (Moll et al., 2014). MotionBenchMaker’s explicit SQL/PlannerArena compatibility indicates operational continuity with that infrastructure, while adding procedural generation of manipulation datasets rather than only execution and visualization of planner runs.

A recurrent source of confusion in the literature is the phrase “the MotionBenchMaker dataset.” In many later papers, that phrase refers specifically to the released benchmark suite associated with the 2021 tool, rather than to the generator alone. This usage is consistent with the original release, which combines generation modules, manifest files, robot and scene descriptions, and prefabricated datasets into a reusable benchmarking package (Chamzas et al., 2021).

2. Internal architecture and data-generation pipeline

MotionBenchMaker is organized as four core C++ modules: SceneSampler, OctomapGenerator (optional), ProblemGenerator, and Setup (Chamzas et al., 2021). The data flow begins with three user-provided inputs: a nominal scene description, a robot description, and a set of manipulation-query templates. The nominal scene may be specified using meshes or primitives, or via a URDF; the robot is provided through a URDF together with any sensor offsets; and the query templates encode object-centric grasp or place offsets.

SceneSampler perturbs the nominal scene according to a YAML configuration of sampling parameters and produces sampled scenes $\{S_i\}$. For rigid objects, the pose of object $j$ is sampled either from a Gaussian

$p_j(x_j)=\mathcal{N}(\mu_j,\Sigma_j)$

or from a uniform box

$x_j \sim \mathrm{Uniform}([a_j,b_j]).$

For articulated objects modeled as kinematic chains, such as a drawer, MotionBenchMaker performs URDF sampling in joint space,

$$q_{\text{object}} \sim \mathrm{Uniform}(q_{\min},q_{\max}),$$

while rejecting self-colliding configurations (Chamzas et al., 2021).

OctomapGenerator, when enabled, simulates an RGB-D camera for each sampled scene, producing point clouds $P_i$ and octrees $O_i$. This permits parallel benchmark releases in both geometric and sensed representations. ProblemGenerator then combines the scene, the robot, and the query templates, solving collision-aware inverse kinematics to obtain start/goal joint pairs $(q_{\text{start}}, q_{\text{goal}})$. The stored queries are accepted only if valid IK solutions exist in free space. The corresponding optimization problem is written in the 2021 description as

$$q_{\text{start}}^{(k)}=\arg\min_q \|F(q)-S_{\text{start}}^{(k)}\|^2$$

subject to $q \in [q_{\min},q_{\max}]$ and $j$0, with an analogous construction for the goal template (Chamzas et al., 2021).

At the dataset level, the procedural generation algorithm constructs $j$1 scenes and then selects $j$2 planning problems per scene. Scene diversity arises from global versus local pose perturbations, multi-query templates per object, optional robot-base variation, and the combination of rigid and articulated sampling. The Setup module finally packages scenes, point clouds or octomaps, requests, and a manifest file conf.yaml into a self-contained dataset (Chamzas et al., 2021).

3. Robot, environment, and file abstractions

MotionBenchMaker’s environment model supports both flat collections of collision objects and articulated URDF scenes. Collision geometry is taken directly from meshes in the scene or from the collision tags in URDF files. Robots are loaded from standard URDFs, and their kinematic solvers are provided via KDL or a user-specified IK plugin. Collision checking is performed through FCL, and the ProblemGenerator enforces that sampled start and goal states lie in $j$3 (Chamzas et al., 2021).

The released benchmark suite contains 40 datasets defined as 5 robot platforms crossed with 8 environments, with each dataset containing 100 problems (Chamzas et al., 2021). The five robot platforms are Fetch (7–8 DoF), Franka Panda (7 DoF), UR5 (6 DoF), Baxter (14 DoF, dual-arm), and ShadowHand on KUKA arm (31 DoF). The eight environments are shelf picking with cylindrical objects, box picking from open box, table-top grasping on a cluttered table, cabinet-drawer opening, narrow-passage “tunnel,” multi-object rearrangement on table, bimanual shelf access, and dexterous hand manipulations. Because each environment has both a geometric model and a sensed octomap version, the release yields 80 distinct datasets if geometric and sensed variants are counted separately (Chamzas et al., 2021).

The file structure is correspondingly explicit. A typical dataset directory contains a manifest conf.yaml, per-scene files under scenes/, optional sensed data under pointclouds/ and octomaps/, request files with start/goal joint vectors under requests/, and robot descriptions under robots/. After benchmarking, the same setup can produce results.log, results.csv, or results.db (Chamzas et al., 2021). This explicit packaging supports reproducibility at the level of scenes, robot models, planning requests, and planner outputs.

4. Benchmark execution, metrics, and statistical reporting

Benchmark execution is organized around repeated planner runs over the generated requests. In the 2021 API example, a Setup object loads a dataset from conf.yaml, retrieves the robot, creates a planner such as BiEST, iterates over sampled scenes and requests, accumulates queries in an Experiment, and dumps an OMPL-compatible result log through OMPLPlanDataSetOutputter (Chamzas et al., 2021). The same paper also describes a CLI-style workflow in which a dataset is first generated and then benchmarked with one or more planners.

MotionBenchMaker’s reported metrics are standard but explicitly formalized. Success rate is

$j$4

If $j$5 denotes the planning time for trial $j$6 on problem $j$7, with timeout $j$8, then the average time per problem is

$j$9

and the aggregate over all problems is

$p_j(x_j)=\mathcal{N}(\mu_j,\Sigma_j)$0

Joint-space path length is defined as

$p_j(x_j)=\mathcal{N}(\mu_j,\Sigma_j)$1

while clearance cost is

$p_j(x_j)=\mathcal{N}(\mu_j,\Sigma_j)$2

For optimizing planners, MotionBenchMaker tracks the best cost $p_j(x_j)=\mathcal{N}(\mu_j,\Sigma_j)$3 found by time $p_j(x_j)=\mathcal{N}(\mu_j,\Sigma_j)$4, and it aggregates results with medians, means, and 95% confidence intervals (Chamzas et al., 2021).

These outputs align naturally with the earlier OMPL benchmarking infrastructure, which defined run-level and progress-level logging, a SQLite3 schema, and visualization through overall performance plots, progress plots, and regression plots in Planner Arena (Moll et al., 2014). MotionBenchMaker’s SQL export for PlannerArena compatibility therefore provides not only raw planner outcomes but also immediate compatibility with established statistical visualizations such as ECDFs, box plots, and convergence curves.

5. Empirical findings on evaluation methodology

A central empirical claim of MotionBenchMaker is methodological rather than purely algorithmic: small or poorly chosen benchmark samples can produce unstable or incorrect conclusions about planner quality. The paper’s “Wrong Hypothesis” demonstration used UR5 shelf-picking to compare BiEST with RRT at two range settings across 100 problems and 20 seeds. Two different adversarial orderings of the same 100 problems yielded opposite conclusions when only 5–10 samples were used, whereas the full 100-problem aggregate correctly revealed that the RRT variants outperformed BiEST (Chamzas et al., 2021). The implication is that planner rankings can be highly sensitive to benchmark subsampling.

The same paper reports a comparison among optimizing planners on Panda box picking, using normalized joint-path length and a 180 s timeout. Single-instance runs produced contradictory best-planner rankings among RRT*, BIT*, and AIT*, while aggregation over 100 problems stabilized the conclusion (Chamzas et al., 2021). This finding reinforces the argument that planner evaluation should be dataset-based rather than anecdotal.

In a broader planner challenge covering 12 of the 40 datasets, with 4 environments and 3 robots, the evaluated planners were BiEST, RRT, and BKPIECE, each swept over range values from 0 to 7 in steps of 0.25. The reported findings were that no single planner dominates all tasks, the sensed octomap versions incur only a modest overhead while remaining challenging, and each planner has a distinct “sweet-spot” range parameter per robot/environment (Chamzas et al., 2021). MotionBenchMaker’s procedural construction was therefore presented not only as a convenience tool but as an instrument for reducing evaluation bias and exposing parameter sensitivity.

The 2021 release also codified several operational recommendations: run a small feasibility pass with a high-timeout planner to filter impossible queries, tune sampling variances so that problems span “easy,” “medium,” and “hard,” and use at least 50 problems per dataset to avoid statistical anomalies (Chamzas et al., 2021). These recommendations formalize benchmark curation as part of the experimental method.

6. Later reuse, extensions, and neighboring systems

Subsequent work has used the MotionBenchMaker dataset as a standardized stress test for new planning algorithms. The pRRTC paper evaluates GPU-parallel RRT-Connect on MotionBenchMaker with three manipulators—Panda (7 DoF), Fetch (8 DoF), and Baxter (14 DoF)—over reaching tasks in seven environments drawn from table-top, bookshelf, bin, drawer, and cage geometries, with collision-checking resolutions up to 512 subdivisions per unit step. Reported outcomes include a Fetch average solve time of 4.23 ms versus 13.85 ms for VAMP-RRTC, a 38× smaller solution-time variance on Fetch, path-cost improvements from 21.29 to 13.95 on Fetch and from 17.49 to 11.22 on Baxter, and up to 6× speedup at 512 collision-check resolution on Fetch-Cage, while maintaining 100% solve rates across all robots (Huang et al., 9 Mar 2025).

AORRTC uses benchmark problems from the MotionBenchMaker repository for Panda and Fetch, with 100 pre-defined start/goal pairs in seven environments: bookshelf_small, bookshelf_tall, bookshelf_thin, table_pick, table_under_pick, box, and cage. In the VAMP implementation, AORRTC solves 100% of all seven Panda environments, with median initial times from 0.3 ms up to 1.0 ms, and it solves 100% of the more-constrained Fetch bookshelf and table tasks together with 98% of Fetch cage. The same study reports that VAMP AORRTC is 3–5× faster than OMPL AORRTC on identical problems, attributing much of the difference to vectorized collision checking (Wilson et al., 15 May 2025).

MotionBenchMaker has also been used outside classical planner-comparison studies. The VaPr paper evaluates variable-precision tensors for motion generation on 800 Franka Panda planning problems across 8 MotionBenchMaker environments—bookshelf_small_panda, bookshelf_tall_panda, bookshelf_thin_panda, box_panda, box_panda_flipped, cage_panda, table_pick_panda, and table_under_pick_panda. It reports that several large tensors can be reduced to 4–5 bits, that memory footprint is reduced by 3.5×–4.4× on average for the selected tensors, and that overall motion-generation speedups of approximately 6.3% on Jetson Orin and 6.3% on RTX 2080 Ti rise to 9.9% and 17.7% with converter hardware, without loss of the baseline 100% success rate across all 800 problems (Hsiao et al., 2023).

A distinct but conceptually related development is M3BenchMaker, introduced in the M3Bench benchmark for whole-body motion generation in mobile manipulation. M3BenchMaker is described as an automatic data generation tool that produces whole-body motion trajectories from high-level task instructions using basic scene and robot information. Its pipeline comprises a Task Builder, Conditional Scene Sampler, Goal Configuration Generator, and VKC Problem Generator, and it was used to produce 30,000 valid pick/place demonstrations spanning 119 household scenes and 32 object types (Zhang et al., 2024). Although M3BenchMaker is a separate system rather than MotionBenchMaker itself, a plausible implication is that the procedural benchmark-generation paradigm inaugurated for manipulator planning has been extended toward embodied whole-body mobile manipulation in 3D scenes.