ActiveManip-Bench Benchmark Suite

• ActiveManip-Bench is a platform-agnostic benchmark suite for quantifying planning and control performance in active in-hand manipulation using standardized YCB object meshes and pose/contact metrics.
• It defines structured in-hand tasks across three difficulty levels by specifying initial and target hand poses along with permissible contact regions for reproducible comparisons.
• The benchmark provides detailed error metrics—including position, orientation, and contact errors—and supports evaluation on both simulation and real robotic systems.

ActiveManip-Bench is a platform-agnostic benchmark suite for quantifying the planning and control performance of active in-hand manipulation algorithms. It leverages a standard set of Yale–CMU–Berkeley (YCB) object meshes, a structured taxonomy of in-hand manipulation tasks, and unified pose/contact-based metrics to facilitate rigorous, reproducible comparison of robotic manipulation systems across diverse hardware and software platforms (Cruciani et al., 2020).

1. Objectives and Scope

ActiveManip-Bench is designed to assess a robotic system’s ability to change the configuration of an object held in-hand—excluding re-grasps on external supports—and to measure both planning and execution performance in both real and simulated environments. The benchmark is hardware-agnostic: it accommodates a broad spectrum of end-effectors, including parallel grippers, multi-finger hands, and multi-arm “virtual” hands. The only requirements are mesh models of the YCB object, mesh geometry of hand links, and a perception pipeline for estimation of object and hand poses. This minimal set of dependencies ensures that the core contributions—task definition and error measurement—remain decoupled from specific robot architectures and sensing modalities.

2. Task Taxonomy and Definition

ActiveManip-Bench defines each in-hand manipulation task through:

• $C_i$, $C_d$: Initial and desired fingertip contact regions, specified as submeshes of the YCB object mesh.
• $H_i$, $H_d$: Initial and desired hand-to-object transforms, i.e., the object’s pose with respect to the hand (elements of SE(3)).

Contact regions $C_i$ and $C_d$ are drawn per object class (e.g., mug, spatula) and enumerate permissible finger-object contacts. The robot starts at pose $H_i$, with fingertip contacts $P_i \subseteq C_i$, and must transition to $H_d$, with contacts $P_r \subseteq C_d$.

Tasks are organized in three levels of difficulty:

• Level I: Only $C_d$0 (desired hand pose) is specified; contacts may lie anywhere.
• Level II: Only $C_d$1 (desired contact region) is specified; hand pose may vary.
• Level III: Both $C_d$2 and $C_d$3 are specified together.

Procedural setup involves: (1) Initializing the object in the hand at a measured pose $C_d$4 (discarding trials if position/orientation errors from $C_d$5 exceed 10%), (2) executing the manipulation method, (3) logging outcome state ($C_d$6, $C_d$7), planning and execution times.

3. Evaluation Metrics

ActiveManip-Bench provides task- and system-level metrics decomposing pose and contact errors:

A. Hand Pose Error

• Position error (absolute): $C_d$8
• Position error (percent): $C_d$9
• Orientation error (percent): $H_i$0

Here, $H_i$1 is the position, $H_i$2 is a unit quaternion.

B. Contact Region Error

• All final fingertip contacts $H_i$3 must lie within $H_i$4.
• Maximum error among all fingers is measured as the farthest distance from reached contacts to the desired region:
  • Euclidean: $H_i$5
  • Geodesic: $H_i$6 is length of shortest path on the object surface.
• Algorithm 1 (detailed in the benchmark) computes this maximum using closest mesh vertices and returns $H_i$7.

The mesh granularity is characterized by $H_i$8, the minimum inter-vertex distance.

C. Aggregation and Reporting

• Each task is repeated five times; for each metric, the median, interquartile range, and outliers are reported (boxplot style). The system-level score is the average (or median) over all tasks at a given difficulty level. Failure rate (percentage of trials with object drop or timeout) is reported separately.

4. Software and Data Infrastructure

All objects, task definitions, mesh files, and error evaluators are hosted publicly at https://robot-learning.cs.utah.edu/project/benchmarking_in_hand_manipulation. The suite includes:

• YCB object meshes and corresponding $H_i$9, $H_d$0 definitions.
• Example $H_d$1, $H_d$2 configurations.
• JSON/YAML formats for task specification, SE(3) transforms, and optional contact sets.
• Reference Python/C++ scripts for error computations and results visualization.

Workflow:

1. Select a YCB object and load its mesh.
2. Adapt $H_d$3, $H_d$4 to the robot by applying the provided relative transform $H_d$5.
3. Use a suitable perception pipeline to estimate $H_d$6 and $H_d$7 (e.g., using Vicon, markers, depth cameras).
4. Run the manipulation planner/controller; record $H_d$8, $H_d$9, timings.
5. Evaluate using provided scripts, returning position/orientation/contact errors, mesh resolution, success/failure, and timings.

5. Example Benchmark Results

Level I (Allegro Hand, 10 YCB Objects)

Five algorithmic variants were compared, with 500 trials per method, yielding:

Method	$C_i$0 (cm)	$C_i$1	$C_i$2	Failure Rate (%)	Median Planning Time (s)
Point-contact	1.69	36.8	9–10	5	14.8
IK-rigid	1.64	30.9	9–10	9	4.4
Relaxed-position	1.54	29.2	9–10	7	0.5
Relaxed-pos-orient	1.54	28.7	9–10	0	0.3
Relaxed-rigidity	1.32	—	—	0	0.5

The relaxed-rigidity variant achieved the lowest error and zero drops, with minimal planning times, indicating the best tradeoff among accuracy, speed, and reliability.

Level III (ABB Yumi Gripper, 4 Tasks Using DMG)

Metrics for four tasks (gelatin box, cracker box, spatula, potted meat can) were as follows:

Metric	Range per Task
$C_i$3 (cm)	0.27–0.61
$C_i$4	2.7–11
$C_i$5	0.016–0.049
$C_i$6, $C_i$7 (cm)	0.66–1.12
$C_i$8 (cm)	0.004–0.057
Offline DMG Build Time (s)	10–18
Planning Time (s)	0.00007–0.023

These results illustrate the achievable precision and repeatability on tasks of highest difficulty using the Dexterous Manipulation Graph planner.

6. Extending and Customizing ActiveManip-Bench

• New objects: Add scanned or modeled meshes; define $C_i$9/$C_d$0 via mesh subregions.
• New tasks: Specify $C_d$1, $C_d$2 (or $C_d$3, $C_d$4) in the object frame. The required relative change $C_d$5 must be preserved for robot adaptation.
• New metrics: Augment the evaluator to include additional error functions (e.g., energy, torque). Geodesic evaluations require watertight, manifold meshes (as needed for heat method geodesic solvers).
• Software additions: Extend task datasets and evaluator outputs by modifying supplied JSON/YAML and entry-point scripts.

ActiveManip-Bench supports direct comparison between disparate in-hand manipulation approaches and provides a reproducible, extensible baseline for evaluating algorithmic and hardware advances in dexterous object manipulation.

7. Broader Context and Impact

ActiveManip-Bench addresses a critical need for standardized, reproducible methodology in the evaluation of in-hand robotic manipulation, enabling system-agnostic comparison across differing kinematic architectures and control paradigms. By specifying tasks through object-centric contact regions and SE(3) transforms, and by enforcing geometric consistency via mesh-based error metrics, the benchmark supports both analytical algorithm development and learning-based approaches. The detailed task taxonomy and granular metric reporting facilitate nuanced error analysis, identification of failure modes (e.g., object drops, orientation inaccuracies), and longitudinal tracking of progress within the manipulation community (Cruciani et al., 2020). These features have established ActiveManip-Bench as a reference methodology for benchmarking in-hand dexterity and adaptability in both research and applied robotics settings.