---
title: Functional Manipulation Benchmark (FMB)
url: https://www.emergentmind.com/topics/functional-manipulation-benchmark-fmb
type: topic
---

# Functional Manipulation Benchmark (FMB)

The Functional Manipulation Benchmark (FMB) is a standardized, extensible framework for evaluating robotic manipulation systems on contact-rich, generalizable, and hierarchically compositional tasks. FMB defines a suite of multi-stage, high-precision manipulation challenges using procedurally generated 3D-printed objects, supporting rigorous assessment of skills such as functional grasping, object reorientation, compliant insertion, and multi-part assembly in settings that demand robustness to perceptual uncertainty and physical tolerance [2401.08553; 2404.06645]. The benchmark emphasizes reproducibility, broad object variability, and scalability, enabling comprehensive evaluation of perception, planning, and control pipelines in both learning-based and classical robotics approaches.

## 1. Design Principles and Objectives

FMB addresses limitations in both trivial pick-and-place tests (which lack functional and contact complexity) and overly specialized industrial insertion tasks (which have limited generality). Its primary objectives are:

- **Contact-Rich Interaction:** Task structures demand reasoning over force and torque, with emphasis on compliant control strategies.
- **Generalization Across Geometry:** Procedural object generation introduces systematic variation in shape, size, and appearance, allowing controlled studies of model robustness to novel configurations.
- **Reproducibility and Accessibility:** All hardware elements (CAD files, robot configuration, fixtures) and software components (data collection, evaluation scripts) are open-sourced, targeting common research platforms (e.g., Franka Panda, RealSense D405).
- **Compositionality:** FMB comprises not only isolated manipulation primitives but multi-stage pipelines (grasp–reorient–insert–assemble), measuring both atomic skills and their functional integration in long-horizon sequences.

## 2. Task Suite and Procedural Object Generation

FMB’s task suite comprises single-object multi-stage tasks and multi-object assembly scenarios, leveraging a family of parametric 3D-printed parts:

- **Object Set:** Nine base shape families (e.g., rectangle, star, hexagon), each with six size variants (controlled via shape parameter $s \sim \mathrm{Uniform}\{1,\ldots,6\}$), and eight color options, yielding 54 base parts and multi-piece interlocking boards.
- **Task Primitives:** Fundamental skills include functional grasping (selection of grasp pose according to downstream utility), repositioning/regrasp using external fixtures, and compliant insertion into precisely dimensioned slots.
- **Assembly Tasks:** Boards featuring 4–5 interlocking slots require ordered insertion of shape-mated parts—a controlled testbed for coordination and error compounding in hierarchical policies.
- **Procedural Generation:** Object placements, board positions, and part orientations are randomized according to bounded uniform distributions (e.g., part positions over $20\times30$ cm grids, board yaw $\sim \mathrm{Uniform}[0,2\pi]$) [2401.08553].

## 3. Hardware, Perceptual Pipeline, and Data Collection

FMB’s hardware and data infrastructure is specified for rigorous comparison:

- **Robot Platform:** Experiments are designed for the Franka Emika Panda arm (7 DoF), equipped with a parallel-jaw gripper.
- **Sensing:** Four RealSense D405 (RGB+D) cameras—two wrist-mounted, two stationary—supply multi-view perception; the robot’s built-in torque estimation provides $\approx0.05$ N force/torque feedback.
- **Human Demonstrations:** Teleoperation (6D twist via SpaceMouse) collects approx. $22,550$ segmented trajectories, temporally aligned with primitive labels for skill learning.
- **Recording Modalities:** Each timestep includes multi-view RGB/D, end-effector $(\mathbf{x}_t, \mathbf{q}_t)$, velocity, force/torque $\tau_t \in \mathbb R^6$, and binary gripper state $g_t$ [2401.08553].

## 4. Evaluation Metrics and Protocols

FMB uses explicit metrics and standardized empirical procedures:

- **Success Rate:** For policy $\pi$ on task $T$, $R(\pi;T) = (1/N) \sum_{i=1}^N S_i$, with $S_i\in\{0,1\}$ indicating trial outcome.
- **Generalization Gap:** $\Delta_{\mathrm{gen}} = R_{\mathrm{seen}} - R_{\mathrm{unseen}}$, contrasting performance on trained-vs-held-out objects (by shape or size); uniform sampling enables controlled generalization studies [2401.08553].
- **Completion Time:** Mean completion over successful trials, enabling comparison of temporal efficiency.
- **Success Tolerances:** For insertion, translation error is constrained to $||\Delta p||\le1$ mm, and orientation $\lesssim0.5^\circ$ (timeout-enforced); force thresholds (\textit{e.g.}, $|F_z|\ge 1$ N for contact, $F_z \le -1$ N for misalignment, $F_z \ge 2$ N for insertion push) parameterize compliant control APIs [2404.06645].
- **Multi-Stage/Holistic Evaluation:** Pipelines integrate multiple primitives; success is only logged if all stages succeed, benchmarking compounding error and closed-loop adaptability.

**Table. FMB Peg-In-Hole Subtasks and Baseline Success Rates (10 Trials Each) [2404.06645]**

| Shape         | Yaw Sampling                   | GenCHiP (0-shot) | Scripted | Point-to-Point |
|---------------|-------------------------------|------------------|----------|----------------|
| Circular peg  | N/A (cylindrical symmetry)    | 100%             | 100%     | 70%            |
| Star peg      | Uniform in $[0, \pi/2]$       | 80%              | 10%      | 0%             |
| Half-pipe peg | $\{0, \pi\}$ (two-fold)       | 50%              | 0%       | 0%             |

## 5. Sensing, Perception, and Compliance Control

- **Pose Estimation:** Wrist-mounted RGB and depth imaging, Fast R-CNN with FPN backbone, and 6D keypoint regression yield object pose estimates with up to 4 mm residual error.
- **Adaptive Control:** A Cartesian admittance interface exposes compliance, parameterized via per-axis stiffness vector $\sigma\in[0,1]^6$. The control law is
  $$
  \ddot{x} = M^{-1}\big[K_p(x_d-x) + K_d(\dot{x}_d - \dot{x}) + (f_d - f_s)\big]
  $$
  with $K_p$, $K_d$ diagonal gains from $\sigma$, $f_s$ sensed, $f_d$ target wrench. Conditional logic combines pose and force predicates for sub-motion terminations (e.g., "stop on contact," "stop if inserted") [2404.06645].
- **Perception-Action Loop:** Policies can call $\mathrm{estimate\_and\_update\_pose}(\mathrm{object\_id})$ before each primitive, capturing runtime pose errors.

## 6. Policy Architectures and Learning Baselines

FMB provides benchmarks not only for classical models but deep learning systems:

- **Baselines:** Behavior cloning via ResNet-34 or Transformer architectures, leveraging both vision and force/torque as input modalities. Policies are conditioned on task primitive and object identity (object ID).
- **Reported Results:** In insertion, force/torque input is crucial (RGB+D+τ: 11/25 vs. RGB-only: 2/25). Conditioning policies by object ID improves shape-wise generalization (e.g., Transformer: 27/45 vs. ResNet: 14/45) [2401.08553].
- **Hierarchical vs. Flat Policies:** End-to-end ("flat") policies fail consistently on multi-stage tasks (0/10), while hierarchical pipelines—where primitives are composed based on labeled transitions—exhibit superior robustness (up to 19/30 on single-object, 7/10 on multi-object tasks).

## 7. Reproducibility and Extension

All FMB resources are open-sourced:

- **CAD and Assembly:** Downloadable parts for all objects and boards, along with assembly and calibration instructions.
- **ROS Controller Stack:** Modular codebase supports demonstration capture, primitive segmentation, low-level impedance control, and evaluation protocols.
- **Evaluation and Leaderboard:** Scripts for generating success metrics, generalization gaps, and summary tables. Leaderboards can be generated from submitted logs for community-wide comparison.
- **Simulation Adapters:** Wrappers are available for MuJoCo, Drake, Bullet, and other simulation environments, ensuring alignment between real and simulated protocols [2401.08553; 2001.03070].

FMB’s extensible software and data structures support rapid expansion to new tasks, hardware classes, perception pipelines, and manipulation modalities, preserving its utility as a foundation for comparative evaluation and ablation of robotic manipulation policies [2401.08553; 2001.03070; 2404.06645].

Source: https://www.emergentmind.com/topics/functional-manipulation-benchmark-fmb