AssemBench: Multimodal Benchmark for Robotic Assembly

• AssemBench is a large-scale, multimodal benchmark suite designed for fine-grained 6-DoF robotic assembly with sub-millimeter precision.
• It integrates visual, point cloud, and language data to support robust evaluation of single-step pose estimation, part grounding, and multi-step assembly planning.
• Its deterministic, high-fidelity annotations and diverse object categories enable rigorous cross-domain comparisons against existing embodied AI and robotics datasets.

AssemBench is the designation for multiple distinct, domain-specific benchmark suites—each widely recognized under this or closely related spelling—across robotics, program analysis, and AutoML research. The most prominent current instance is the large-scale, multimodal, pose-annotated dataset for embodied spatial reasoning in robotic assembly, introduced by Gan et al. (Jing et al., 10 Apr 2026). Additionally, the term was previously applied as a branding for the first community-maintained disassembly benchmark for software binaries (Li et al., 2020), and as a colloquial variant for the Assembled-OpenML ensemble meta-learning toolkit (Purucker et al., 2023). This article focuses primarily on the robotics-oriented AssemBench, then briefly surveys prior and related usages.

1. Motivation and Conceptual Overview

AssemBench was introduced to address the gap in benchmarks for fine-grained 3D spatial reasoning in robotic assembly. Unlike most existing embodied AI datasets, which focus on 2D understanding, object detection, or coarse action selection, robotic assembly necessitates metric-accuracy 6-DoF pose inference. Specifically, for successful step-wise assembly, an agent must precisely align and mate component geometries with sub-millimeter translation and sub-degree rotation tolerances. Previous datasets either lacked 6D pose supervision, were limited to a few domains (e.g., furniture, toys), or did not couple visual, point cloud, and language modalities at scale.

2. Dataset Composition, Modalities, and Statistical Properties

AssemBench comprises over 150,000 unique assembly steps, each further expanded by visual rendering style and language instruction granularity to yield over 900,000 multimodal annotated samples. Input and annotation conventions are summarized in Table 1.

Aspect	Description	Scale / Types
Steps	Distinct assembly actions (e.g., fit leg to seat)	150,000
Augmentation	×3 rendering styles ×2 instruction types	≈900,000 total
Categories	Furniture, daily objects, fragments (>50 domains)	Furniture: 60k, Daily: 50k, Fragments: 40k
Modalities	3D point clouds (1024 points/part), before/after images, language	All steps multimodal
Pose Labels	Full SE(3): $$T = \begin{bmatrix} R & t \ 0 & 1 \end{bmatrix}$$; 9D vector	Deterministic (Blender)
Rendering Styles	Freestyle (photoreal+edges), Non-Freestyle (Cycles), Lineart	3 variants
Instruction Styles	Precise (e.g., ‘attach X to Y’), Vague (‘assemble the chair’)	2 variants
Distribution	Stratified by category; independent SO(3) rotation augmentation	Canonical z-up, origin anchored

All object meshes and point clouds share a normalized, canonical coordinate system (right-handed, z-up, unit length for longest axis, origin at object footprint center), facilitating consistent rotation and transformation learning. Visual manuals are provided as high-resolution raster images (PNG/JPEG) under each rendering variant. All label, instruction, and pose meta-data are JSON-serialized.

3. Annotation Protocol and Precise Ground Truth

Ground-truth 6D poses are generated algorithmically within Blender using mesh-based anchors for deterministic SE(3) transformations. Each pose comprises a translation $t \in \mathbb{R}^3$ and a rotation $R \in \mathrm{SO}(3)$, internally stored as a 6D continuous representation following Zhou et al. (2019) to avoid quaternion and Euler angle discontinuities. Each sample is thus uniquely defined by a 9D vector $(t_x, t_y, t_z, r_{11}, ..., r_{23})$. Labels are generated with sub-millimeter translation and sub-degree rotation fidelity, ensuring that superhuman precision is attainable for geometric supervision. Input point clouds are sampled by area-weighted farthest point sampling from watertight meshes, supporting algorithms sensitive to surface detail.

4. Benchmark Tasks and Evaluation Metrics

AssemBench formalizes three principal tasks to quantify embodied spatial reasoning proficiency:

• Task A: Single-Step Pose Estimation
  • Translation RMSE: $$\mathrm{RMSE}(T) = \sqrt{E\|t_{\text{pred}} - t_{\text{gt}}\|^2}$$
  • Symmetric Chamfer Distance (SCD): $$2 \times \text{CD}(P_{\text{moved}}, P_{\text{target}})$$, capturing shape-symmetry-agnostic rotational deviations
  • Success Rate (SR): Fraction with SCD $< 0.02$ (approx. CD $< 0.01$)
• Task B: Part Grounding Classification: Identify which among multiple free parts should be moved next and predict its pose. Metric: Top-1 classification accuracy and pose metrics as above.
• Task C: Multi-Step Assembly Planning Given a full instruction manual (sequence of before/after pairs) and a list of parts, predict a sequence of ordered SE(3) transforms. Metrics: Cumulative step-wise RMSE, SCD, and SR across $k$-step plans.

Standard splits are 90% train / 10% test (135k/15k steps), stratified by category. Challenge subsets include zero-shot (unseen IKEA categories) and “low-data” ablations (TwoByTwo, 350 steps for data efficiency studies).

5. Comparative Analysis with Related Benchmarks

Relative to existing robotics and embodied AI datasets:

• Distinction: AssemBench is unique in its scale ($t \in \mathbb{R}^3$0900k samples), breadth ($t \in \mathbb{R}^3$150 object categories across everyday, furniture, and broken fragments), and fully multimodal, precisely labeled 6-DoF coverage.
• Prior datasets: LineMOD and YCB house small-scale 6D pose data, but with limited object diversity and without multimodal supervision. IKEA-ASM, Assembly101, and ReAssemble focus on single domains or 2D/3D pose estimation but lack precision and multimodal fusion.
• Gap filled: Targets the transition from "where is the part" detection to "how should this part move (translate/rotate) in 6D to fit correctly," a critical distinction for physical robot deployment (Jing et al., 10 Apr 2026).

6. Applications, Baseline Performance, and Empirical Insights

AssemBench supports the development, training, and rigorous evaluation of next-generation multimodal models. The canonical baseline is AssemLM, a spatial reasoning MLLM that ingests point clouds, images, and language. Notable use cases and performance results:

• Cross-modal spatial reasoning: AssemLM achieves 89.4% SR on AssemBench (RMSE(T)$t \in \mathbb{R}^3$2), compared to 14.5% for SE(3)-only TwoByTwo and 7.9% for LLM-only GPT-5.2.
• Zero-shot generalization: On unseen IKEA categories, AssemLM (trained on AssemBench) obtains ~81% SR, whereas purely geometric baselines attain ~6%.
• Robotic deployment: Deployed on a Flexiv Rizon 4s, AssemLM attains 50.8% real-world SR on four tasks (Insert Plug, Store Cans, Insert Flower, Build Blocks), doubling the best prior SE(3)–equivariant alternatives (24%).
• Annotation conventions: Compositional and challenge splits support robust ablations and “category-gap” tests essential for future foundation-model research (Jing et al., 10 Apr 2026).

7. Prior and Alternative Usages of the "AssemBench" Name

Historically, "AssemBench" and orthographic variants have also denoted:

• Disassembly Benchmark for Binary Analysis:

The first open, cross-platform benchmark for evaluating binary disassemblers on instruction-level accuracy, diversity (multiple compilers, optimization levels), and resource usage (Li et al., 2020). This suite comprises 879 binaries across 15 C projects, with full ground-truth via compiler listing files. Evaluation metrics include precision, recall, F1 accuracy, and wall-clock/memory usage. It fundamentally advanced reproducibility and extensibility in disassembler research.

• AutoML Ensembles Meta-benchmark ("Assembled-OpenML"):

Python toolkit and meta-dataset suite for simulating ensemble-learning techniques on OpenML classification tasks, using “metatasks” that aggregate predictions/confidences from 1000s of model runs (Purucker et al., 2023). Benchmarking supports fast, efficient evaluation of stacking, voting, dynamic selection methods, and establishes a normalized “closed-gap” performance metric.

• Assembly-to-Source Translation Evaluation:

Asm2SrcEval, a systematic LLM benchmark for assembly-to-source translation, is proposed as a core task for future AssemBench deployments, increasing cross-domain relevance (Hamedi et al., 28 Nov 2025).

8. Impact and Future Directions

AssemBench establishes new standards in geometric, multimodal benchmark design for robotics, grounding future development of spatial-reasoning models suited for fine-grained manipulation. Principal open problems include:

• Enhanced performance on low-data and zero-shot settings.
• Extension to multi-stage, force-augmented, and real-world noisy sensor input.
• Integration with related evaluation paradigms (e.g., spatio-temporal video reasoning benchmarks such as Flat-Pack Bench (Chetan et al., 20 May 2026)).
• Community-driven expansion to include new part categories, assembly types, and additional annotation dimensions for higher-fidelity real-world modeling.

A plausible implication is that AssemBench's approach to deterministic, highly precise pose labeling and multimodal supervision will form the template for embodied AI benchmarks across physically grounded manipulation tasks. Its open scale and stratification make it suitable for rigorous, extensible standardized evaluation of both future foundation models and task-specific architectures for robotic assembly (Jing et al., 10 Apr 2026).