AssemLM: Spatial Multimodal LLM for Robotic Assembly

• The paper introduces AssemLM, a spatial multimodal LLM that combines visual, geometric, and textual cues using a unified Transformer backbone for precise 6D assembly pose prediction.
• It leverages a SE(3)-equivariant VN-DGCNN for encoding point clouds and integrates multi-scale features from a SigLIP-2 Vision Transformer, ensuring effective cross-modal spatial reasoning.
• Evaluations on AssemBench and real-robot tasks demonstrate robust performance with an 89.4% success rate, significantly outperforming prior vision-language models in fine-grained assembly tasks.

AssemLM is a spatial multimodal LLM designed for high-fidelity robotic assembly. It directly integrates visual manuals, 3D point cloud perception, and textual instructions to achieve accurate reasoning over 6D assembly poses, addressing the core limitations of prior vision-LLMs in fine-grained 3D spatial inference (Jing et al., 10 Apr 2026). The AssemLM architecture utilizes a unified multimodal Transformer backbone to support explicit, cross-modal spatial reasoning—enabling step-wise, multistage assembly and robust transfer to real-robot tasks.

1. Multimodal System Architecture

AssemLM employs a unified Qwen3-VL-2B-Instruct Transformer backbone that processes multimodal inputs:

• Two rendered images from the assembly manual (before/after the target step)
• Natural language assembly instruction for the current step
• Two 3D point clouds: one of the fixed subassembly and one of the part to be attached

Each modality is individually encoded and linearly projected to a common embedding space. Point clouds are encoded by a SE(3)-equivariant Vector Neuron Dynamic Graph CNN (VN-DGCNN), capturing both equivariant (orientation/position-preserving) and invariant (transformation-independent) geometric features. Manual images are processed by a SigLIP-2 Vision Transformer, with multi-scale features injected into early layers of the main Transformer backbone via DeepStack modules. Standard text tokens are included directly.

All embeddings, including geometric features and visual representations, are concatenated via modality-specific position encodings using M-ROPE. This preserves spatial ordering and supports effective cross-modal attention.

The model predicts assembly action as a sequence of nine discrete pose tokens, which are converted via a de-tokenizer MLP into a 6D assembly pose (3D translation, continuous 6D rotation). This explicit pose prediction supports direct robotic execution (Jing et al., 10 Apr 2026).

2. Rigorous 6D Pose Representation

AssemLM is formulated for explicit prediction of task-critical 6D rigid transforms:

• Translation vector $t = [t_x,\, t_y,\, t_z]^T \in \mathbb{R}^3$
• Rotation parameterized in a 6D continuous representation, convertible to quaternion $q \in S^3$ or SO(3) matrix

Each 6D pose is represented as a 9D continuous vector $[t_x, t_y, t_z, r_1, \ldots, r_6]$; each dimension is normalized and discretized into 201 bins. The learned tokenizer and de-tokenizer ensure fidelity in both translation and orientation.

For evaluation, both standard rigid transforms and homogeneous composition (via $4 \times 4$ matrices) are supported. This enables the model to reason about and apply incremental frame-to-frame spatial relations in multi-step assembly (Jing et al., 10 Apr 2026).

3. Training Regimen and Data Construction

AssemLM utilizes a two-stage supervised fine-tuning protocol:

1. Geometry Warm-up: Only the point cloud encoder and a projection head are trained to predict 6D pose. Warm-up minimizes the translational $L_1$ loss $\| \hat{t} - t^* \|_1$ and the rotation angular error $$\arccos \left(\frac{\text{trace}(\hat R^T R^*) - 1}{2}\right)$$, with unit weights for translation and rotation.
2. Full Multimodal Alignment: With the warm-up head frozen and then removed, the trained point cloud encoder is connected to the Transformer. The LLM is then fine-tuned to predict next pose tokens under standard cross-entropy loss.

The training is supervised by AssemBench, a curated dataset comprising approximately 900,000 multimodal samples: 150,000 assembly steps from over 50 object categories, each annotated with precise 6D pose. Each datapoint spans 3 manual rendering styles, two instruction granularities, and canonical point cloud normalization (z-up, scale to unit axis, FPS to 1,024 points). The AssemBench test split comprises 10% of all steps, ensuring every object and source pair appears in both train and test partitions (Jing et al., 10 Apr 2026).

4. Evaluation Protocols and Benchmarks

Performance is rigorously evaluated via multiple metrics relevant to assembly and pose accuracy:

Metric	Definition/Measurement
RMSE(T)	Root mean square error in translation prediction
SCD	Symmetric Chamfer Distance between predicted & ground-truth clouds
SR	Success Rate: fraction of cases with $SCD < 0.02$
ADD / ADD-S	Average Distance, with or without symmetry matching

On the AssemBench test set (multi-category, ∼30,000 steps), AssemLM achieves:

Method	RMSE(T) ↓	SCD ↓	SR ↑
TwoByTwo	0.1669	0.1452	14.5%
GPT-5.2	0.2238	0.2512	7.9%
DeepSeek-V3.2	0.2747	0.4162	3.3%
AssemLM	0.0203	0.0109	89.4%

Ablation shows removal of the vision stream or SE(3)-equivariant encoding yields the largest performance decrease (SR drops to 68.3% and 67.6% respectively). This suggests robust multimodal and geometric fusion is necessary for fine assembly tasks. (Jing et al., 10 Apr 2026)

5. Real-Robot and Generalization Performance

AssemLM is validated on a Flexiv Rizon 4s manipulator in four multi-stage tasks ("Insert Plug," "Store Cans," "Insert Flower," "Build Blocks"), directly inferring 6D goal poses. Success rates range from 30%–66% depending on task, and overall 50.8%, doubling the performance of the strongest baseline (TwoByTwo, 24.0%).

Zero-shot evaluation on unseen IKEA objects (Bench/Chair/Desk)—without any fine-tuning—yields SR of 81.0% (AssemLM) versus 6.5% (TwoByTwo), with RMSE(T) improved an order of magnitude (0.0263 vs. 0.2012). This demonstrates that the explicit 3D spatial reasoning enabled by AssemLM generalizes beyond the supervised categories and to real-world objects (Jing et al., 10 Apr 2026).

6. Comparison with Graph-Based Assembly LLMs

AssemLM contrasts with graph-based LLMs such as AssemMate (Zheng et al., 15 Sep 2025), which utilize structured knowledge graphs—rather than raw 3D perception—as the grounding mechanism for spatial and procedural reasoning. AssemMate leverages self-supervised Graph Convolutional Networks (CompGCN) for entity-relation encoding, supporting knowledge graph question answering (KGQA) and grasp planning. However, this approach is dependent on the completeness and accuracy of symbolic assembly descriptions, and primarily operates in the discrete, graph-structured semantic space.

AssemLM, in contrast, enables direct geometric reasoning—mapping observed 3D structure and spatial cues into pose actions, with integrated vision and instruction signals. The ablation data show that point cloud and vision are both essential modalities for AssemLM’s high precision in pose prediction.

7. Limitations and Implications

AssemLM's spatial multimodal reasoning architecture confirms the necessity of incorporating both explicit geometric encodings and perceptual multimodality for fine-grained manipulation. The most substantial limitations reported include:

• Performance depends on the availability of precise 3D point clouds and high-fidelity vision streams.
• Assembly instructions and manuals must be rendered in a compatible format, with canonical coordinate normalization.
• The architecture is validated primarily in bench-scale and furniture-scale tasks; scalability to highly complex or real-time multi-agent environments remains to be demonstrated.

A plausible implication is that further advances will require: (1) continuous online learning from unstructured multimodal streams; (2) more generalizable geometric encoders; and (3) tighter integration with dynamic scene understanding and task-level planning (Jing et al., 10 Apr 2026).

• "AssemLM: Spatial Reasoning Multimodal LLMs for Robotic Assembly" (Jing et al., 10 Apr 2026)
• "AssemMate: Graph-Based LLM for Robotic Assembly Assistance" (Zheng et al., 15 Sep 2025)