AssemLM: Spatial Reasoning Multimodal Large Language Models for Robotic Assembly

Abstract: Spatial reasoning is a fundamental capability for embodied intelligence, especially for fine-grained manipulation tasks such as robotic assembly. While recent vision-LLMs (VLMs) exhibit preliminary spatial awareness, they largely rely on coarse 2D perception and lack the ability to perform accurate reasoning over 3D geometry, which is crucial for precise assembly operations. To address this limitation, we propose AssemLM, a spatial multimodal LLM tailored for robotic assembly. AssemLM integrates assembly manuals, point clouds, and textual instructions to reason about and predict task-critical 6D assembly poses, enabling explicit geometric understanding throughout the assembly process. To effectively bridge raw 3D perception and high-level reasoning, we adopt a specialized point cloud encoder to capture fine-grained geometric and rotational features, which are then integrated into the multimodal LLM to support accurate 3D spatial reasoning for assembly tasks. In addition, we construct AssemBench, a large-scale dataset and benchmark for assembly-oriented spatial reasoning, comprising over 900K multimodal samples with precise 6D pose annotations. AssemBench extends spatial reasoning evaluation beyond 2D and grounding tasks into full 3D geometric inference, filling a critical gap in existing embodied AI benchmarks. Extensive experiments demonstrate that AssemLM achieves state-of-the-art performance in 6D pose reasoning across diverse assembly scenarios. Furthermore, real-robot evaluations show that our model can support fine-grained and multi-step assembly execution in real-world settings, demonstrating its potential for robotic assembly applications.

• The paper introduces AssemLM, a unified multimodal model that fuses assembly manuals, point clouds, and language to predict accurate 6D poses in robotic assembly.
• The approach leverages a SE(3)-equivariant encoder and embedding-level fusion with DeepStack, achieving significant improvements in pose accuracy and zero-shot generalization.
• Real-world experiments demonstrate robust sim-to-real transfer with >80% per-step success, underscoring the model's potential for complex robotic assembly tasks.

AssemLM: A Multimodal LLM Framework for Robotic Assembly

Introduction

AssemLM provides a spatially rigorous, multimodal LLM architecture designed to bridge 3D geometric perception and high-level reasoning for robotic assembly. Unlike previous approaches, which are constrained by either their reliance on single-modal inputs (e.g., only point clouds) or their inability to achieve fine-grained 6D (SO(3) $\times$ SE(3)) spatial registration, AssemLM achieves precise, generalizable pose prediction by fusing rich modalities including assembly manuals, point clouds, and natural language instructions. Trained on the comprehensive AssemBench dataset—which contains over 900K annotated multimodal samples spanning 150K assembly steps and 50+ object categories—AssemLM demonstrates robust zero-shot generalization, achieving high-fidelity pose predictions across seen and unseen domains, including in real-world manipulation tasks. The architecture's modular design introduces a SE(3)-equivariant point cloud encoder, embedding-level intermodal fusion with DeepStack, and a highly specialized pose tokenizer. These elements collectively advance the state of the art in spatial reasoning for embodied AI.

Architecture and Methodology

AssemLM is centered on a 2B-parameter transformer backbone based on Qwen3-VL-2B-Instruct. The architecture is characterized by the following components and strategies:

• SE(3)-equivariant Geometric Perception: Point clouds encoding fixed and moving parts are processed by a Vector Neuron DGCNN, extracting both SE(3)-equivariant features $F$ and SE(3)-invariant features $G$. The model constructs geometric correlation features for robust pose reasoning across all rigid body transformations.
• Multimodal Embedding-level Fusion with DeepStack: Assembly manuals are encoded by a SigLIP-2-based vision encoder and injected—together with point cloud features—into early transformer layers via DeepStack. This approach allows multi-scale spatial and semantic consistency without complex cross-attention.
• Pose Tokenization: Assembly pose prediction is reframed as sequence modeling via a custom vocabulary representing uniformly binned, normalized 9D pose vectors (3D translation, 6D continuous rotation). This avoids subword redundancy and instability, enabling stable training and geometric regression in the LLM context.
• Optimized SFT: Initial geometry warm-up stabilizes the point cloud encoder, decoupling geometric feature acquisition from multimodal alignment. End-to-end SFT is conducted on the comprehensive AssemBench corpus.

  Figure 1: AssemLM architecture features unified embedding-space fusion of assembly manuals, point clouds, and language for autoregressive 6D pose prediction.

AssemBench: The Multimodal Assembly Benchmark

Addressing the critical deficit in large-scale, multimodal, spatially annotated datasets for assembly, AssemBench leverages procedural mesh generation, rendering, and multimodal annotation (combining generative and rejection sampling approaches). The dataset encompasses 150K unique steps and 900K multimodal samples, with assets normalized into a canonical coordinate frame and annotated with deterministic, high-precision 6D pose ground truths. Render styles (Freestyle, Non-Freestyle, Lineart) diversify visual supervision, while instructions are curated at varying semantic granularities.

Figure 2: AssemBench statistics and data pipeline, illustrating scaling and curation mechanisms for multimodal annotation.

Quantitative and Qualitative Evaluation

Multi-Category and Zero-Shot Generalization

Compared to leading SE(3)-equivariant assembly models (e.g., TwoByTwo) and large foundation LMMs (GPT-5.2, DeepSeek-V3.2), AssemLM achieves:

• Mean RMSE(T): 0.0203 (order of magnitude lower vs. baselines)
• Symmetric Chamfer Distance (SCD): 0.0109
• Success Rate (SR): 89.4% (vs. 14.5% for TwoByTwo)

Performance remains robust in zero-shot evaluation on the full IKEA dataset—a notably OOD source—with SR of 81.0%. Category-wise, AssemLM maintains strong generalization across Fragments, Furniture, and Daily Objects.

Modality Importance and Ablations

Interpretability analysis via cross-attention attribution demonstrates a dominant reliance on SE(3) point cloud features (82%), with visual manuals (14%) resolving part identity/global context and language (4%) guiding task intent—consistent with the architectural aim of geometric grounding rather than superficial pattern learning.

Figure 3: Aggregated attention heatmap and modality contribution analysis highlight geometric dominance and auxiliary augmentations from images/instructions.

Ablation studies affirm the indispensable contributions of the pose tokenizer, DeepStack multimodal injection, and the explicit SE(3)-equivariant module. Removal of any component yields substantial translation/rotation accuracy degradation.

Figure 4: Ablation study: dataset scale, rotation randomization, and tokenizer design critically impact generalization and geometric stability.

Real-World Assembly and Asset Workflow

Evaluations on a Flexiv Rizon 4s robot across four high-precision, multi-step assembly tasks (Insert Plug, Store Cans, Insert Flower, Build Blocks) confirm effective sim-to-real transfer without domain-specific fine-tuning. AssemLM surpasses TwoByTwo on all tasks—Insert Plug (43.3% vs 20.0%), Store Cans (63.3% vs 0.0%), Insert Flower (66.7% vs 63.3%), Build Blocks (30.0% vs 13.3%)—demonstrating not only single-step accuracy but improved task-horizon resilience. The model maintains >80% per-step success, with error accumulation rather than intrinsic prediction error as the principal limiting factor.

Figure 5: Real-world experimental setup: four robotic assembly tasks reveal real-world applicability with challenging physical constraints.

Figure 6: Real-world assembly steps and corresponding instruction manuals utilized for multimodal pose prediction.

Figure 7: Visualization of assembly pose predictions by AssemLM, demonstrating high correspondence with ground truth across diverse assemblies.

Implications and Future Directions

AssemLM represents a significant step towards scalable, generalizable spatial reasoning in embodied AI:

1. Unified Multimodal Reasoning: The architectural innovations enable cross-modal grounding of language, vision, and geometry, supporting compositional generalization essential for long-horizon robotic manipulation.
2. High-Fidelity Pose Prediction as Autoregressive Modeling: The pose tokenization approach facilitates stable, discrete regression goals, offering a blueprint for integrating geometric targets into next-token prediction pipelines—potentially applicable to trajectory or action planning in broader manipulation settings.
3. Scalable Training and Sim-to-Real Transfer: The combination of large-scale synthetic training (with procedural supervision) and strong real-world generalization underscores the potential for data-centric sim2real pipelines in assembly and beyond.

Several theoretical and practical research avenues emerge:

• Extension to General Manipulation Pipelines: AssemLM's structured outputs position it as a high-level reasoning module for downstream planning, simulation, and instruction-following tasks, particularly when compositionality or multi-modal grounding is critical.
• Policy Learning and Closed-Loop Control: Integration with low-level motion policies, tactile sensing, and vision-based feedback loops can leverage AssemLM's pose grounding for robust closed-loop execution.
• Further Dataset Expansion: With easy scalability, AssemBench can be extended for hierarchical, multi-object, or damage-aware assembly settings, supplying a foundation for robust self-supervised representation learning in robotics.

Conclusion

AssemLM establishes a new benchmark for large-scale, multimodal, 6D pose-centric spatial reasoning in robotics. Through tight integration of SE(3)-equivariant geometric encoders, efficient embedding-space fusion, and a discretized pose modeling paradigm, AssemLM achieves state-of-the-art accuracy, robust generalization, and effective deployment in real-world assembly tasks. The architectural and dataset advances in this work are likely to inform both foundational research into 3D-aware LLMs for robotics and practical methodologies for sim2real transfer, spatial program induction, and human-robot collaboration.