PartCrafter: Structured 3D Mesh Generation

Last updated: June 11, 2025

Recent advances in structured 3D generation ° are transforming how computer vision, graphics, and fabrication converge in digital and physical creation. PartCrafter introduces an end-to-end, part-aware 3D generative model ° that synthesizes multiple semantically meaningful and geometrically distinct meshes from a single RGB image, establishing robust new capabilities for part-based modeling and downstream pipelines (Lin et al., 5 Jun 2025 ° ).

Background and Problem Motivation

Central to digital fabrication, robotics, gaming, and industrial design is the need to generate 3D models ° that are both visually plausible and decomposable into functionally useful parts. Conventional methods—operating on monolithic point clouds, voxels, or meshes—have struggled to produce structured outputs amenable to editing, simulation, or fabrication (Lin et al., 5 Jun 2025 ° ). This presents limitations for tasks requiring semantic part awareness, such as computer-aided manufacturing and procedural content generation °.

Prior pipelines typically required either full-object synthesis followed by post-hoc segmentation, or multi-stage approaches in which parts are processed separately (Lin et al., 5 Jun 2025 ° , Noeckel et al., 2021 ° ). These two-stage and monolithic methods often incur inefficiencies and fail to guarantee physically realistic, part-aware outputs, hampering practical design workflows.

Foundational Approaches

Part-Based Representation

The concept of the “part” as a distinct operational unit recurs across several recent areas: fabrication-aware reverse engineering ° (Noeckel et al., 2021 ° ), hierarchical procedural generation ° (Beukman et al., 2023 ° ), and component-level robotic assembly ° (Isume et al., 19 Jul 2024 ° ). PartCrafter formalizes part-centric modeling by encoding a 3D scene or object as a collection of sets of latent variables, each mapped to a geometrically distinct mesh in a canonical space ° (Lin et al., 5 Jun 2025 ° ).

Fabrication-aware pipelines for carpentry leverage domain constraints ° such as planarity and joinery to improve the fidelity and fabricability of reconstructed assemblies (Noeckel et al., 2021 ° ). Hierarchical generation in games and simulation employs recursive composition of modular sub-generators to produce interpretable, optimizable large-scale structures (Beukman et al., 2023 ° ).

Unified, Compositional Mesh Generation

Traditional generative pipelines ° either output monolithic meshes or require explicit segmentation before part-wise handling. PartCrafter unifies this process by performing joint, end-to-end generation of multiple part meshes conditioned directly on an input image, with no reliance on pre-segmented masks or serial processing (Lin et al., 5 Jun 2025 ° ).

Principal Technical Advances

Architecture and Mechanisms

PartCrafter builds on transformer-based 3D mesh ° diffusion models, such as TripoSG, but introduces a compositional latent architecture (Lin et al., 5 Jun 2025 ° ). Each part is encoded as a set of disentangled latent tokens, with learnable identity embeddings ° promoting semantic differentiation among parts. During generation, hierarchical attention ° layers alternate between local (intra-part) and global (inter-part) self-attention °, ensuring both fine-grained part geometry and overall structural coherence.

Mathematical Formulation:

• Compositional latent space:

$\mathcal{Z} = \{\boldsymbol{z}_i\}_{i=1}^N \in \mathbb{R}^{NK \times C}$

where $N$ is number of parts, $K$ tokens per part, $C$ is the channel size (Lin et al., 5 Jun 2025 ° ).

• Hierarchical attention:
  • Local attention ° (intra-part):

  $\mathbf{A}_i^{\mathrm{local}} = \mathrm{softmax}\left(\frac{\boldsymbol{z}_i\boldsymbol{z}_i^\top}{\sqrt{C}}\right)$ - Global attention ° (inter-part):

  $\mathbf{A}^{\mathrm{global}} = \mathrm{softmax}\left(\frac{\mathcal{Z}\mathcal{Z}^\top}{\sqrt{C}}\right)$

These mechanisms alternate at each layer for integrated modeling.

PartCrafter applies image features from DINOv2 ° via cross-attention at both the local and global levels, facilitating semantic alignment and eliminating the need for pre-segmented input images (Lin et al., 5 Jun 2025 ° ).

Dataset and Training Pipeline

To enable part-level supervision, a new large-scale dataset is curated by mining part-labeled 3D objects ° from Objaverse, ShapeNet, and ABO, filtering for sufficient annotation quality ° and diversity. For scenes, the 3D-Front dataset ° is included. This yields approximately 50,000 objects with part labels and 300,000 individual parts. Training is initially performed with up to 8 parts per object, then fine-tuned for up to 16 (Lin et al., 5 Jun 2025 ° ). Single-part (“monolithic”) objects are regularly included as a regularization strategy °.

PartCrafter is initialized from pretrained TripoSG weights, then optimized end-to-end using a rectified flow matching ° objective with partwise permutation for invariance (Lin et al., 5 Jun 2025 ° ).

Empirical Results

3D Part-Level Generation

PartCrafter demonstrates improved quantitative performance ° over monolithic and two-stage approaches ° on part-aware reconstruction tasks:

PartCrafter not only yields lower Chamfer distance ° and higher F-score, but also produces parts that are better disentangled (lower IoU), critical for practical manipulation and editing. Crucially, it outperforms simply scaling the TripoSG backbone with more tokens, demonstrating the necessity of explicit compositional design ° (Lin et al., 5 Jun 2025 ° ).

Scene Generation

For multi-object scene generation °, PartCrafter maintains geometric and semantic quality even under severe occlusion, outperforming methods dependent on segmentation masks ° such as MIDI °. Generation time remains competitive or better across scene-level tasks (Lin et al., 5 Jun 2025 ° ).

Ablation Studies

• No local attention: Degenerate mesh generation °.
• No global attention: Overlapping, entangled parts.
• No identity embeddings: Loss of part distinction.
• Non-alternating attention: Inferior performance.

These findings systematically validate the architectural choices underpinning PartCrafter (Lin et al., 5 Jun 2025 ° ).

Extensions and Synergies

Fabrication-Aware Reverse Engineering

Carpentry-focused reverse engineering methods demonstrate the value of domain constraints (planarity, assembly contact, part thickness) for reconstructing parametric, editable, and fabricable models directly from photographs (Noeckel et al., 2021 ° ). Outputs are well suited to subsequent procedural manipulation and real-world fabrication steps.

Compositional Procedural Generation

Recursive, hierarchical generators—with reusable, independently optimized sub-components—facilitate efficient creation of complex content for games and simulations while retaining geometric and semantic control ° (Beukman et al., 2023 ° ). PartCrafter’s compositional mechanisms parallel these advances in a generative context.

Component-Based Physical Assembly

Robotic craft assembly employs pipelines that segment target images into parts, retrieve and align templates, and map simplified primitives to available scene objects based on proportional fitting, supporting real-world construction from flexible inventories (Isume et al., 19 Jul 2024 ° ). These methods advance automated matching and part-aware mapping beyond purely virtual domains.

Interactive Workflow Exploration

Tools such as CAMeleon support modular, extensible workflow definition within CAD ° environments, allowing users to experiment with and compare different fabrication processes on a given design (Feng et al., 23 Oct 2024 ° ). Usability studies ° highlight the benefit of discovery and visual feedback ° for both expert and novice users.

Current Applications

PartCrafter’s unified approach enables:

• Directly editable 3D models from images, with explicit part separation for modification and adaptation (Lin et al., 5 Jun 2025 ° ).
• Scene and object synthesis that preserves part structure, facilitating downstream simulation, animation, and robotics (Lin et al., 5 Jun 2025 ° , Isume et al., 19 Jul 2024 ° ).
• Integration with fabrication-aware design, lowering the barrier for realization in CAD-based or manual manufacturing workflows (Lin et al., 5 Jun 2025 ° , Noeckel et al., 2021 ° ).
• Procedural content generation leveraging reusable, parameterized parts (Beukman et al., 2023 ° ).

Limitations

• Data coverage: Performance is reduced when training data lacks rare ° part types or configurations (Lin et al., 5 Jun 2025 ° ).
• Generality: Application to new object categories or materials outside the core datasets and fabrication domains remains to be established (Lin et al., 5 Jun 2025 ° , Noeckel et al., 2021 ° ).
• Fabrication constraints: Domain-specific modeling (e.g., carpentry joints) may not transfer directly to freeform or organic assemblies (Noeckel et al., 2021 ° ).
• Usability and workflow integration: While modular and extensible tools like CAMeleon show promise, seamless integration ° of part-aware generation into physical and digital pipelines is ongoing work (Feng et al., 23 Oct 2024 ° ).

Conclusion

PartCrafter demonstrates that transformer-based, compositional latent generative models ° can achieve state-of-the-art decomposable 3D synthesis directly from single images, overcoming the rigidity and error-proneness of monolithic or two-stage approaches. Supported by methodological advances in assembly modeling, procedural generation, component selection, and workflow exploration, these models provide a foundation for more flexible, efficient, and user-friendly 3D content creation ° and fabrication.

References

• (Lin et al., 5 Jun 2025 ° ) PartCrafter: Structured 3D Mesh Generation ° via Compositional Latent Diffusion ° Transformers, 2025
• (Noeckel et al., 2021 ° ) Fabrication-Aware Reverse Engineering for Carpentry, 2021
• (Beukman et al., 2023 ° ) Hierarchically Composing Level Generators for the Creation of Complex Structures, 2023
• (Isume et al., 19 Jul 2024 ° ) Component Selection for Craft Assembly Tasks, 2024
• (Feng et al., 23 Oct 2024 ° ) CAMeleon: Interactively Exploring Craft Workflows in CAD, 2024

Speculative Note

The potential convergence of part-aware generative models like PartCrafter with modular workflow tools such as CAMeleon may lead to integrated systems capable of context-aware design, adaptation to fabrication constraints, and end-to-end automation ° from image-based modeling to final assembly. Realization of such workflows—bridging AI-driven generation, human-in-the-loop ° editing, and digital-to-physical fabrication—remains a promising direction for future research and development.