PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models (2412.18608v2)

Abstract: Text- or image-to-3D generators and 3D scanners can now produce 3D assets with high-quality shapes and textures. These assets typically consist of a single, fused representation, like an implicit neural field, a Gaussian mixture, or a mesh, without any useful structure. However, most applications and creative workflows require assets to be made of several meaningful parts that can be manipulated independently. To address this gap, we introduce PartGen, a novel approach that generates 3D objects composed of meaningful parts starting from text, an image, or an unstructured 3D object. First, given multiple views of a 3D object, generated or rendered, a multi-view diffusion model extracts a set of plausible and view-consistent part segmentations, dividing the object into parts. Then, a second multi-view diffusion model takes each part separately, fills in the occlusions, and uses those completed views for 3D reconstruction by feeding them to a 3D reconstruction network. This completion process considers the context of the entire object to ensure that the parts integrate cohesively. The generative completion model can make up for the information missing due to occlusions; in extreme cases, it can hallucinate entirely invisible parts based on the input 3D asset. We evaluate our method on generated and real 3D assets and show that it outperforms segmentation and part-extraction baselines by a large margin. We also showcase downstream applications such as 3D part editing.

• The paper introduces PartGen, a novel framework that segments 3D objects into semantic parts using a two-stage multi-view diffusion model.
• The approach completes occluded components by leveraging context from unmasked views, achieving superior results with metrics like mAP, CLIP, LPIPS, and PSNR.
• Applications include part-aware text-to-3D generation, practical editing for gaming, VR, manufacturing, and enhanced robotic vision systems.

Insights into PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models

The paper introduces "PartGen," a novel approach for creating and reconstructing 3D objects that are distinctively composed of multiple meaningful parts. These 3D objects can originate from three different input types: text prompts, images, or unstructured 3D assets. The challenge that PartGen effectively addresses is the conversion of a monolithic 3D object into a composition of parts that afford reuse, editing, and dynamic animation, similar to the workflow of human artists.

Methodology Overview

At the core of PartGen is the utilization of multi-view diffusion models. This method distinguishes itself by generating segmentations and completing occluded parts of 3D models through a two-stage diffusion pipeline. Initially, it uses a stochastic multi-view diffusion model to segment a 3D object into parts by rendering it from various viewpoints. This stage leverages datasets of artist-created 3D models that naturally incorporate a breakdown into semantic parts, providing a foundational guide that reflects artists' intents in decomposition.

The second phase leverages a generative approach to complete the segmented parts, addressing challenges such as occlusion where parts are either partially visible or invisible. Through the context offered by the entire object and leveraging the unmasked object, this phase enables the generation of plausible completions with consistent results even in the absence of complete view information.

Empirical Evaluation

Empirical results highlight PartGen's significant advances over existing methods in multi-view segmentation and part completion. The paper follows a robust evaluation protocol by utilizing mean Average Precision (mAP) to assess the quality of segmentation tasks, indicating a performance uplift over baselines, including recent segmentation methodologies such as SAM2. For part completion, the paper employs metrics like CLIP similarity, Learned Perceptual Image Patch Similarity (LPIPS), and Peak Signal-to-Noise Ratio (PSNR) to demonstrate the generative model's superior capability in achieving coherent 3D reconstructions. These numerical results substantiate PartGen's ability to reliably generate and reconstruct coherent and cohesive 3D objects from various multi-view inputs.

Application and Future Implications

PartGen's framework extends to several practical applications, which include part-aware text-to-3D generation, image-to-3D templating and practical 3D editing of objects by leveraging textual commands for aesthetic and functional modifications. These applications underscore PartGen's potential utility in creative industries, gaming, virtual reality, and automated manufacturing systems where complex objects can be selectively edited, animated, or replaced dynamically for better usability and realism.

Furthermore, the implications of this work stretch into broader domains of 3D understanding, such as robotic manipulation and interaction with object parts, where recognizing and modifying specific components of an object is crucial. The generative nature of PartGen for parts completion suggests it could inspire innovations in 3D reconstruction pipelines that handle occlusions, enhancing robotic vision systems and other AI-driven 3D applications.

Conclusion

The paper presents a sophisticated approach to 3D object modeling that introduces substantial improvements in the precise and context-aware generation of part-based models. While the work demonstrates a significant stride in bridging the gap between unstructured 3D assets and structured outputs needed for professional use, the extension towards scene-level or more complex 3D environments remains a pivotal direction for expansion. As the domain of 3D modeling continues to grow, methodologies like PartGen will significantly contribute to the evolving narrative of AI-driven creativity and functionality in artificial intelligence.