3x2: 3D Object Part Segmentation by 2D Semantic Correspondences (2407.09648v1)

Abstract: 3D object part segmentation is essential in computer vision applications. While substantial progress has been made in 2D object part segmentation, the 3D counterpart has received less attention, in part due to the scarcity of annotated 3D datasets, which are expensive to collect. In this work, we propose to leverage a few annotated 3D shapes or richly annotated 2D datasets to perform 3D object part segmentation. We present our novel approach, termed 3-By-2 that achieves SOTA performance on different benchmarks with various granularity levels. By using features from pretrained foundation models and exploiting semantic and geometric correspondences, we are able to overcome the challenges of limited 3D annotations. Our approach leverages available 2D labels, enabling effective 3D object part segmentation. Our method 3-By-2 can accommodate various part taxonomies and granularities, demonstrating interesting part label transfer ability across different object categories. Project website: \url{https://ngailapdi.github.io/projects/3by2/}.

• The paper presents the 3-By-2 approach that uses 2D semantic correspondences from pre-trained models for effective 3D part segmentation.
• It introduces a mask-consistency module that aggregates multi-view 2D predictions into coherent, high-fidelity 3D segmentation without extra training.
• Experimental results on datasets like PartNet demonstrate significant improvements in zero- and few-shot settings over traditional methods.

3-by-2: 3D Object Part Segmentation by 2D Semantic Correspondences

In the paper titled "3-by-2: 3D Object Part Segmentation by 2D Semantic Correspondences," the authors introduce a novel, training-free method for 3D object part segmentation called "3-By-2." This method harnesses the power of 2D semantic correspondences derived from feature representations of pretrained foundation models, achieving state-of-the-art (SOTA) performance on various low-shot segmentation benchmarks. This paper addresses the challenges faced in 3D part segmentation, particularly the high cost and scarcity of annotated 3D datasets, by leveraging richly annotated 2D datasets to transfer part labels to 3D objects.

Method Overview

The paper details the 3-By-2 method which consists of three primary steps: 1) rendering multiple 2D views of a 3D object, 2) performing 2D part segmentation on each view using semantic correspondences, and 3) aggregating the 2D predictions into a coherent 3D segmentation using a mask-consistency module. The core innovation lies in utilizing features from image diffusion models and integrating these with a class-agnostic segmentation model like SAM (Segment Anything Model) to achieve precise part label transfer.

Key Contributions

1. Training-Free Methodology: The 3-By-2 method eliminates the need for extensive labeled 3D training data by leveraging 2D annotated datasets, significantly reducing annotation costs and complexities involved in traditional 3D segmentation tasks.
2. Non-Overlapping Mask Generation: The authors propose a method to generate non-overlapping 2D masks, refining the output of SAM to more accurately reflect part boundaries and improve segmentation fidelity.
3. Mask-Level Label Transfer and Consistency: By transferring labels at the mask level and enforcing consistency across multiple views, the method ensures high-quality segmentation across various parts and object categories.

Experimental Analysis

The paper provides a comprehensive evaluation of the 3-By-2 method on multiple datasets including PartNet-Ensembled (PartNetE) and PartNet with level-3 annotations. The results demonstrate that 3-By-2 achieves superior performance on both zero-shot and few-shot settings compared to existing methods.

• Few-Shot Setting: On PartNetE, 3-By-2 achieved an average mIoU of 0.642 across 45 categories, outperforming both fully-supervised and few-shot baseline methods. Specifically, it improved performance by up to 10% on certain categories compared to fully-supervised methods.
• Zero-Shot Setting: Using the PACO dataset for 2D labels, the method showed substantial improvements over other baselines like PartSLIP and SAMPro3D, achieving a notable performance boost on challenging categories with fine-grained annotations.
• PartNet with Level-3 Annotations: The method demonstrated competitiveness with MvDeCor, a model pretrained and finetuned on PartNet data, emphasizing the robustness and flexibility of 3-By-2 in handling highly granular part annotations without additional training.

Theoretical and Practical Implications

The paper showcases the effectiveness of leveraging 2D semantic correspondences for 3D segmentation tasks, shedding light on the broader applicability of 2D vision models in 3D contexts. The flexibility of the 3-By-2 method in handling various part taxonomies and finely-grained segmentation tasks highlights a significant advance in the field.

Practically, this approach can be highly beneficial in domains where collecting 3D annotations is prohibitively expensive or logistically challenging, such as robotics, AR/VR, and graphics. The ability to perform accurate 3D part segmentation using abundantly available 2D data opens new avenues for rapid prototyping and deployment in these applications.

Future Directions

Future research may focus on optimizing the feature extraction and mask generation components to further enhance performance. The application of 3-By-2 to dynamic or deformable objects, as well as exploring transfer learning capabilities across even more diverse object categories, could provide additional insights into the robustness and scalability of the method.

Conclusion

The 3-By-2 method represents a significant step forward in 3D object part segmentation by innovatively leveraging 2D annotated datasets. Its training-free nature, combined with the robust label transfer and aggregation mechanisms, positions this method as a highly effective tool for a wide range of 3D vision applications.