PartSLIP++: Enhancing Low-Shot 3D Part Segmentation via Multi-View Instance Segmentation and Maximum Likelihood Estimation (2312.03015v1)

Abstract: Open-world 3D part segmentation is pivotal in diverse applications such as robotics and AR/VR. Traditional supervised methods often grapple with limited 3D data availability and struggle to generalize to unseen object categories. PartSLIP, a recent advancement, has made significant strides in zero- and few-shot 3D part segmentation. This is achieved by harnessing the capabilities of the 2D open-vocabulary detection module, GLIP, and introducing a heuristic method for converting and lifting multi-view 2D bounding box predictions into 3D segmentation masks. In this paper, we introduce PartSLIP++, an enhanced version designed to overcome the limitations of its predecessor. Our approach incorporates two major improvements. First, we utilize a pre-trained 2D segmentation model, SAM, to produce pixel-wise 2D segmentations, yielding more precise and accurate annotations than the 2D bounding boxes used in PartSLIP. Second, PartSLIP++ replaces the heuristic 3D conversion process with an innovative modified Expectation-Maximization algorithm. This algorithm conceptualizes 3D instance segmentation as unobserved latent variables, and then iteratively refines them through an alternating process of 2D-3D matching and optimization with gradient descent. Through extensive evaluations, we show that PartSLIP++ demonstrates better performance over PartSLIP in both low-shot 3D semantic and instance-based object part segmentation tasks. Code released at https://github.com/zyc00/PartSLIP2.

The paper "PartSLIP++: Enhancing Low-Shot 3D Part Segmentation via Multi-View Instance Segmentation and Maximum Likelihood Estimation" addresses the critical challenge of open-world 3D part segmentation, particularly relevant in fields like robotics and augmented/virtual reality (AR/VR). Traditional supervised 3D part segmentation methods often struggle with limited data and poor generalization to unseen categories. PartSLIP presented an innovative solution using 2D open-vocabulary detection methods and a heuristic technique for transforming multi-view 2D bounding box predictions into 3D segmentation masks.

PartSLIP++, the enhanced version, integrates two significant improvements that deepen the capabilities and accuracy of 3D part segmentation:

1. Advanced 2D Segmentation with SAM:
   • PartSLIP++ employs a pre-trained 2D segmentation model called SAM to produce pixel-wise 2D segmentations. This is an improvement over the 2D bounding boxes used in PartSLIP, resulting in more precise and accurate annotations. Pixel-wise segmentation helps in capturing finer details, which is crucial for effective part segmentation.
2. Expectation-Maximization for 3D Conversion:
   • To refine the process of converting 2D segmentations to 3D masks, PartSLIP++ introduces a modified Expectation-Maximization (EM) algorithm. Unlike the heuristic method in PartSLIP, the EM algorithm treats 3D instance segmentation as unobserved latent variables. It iteratively refines these through an alternating process of 2D-3D matching and gradient descent optimization. This probabilistic approach ensures a more systematic and refined convergence to accurate 3D segmentations.

The paper presents extensive evaluations demonstrating that PartSLIP++ surpasses PartSLIP in low-shot 3D semantic and instance-based object part segmentation tasks. By combining more detailed 2D segmentation with a sophisticated probabilistic method for 3D conversion, PartSLIP++ makes substantial advancements in addressing the challenges of limited data and generalization in 3D part segmentation.

The authors have also released the code for PartSLIP++ at https://github.com/zyc00/PartSLIP2, which facilitates further research and implementation in relevant applications. This contribution is notable for enhancing the precision and effectiveness of 3D part segmentation in settings with scarce training data and unseen object categories.