OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding (2406.02058v2)

Abstract: This paper introduces OpenGaussian, a method based on 3D Gaussian Splatting (3DGS) capable of 3D point-level open vocabulary understanding. Our primary motivation stems from observing that existing 3DGS-based open vocabulary methods mainly focus on 2D pixel-level parsing. These methods struggle with 3D point-level tasks due to weak feature expressiveness and inaccurate 2D-3D feature associations. To ensure robust feature presentation and 3D point-level understanding, we first employ SAM masks without cross-frame associations to train instance features with 3D consistency. These features exhibit both intra-object consistency and inter-object distinction. Then, we propose a two-stage codebook to discretize these features from coarse to fine levels. At the coarse level, we consider the positional information of 3D points to achieve location-based clustering, which is then refined at the fine level. Finally, we introduce an instance-level 3D-2D feature association method that links 3D points to 2D masks, which are further associated with 2D CLIP features. Extensive experiments, including open vocabulary-based 3D object selection, 3D point cloud understanding, click-based 3D object selection, and ablation studies, demonstrate the effectiveness of our proposed method. The source code is available at our project page: https://3d-aigc.github.io/OpenGaussian

• The paper introduces a 3D point-level framework that leverages Gaussian splatting and SAM masks to enforce intra-object consistency and inter-object distinction.
• It employs a two-level codebook discretization to refine features from coarse positional clustering to fine-grained instance representation.
• It establishes a robust 3D–2D feature association by linking lossless CLIP features to 3D points, outperforming state-of-the-art methods in object and point cloud tasks.

OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding

Overview

The paper "OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understanding" introduces an innovative 3D point-level understanding framework utilizing 3D Gaussian Splatting (3DGS). The primary challenge addressed by this work is the weak feature discrimination and inaccurate 2D-3D feature associations in existing 3DGS-based open vocabulary methods, which predominantly focus on 2D pixel-level parsing.

Core Contributions

1. 3D Point-Level Consistency and Distinction: The authors propose training instance features that maintain 3D consistency using SAM masks. This technique ensures both intra-object consistency and inter-object distinction by employing an intra-mask smoothing loss and an inter-mask contrastive loss, thereby enhancing feature expressiveness at the 3D point level.
2. Two-Level Codebook Discretization: A novel two-level codebook approach is proposed to discretize the instance features from coarse to fine levels. At the coarse level, positional information of 3D points is utilized for location-based clustering, which is refined at the fine level. This method effectively enhances the distinctiveness and granularity of the 3D features.
3. Instance-Level 3D-2D Feature Association: The paper introduces an instance-level 3D-2D feature association method that links 3D points to 2D masks. These 2D masks are further associated with high-dimensional, lossless 2D CLIP features, thus enabling robust open vocabulary capabilities without additional compression or quantization networks.

Experimental Validation

Extensive experiments demonstrate the efficacy of the proposed method across various 3D scene understanding tasks, including:

1. Open-Vocabulary 3D Object Selection: The OpenGaussian approach outperforms the state-of-the-art methods such as LangSplat and LEGaussians in identifying 3D objects corresponding to text queries. Notably, the method achieves significant improvements in metrics like mIoU and accuracy, as evidenced by experiments on the LERF dataset.
2. 3D Point Cloud Understanding: The proposed method also excels in open-vocabulary point cloud understanding tasks, substantially surpassing LangSplat and LEGaussians on the ScanNetv2 dataset in both mIoU and accuracy, particularly in sparse scenarios where other methods struggle due to their reliance on dense point representations.
3. Click-based 3D Object Selection: OpenGaussian showcases superior performance in click-based 3D object selection tasks compared to methods like SAGA, which relies on additional post-processing steps. The results highlight the completeness and accuracy of object selection achieved by OpenGaussian without needing extra inference post-processing.

Implications and Future Directions

Practical Implications

1. Robotics and Embodied AI: The robust 3D point-level understanding facilitated by OpenGaussian can significantly enhance robotics and embodied AI applications by providing precise localization, interaction capabilities, and 3D scene comprehension.
2. Interactive 3D Systems: The method's ability to accurately select and manipulate 3D objects based on natural language queries or direct interactions could be pivotal for developing advanced AR/VR systems and interactive design tools.

Theoretical Implications

1. Feature Learning: The intra-mask smoothing loss and inter-mask contrastive loss contribute to the body of knowledge on feature learning by providing effective means to ensure feature consistency and distinction across objects within 3D space.
2. Cross-Modal Associations: The instance-level 3D-2D feature association method offers a novel paradigm for establishing robust connections between high-dimensional 2D features and 3D representations, potentially influencing future research on multimodal learning and integration.

Speculation on Future AI Developments

1. Enhanced Scene Understanding: Future research could explore extending the OpenGaussian framework to dynamic scenes and moving objects, thereby enabling real-time 3D scene understanding in more complex environments.
2. Integration with Other Modalities: Integrating audio or haptic feedback with the 3D understanding capabilities of OpenGaussian could unlock new possibilities in multimodal AI systems, potentially leading to more immersive and intuitive human-computer interactions.

Conclusion

The paper presents a method addressing critical limitations in existing 3DGS-based open vocabulary approaches, demonstrating significant advancements in point-level 3D scene understanding. Through innovative feature learning techniques and efficient 3D-2D associations, OpenGaussian sets a new standard for 3D point-level open vocabulary understanding, showcasing its potential across various applications and laying the groundwork for future research in this field.