OVGaussian: Generalizable 3D Gaussian Segmentation with Open Vocabularies (2501.00326v1)

Abstract: Open-vocabulary scene understanding using 3D Gaussian (3DGS) representations has garnered considerable attention. However, existing methods mostly lift knowledge from large 2D vision models into 3DGS on a scene-by-scene basis, restricting the capabilities of open-vocabulary querying within their training scenes so that lacking the generalizability to novel scenes. In this work, we propose \textbf{OVGaussian}, a generalizable \textbf{O}pen-\textbf{V}ocabulary 3D semantic segmentation framework based on the 3D \textbf{Gaussian} representation. We first construct a large-scale 3D scene dataset based on 3DGS, dubbed \textbf{SegGaussian}, which provides detailed semantic and instance annotations for both Gaussian points and multi-view images. To promote semantic generalization across scenes, we introduce Generalizable Semantic Rasterization (GSR), which leverages a 3D neural network to learn and predict the semantic property for each 3D Gaussian point, where the semantic property can be rendered as multi-view consistent 2D semantic maps. In the next, we propose a Cross-modal Consistency Learning (CCL) framework that utilizes open-vocabulary annotations of 2D images and 3D Gaussians within SegGaussian to train the 3D neural network capable of open-vocabulary semantic segmentation across Gaussian-based 3D scenes. Experimental results demonstrate that OVGaussian significantly outperforms baseline methods, exhibiting robust cross-scene, cross-domain, and novel-view generalization capabilities. Code and the SegGaussian dataset will be released. (https://github.com/runnanchen/OVGaussian).

• The paper introduces OVGaussian, a novel framework using 3D Gaussian representations and cross-modal consistency to achieve open-vocabulary semantic segmentation.
• The paper leverages innovative methods like Generalizable Semantic Rasterization (GSR) and Cross-modal Consistency Learning (CCL) alongside a custom SegGaussian dataset.
• The paper demonstrates significant improvements in segmentation accuracy across scenes, domains, and novel views, indicating strong potential for real-world applications.

Overview of "OVGaussian: Generalizable 3D Gaussian Segmentation with Open Vocabularies"

The paper introduces "OVGaussian," a framework for 3D Gaussian-based semantic segmentation capable of open-vocabulary understanding across diverse scenes. This approach addresses limitations in existing methods which predominantly transfer 2D vision model knowledge to 3D Gaussian representations on a scene-specific basis, restricting their ability to generalize to novel 3D environments. OVGaussian overcomes these challenges by enabling cross-scene, open-vocabulary 3D semantic segmentation.

Key Contributions

1. SegGaussian Dataset Construction: The authors construct SegGaussian, a comprehensive dataset comprising 288 3D scenes, each annotated with semantic and instance labels for both Gaussian points and multi-view images. This dataset supports detailed evaluation and training for semantic segmentation tasks.
2. Generalizable Semantic Rasterization (GSR): The core innovation of OVGaussian is the introduction of the GSR method. This technique employs a 3D neural network to predict semantic properties for each Gaussian point. These properties are consistent across multiple views and can be rendered into 2D semantic maps, facilitating cross-scene generalization.
3. Cross-modal Consistency Learning (CCL): A novel CCL framework aligns the semantic vectors of 3D Gaussians with open-vocabulary text embeddings, supplementing it with dense 2D visual-semantic information via CLIP’s visual encoder. This alignment ensures consistent and coherent semantic understanding across modalities, enhancing the model's ability to generalize to unseen categories and domains.

Experimental Results and Analysis

The paper presents experimental evidence indicating that OVGaussian significantly outperforms baseline methods across multiple evaluation metrics. Notably, the framework demonstrates robust generalization capabilities across scenes, domains, and novel view angles. Specific metrics highlighted include superior results in Cross-scene Accuracy (CSA), Open-vocabulary Accuracy (OVA), Novel View Accuracy (NVA), and Cross-Domain Accuracy (CDA).

1. Cross-Scene Generalization: The integration of GSR allows OVGaussian to maintain semantic consistency across varying scenes, achieving higher segmentation accuracy compared to methods like OpenScene and LangSplat. This is evidenced by improvements in mIoU scores for cross-scene evaluation scenarios.
2. Open-Vocabulary Segmentation: The CCL framework equips the model to handle unseen semantic categories effectively, showcased by its performance improvements in open-vocabulary benchmarks. OVGaussian's ability to segment novel classes unseen during training emphasizes its capability in leveraging pre-trained vision-LLMs.
3. Cross-Domain and Novel View Performance: With significant advancement in CDA and NVA, OVGaussian demonstrates adaptability to data from differing domains and various viewpoints, reinforcing its effectiveness in real-world applications.

Implications and Future Directions

The research presents promising implications for applications in 3D scene understanding, robotics, and autonomous navigation, where open-world interaction necessitates the ability to generalize across new environments and categories. The deployment of OVGaussian in practical scenarios could lead to more adaptive and intelligent systems capable of real-time 3D semantic processing.

Going forward, potential areas of improvement include refining the scalability of the framework to handle larger, more complex scenes as well as further exploring self-supervised learning techniques to reduce dependency on densely annotated datasets. Additionally, enhancing computational efficiency in rendering might facilitate broader real-time applications, making OVGaussian a viable solution for diverse industrial applications.