Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views (2404.03650v1)

Published 4 Apr 2024 in cs.CV

Abstract: Large visual-LLMs (VLMs), like CLIP, enable open-set image segmentation to segment arbitrary concepts from an image in a zero-shot manner. This goes beyond the traditional closed-set assumption, i.e., where models can only segment classes from a pre-defined training set. More recently, first works on open-set segmentation in 3D scenes have appeared in the literature. These methods are heavily influenced by closed-set 3D convolutional approaches that process point clouds or polygon meshes. However, these 3D scene representations do not align well with the image-based nature of the visual-LLMs. Indeed, point cloud and 3D meshes typically have a lower resolution than images and the reconstructed 3D scene geometry might not project well to the underlying 2D image sequences used to compute pixel-aligned CLIP features. To address these challenges, we propose OpenNeRF which naturally operates on posed images and directly encodes the VLM features within the NeRF. This is similar in spirit to LERF, however our work shows that using pixel-wise VLM features (instead of global CLIP features) results in an overall less complex architecture without the need for additional DINO regularization. Our OpenNeRF further leverages NeRF's ability to render novel views and extract open-set VLM features from areas that are not well observed in the initial posed images. For 3D point cloud segmentation on the Replica dataset, OpenNeRF outperforms recent open-vocabulary methods such as LERF and OpenScene by at least +4.9 mIoU.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Do as I Can, Not as I Say: Grounding Language in Robotic Affordances. arXiv preprint arXiv:2204.01691, 2022.
  2. Joint 2D-3D-Semantic Data for Indoor Scene Understanding. International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  3. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  4. Arkitscenes: A diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. neurips, 2021.
  5. 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  6. ScanNet: Richly-annotated 3d reconstructions of indoor scenes. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2017a.
  7. BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration. ACM Transactions on Graphics 2017 (TOG), 2017b.
  8. Scenefun3d: Fine-grained functionality and affordance understanding in 3d scenes. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  9. E Knuth Donald et al. The Art of Computer Programming. Sorting and searching, 1999.
  10. Scaling Open-Vocabulary Image Segmentation with Image-Level Labels. In European Conference on Computer Vision (ECCV), 2022.
  11. Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models. In Conference on Robot Learning (CoRL), 2022.
  12. OccuSeg: Occupancy-Aware 3D Instance Segmentation. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  13. VMNet: Voxel-Mesh Network for Geodesic-Aware 3D Semantic Segmentation. In International Conference on Computer Vision (ICCV), 2021.
  14. ConceptFusion: Open-Set Multimodal 3D Mapping. Robotics: Science and Systems (RSS), 2023.
  15. Scaling up Visual and Vision-Language Representation Learning with Noisy Text Supervision. In International Conference on Machine Learning (ICML), 2021.
  16. LERF: Language Embedded Radiance Fields. In International Conference on Computer Vision (ICCV), 2023.
  17. Decomposing NeRF for Editing via Feature Field Distillation. International Conference on Neural Information Processing Systems (NeurIPS), 2022.
  18. 4D-STOP: Panoptic Segmentation of 4D Lidar using Spatio-Temporal Object Proposal Generation and Aggregation. In European Conference on Computer Vision (ECCV) Workshops, 2022.
  19. Panoptic neural fields: A semantic object-aware neural scene representation. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  20. Marc Levoy. Efficient ray tracing of volume data. ACM Transactions on Graphics (TOG), 1990.
  21. Language-Driven Semantic Segmentation. International Conference on Learning Representations (ICLR), 2022a.
  22. Panoptic-phnet: Towards real-time and high-precision lidar panoptic segmentation via clustering pseudo heatmap. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
  23. Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP. International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  24. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
  25. UNISURF: unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In International Conference on Computer Vision (ICCV), 2021.
  26. OpenScene: 3D Scene Understanding with Open Vocabularies. International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  27. Pointnet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. International Conference on Neural Information Processing Systems (NeurIPS), 2017.
  28. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv:2109.08238, 2021.
  29. DenseClip: Language-Guided Dense Prediction with Context-Aware Prompting. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  30. Language-Grounded Indoor 3D Semantic Segmentation in the Wild. In European Conference on Computer Vision (ECCV), 2022.
  31. Mask3D: Mask Transformer for 3D Instance Segmentation. International Conference on Robotics and Automation (ICRA), 2023.
  32. Panoptic Lifting for 3D Scene Understanding with Neural Fields. International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  33. The Replica Dataset: A Digital Replica of Indoor Spaces. arXiv preprint arXiv:1906.05797, 2019.
  34. OpenMask3D: Open-Vocabulary 3D Instance Segmentation. In International Conference on Neural Information Processing Systems (NeurIPS), 2023a.
  35. 3D Segmentation of Humans in Point Clouds with Synthetic Data. In International Conference on Computer Vision (ICCV), 2023b.
  36. Fourier features let networks learn high frequency functions in low dimensional domains. In International Conference on Neural Information Processing Systems (NeurIPS), 2020.
  37. Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representations. In International Conference on 3D Vision (3DV), 2022.
  38. CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  39. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In International Conference on Neural Information Processing Systems (NeurIPS), 2021.
  40. Samuel S Wilks. Certain Generalizations in the Analysis of Variance. Biometrika, 1932.
  41. Volume rendering of neural implicit surfaces. In International Conference on Neural Information Processing Systems (NeurIPS), 2021.
  42. Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. In International Conference on Neural Information Processing Systems (NeurIPS), 2022.
  43. Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  44. In-place Scene Labeling and Understanding with Implicit Scene Representation. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  45. Nice-SLAM: Neural Implicit Scalable Encoding for SLAM. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  46. ICGNet: A Unified Approach for Instance-Centric Grasping. In International Conference on Robotics and Automation (ICRA), 2024.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Francis Engelmann (37 papers)
  2. Fabian Manhardt (41 papers)
  3. Michael Niemeyer (29 papers)
  4. Keisuke Tateno (12 papers)
  5. Marc Pollefeys (230 papers)
  6. Federico Tombari (214 papers)
Citations (15)
View on Semantic Scholar

Summary

OpenNeRF: Advancements in Open Set 3D Neural Scene Segmentation

The paper "OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views" introduces a method aimed at improving the capabilities of 3D scene segmentation through an innovative approach that leverages neural radiance fields (NeRF) in conjunction with pixel-aligned visual-LLM (VLM) features. This research provides a meaningful contribution to the field of open-set 3D scene understanding by addressing the constraints associated with traditional closed-set models and existing open vocabulary methods.

Core Contributions

The authors propose OpenNeRF, a novel approach that integrates pixel-wise VLM features within NeRF. This method is notably distinct from previous techniques like LERF, which rely on global CLIP features. By focusing on pixel-level detail, OpenNeRF enhances the precision of semantic segmentation without the architectural complexities introduced by other regularization strategies, such as DINO.

OpenNeRF's design leverages the inherent strengths of NeRF, particularly its capacity to render novel views. This ability is utilized to extract VLM features from areas inadequately represented in the initial dataset of posed images. The ability to generate novel views is harnessed through a probabilistic mechanism that intelligently determines which areas of the scene necessitate additional camera perspectives, thereby refining the segmentation process.

Evaluation and Results

The paper presents empirical evidence showcasing that OpenNeRF achieves a significant improvement over other methods like LERF and OpenScene in 3D point cloud segmentation tasks. Specifically, on the Replica dataset, OpenNeRF surpasses recent open-vocabulary methods by at least a 4.9 point increase in mean Intersection over Union (mIoU). This substantial metric indicates better performance in terms of accuracy and consistency in segmenting arbitrary, open-set concepts within 3D scenes.

Implications and Future Directions

OpenNeRF's success implies considerable potential for applications in augmented reality (AR), virtual reality (VR), robotic perception, and autonomous driving—domains where a fine-grained understanding of complex environments is essential. The framework's open-set approach facilitates adaptation to novel semantic classes, which is crucial for systems that operate in dynamic and unstructured environments.

For theoretical exploration, the integration of pixel-aligned VLM features with NeRF could pave the way for more sophisticated representations of three-dimensional spaces, enabling advancements in a variety of machine perception tasks. Future research may focus on the exploration of NeRF’s capabilities in handling more diverse and larger-scale datasets, optimizing the rendering of novel views, and examining the use of other types of embeddings in enhancing the semantic understanding of scenes.

The paper presents a meaningful step forward in open-set 3D scene segmentation and offers a foundation for further innovation in neural scene representation technologies. Its contributions hold promise for both enhancing existing systems and inspiring new methodologies within the broader computational and perceptual research communities.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown