OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation (2309.00616v5)
Abstract: In this work, we introduce OpenIns3D, a new 3D-input-only framework for 3D open-vocabulary scene understanding. The OpenIns3D framework employs a "Mask-Snap-Lookup" scheme. The "Mask" module learns class-agnostic mask proposals in 3D point clouds, the "Snap" module generates synthetic scene-level images at multiple scales and leverages 2D vision-LLMs to extract interesting objects, and the "Lookup" module searches through the outcomes of "Snap" to assign category names to the proposed masks. This approach, yet simple, achieves state-of-the-art performance across a wide range of 3D open-vocabulary tasks, including recognition, object detection, and instance segmentation, on both indoor and outdoor datasets. Moreover, OpenIns3D facilitates effortless switching between different 2D detectors without requiring retraining. When integrated with powerful 2D open-world models, it achieves excellent results in scene understanding tasks. Furthermore, when combined with LLM-powered 2D models, OpenIns3D exhibits an impressive capability to comprehend and process highly complex text queries that demand intricate reasoning and real-world knowledge. Project page: https://zheninghuang.github.io/OpenIns3D/
- 3d semantic parsing of large-scale indoor spaces. In CVPR, 2016.
- Look around and refer: 2d synthetic semantics knowledge distillation for 3d visual grounding. In Advances in Neural Information Processing Systems, 2022.
- Language models are few-shot learners. In NeurIPS, 2020.
- End-to-end object detection with transformers. In ECCV, 2020.
- Matterport3d: Learning from rgb-d data in indoor environments. In 3DV, 2017.
- Stpls3d: A large-scale synthetic and real aerial photogrammetry 3d point cloud dataset. In BMVA, 2022.
- Clip2scene: Towards label-efficient 3d scene understanding by clip. In CVPR, 2023a.
- Open-vocabulary panoptic segmentation with embedding modulation. In ICCV, 2023b.
- Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.
- Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
- Pla: Language-driven open-vocabulary 3d scene understanding. In CVPR, 2023a.
- Lowis3d: Language-driven open-world instance-level 3d scene nderstanding, 2023b.
- Open-vocabulary panoptic segmentation with maskclip. In arXiv, 2022.
- Computer Graphics and Mathematics. Springer Verlag, 1992.
- SynthCity: A large-scale synthetic point cloud. In arXiv, 2019.
- SEMANTIC3D.NET: A new large-scale point cloud classification benchmark. In ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2017.
- Sensaturban: Learning semantics from urban-scale photogrammetric point clouds. In International booktitle of Computer Vision, 2022.
- Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In ICCV, 2023.
- Mask scoring R-CNN. In arXiv, 2019.
- Segment anything. In ICCV, 2023.
- Virtual multi-view fusion for 3d semantic segmentation. In ECCV, 2020.
- Lisa: Reasoning segmentation via large language model. In arXiv, 2023.
- Mseg: A composite dataset for multi-domain semantic segmentation. In TPAMI, 2021.
- Campus3d: A photogrammetry point cloud benchmark for hierarchical understanding of outdoor scene. In ACM MM, 2020.
- Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In arXiv, 2023.
- Open-vocabulary point-cloud object detection without 3d annotation. In CVPR, 2023.
- Generative zero-shot learning for semantic segmentation of 3D point cloud. In 3DV, 2021.
- An end-to-end transformer model for 3d object detection. In ICCV, 2021.
- PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In CVPR, 2019.
- Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023.
- Learning transferable visual models from natural language supervision. In ICML, 2021.
- Paris-lille-3d: A large and high-quality ground-truth urban point cloud dataset for automatic segmentation and classification. In The International booktitle of Robotics Research, 2018.
- Language-grounded indoor 3d semantic segmentation in the wild. In ECCV, 2022.
- Unscene3d: Unsupervised 3d instance segmentation for indoor scenes. In arXiv, 2023.
- Mask3D: Mask Transformer for 3D Semantic Instance Segmentation. In ICRA, 2023.
- Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
- Multi-view convolutional neural networks for 3d shape recognition. In ICCV, 2015.
- Openmask3d: Open-vocabulary 3d instance segmentation, 2023.
- Toronto-3D: A large-scale mobile lidar dataset for semantic segmentation of urban roadways. In CVPRW, 2020.
- Point transformer v2: Grouped vector attention and partition-based pooling. In NeurIPS, 2022.
- Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models. In CVPR, 2023.
- Regionplc: Regional point-language contrastive learning for open-world 3d scene understanding, 2023a.
- Sam3d: Segment anything in 3d scenes. In arXiv, 2023b.
- Pointclip: Point cloud understanding by CLIP. In CVPR, 2022.
- Denseclip: Extract free dense labels from clip. In arXiv, 2021.
- Extract free dense labels from clip. In ECCV, 2022.
- Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. In ICCV, 2023.
- Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2020.
- Zhening Huang (6 papers)
- Xiaoyang Wu (27 papers)
- Xi Chen (1035 papers)
- Hengshuang Zhao (117 papers)
- Lei Zhu (280 papers)
- Joan Lasenby (32 papers)