Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models (2405.19326v1)

Abstract: In this paper, we introduce a new task: Zero-Shot 3D Reasoning Segmentation for parts searching and localization for objects, which is a new paradigm to 3D segmentation that transcends limitations for previous category-specific 3D semantic segmentation, 3D instance segmentation, and open-vocabulary 3D segmentation. We design a simple baseline method, Reasoning3D, with the capability to understand and execute complex commands for (fine-grained) segmenting specific parts for 3D meshes with contextual awareness and reasoned answers for interactive segmentation. Specifically, Reasoning3D leverages an off-the-shelf pre-trained 2D segmentation network, powered by LLMs, to interpret user input queries in a zero-shot manner. Previous research have shown that extensive pre-training endows foundation models with prior world knowledge, enabling them to comprehend complex commands, a capability we can harness to "segment anything" in 3D with limited 3D datasets (source efficient). Experimentation reveals that our approach is generalizable and can effectively localize and highlight parts of 3D objects (in 3D mesh) based on implicit textual queries, including these articulated 3d objects and real-world scanned data. Our method can also generate natural language explanations corresponding to these 3D models and the decomposition. Moreover, our training-free approach allows rapid deployment and serves as a viable universal baseline for future research of part-level 3d (semantic) object understanding in various fields including robotics, object manipulation, part assembly, autonomous driving applications, augment reality and virtual reality (AR/VR), and medical applications. The code, the model weight, the deployment guide, and the evaluation protocol are: http://tianrun-chen.github.io/Reason3D/

Reasoning3D: Grounding and Reasoning in 3D

The paper "Reasoning3D - Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-LLMs," authored by Chen et al., presents a novel task in the domain of 3D segmentation. The primary objective of this research is to advance 3D segmentation techniques through Zero-Shot Reasoning Segmentation for parts within 3D objects based on fine-grained contextual understanding facilitated by Large Vision-LLMs (LVLM).

Methodology and Approach

Reasoning3D introduces a new paradigm for 3D segmentation that outperforms traditional methods reliant on extensive manual labeling or rigid rule-based algorithms. The approach leverages pre-trained 2D segmentation networks and LVLMs to interpret and execute complex commands for segmenting specific parts of 3D meshes without additional training. This is achieved through a multi-view rendering process where viewpoints are converted into 2D images, segmented using a pre-trained 2D reasoning segmentation network powered by LVLMs, and then fused back into the 3D space using a specially designed multi-stage fusion and refinement mechanism.

The segmentation process acknowledges the importance of both visual and linguistic input, utilizing embeddings from both to produce segmentation masks and natural language explanations. Specifically, the methodology involves:

1. Multi-View Image Rendering and Face ID Generation: The 3D model is rendered from various viewpoints to generate 2D images with corresponding face IDs, forming a mapping matrix that ensures accurate alignment between 2D images and the original 3D mesh.
2. Reasoning and Segmenting with User Input Prompt: User-input prompts are processed by a multimodal LLM, generating textual responses and segmentation masks that capture the intended parts of the 3D model.
3. Mask Fusion and Refinement in 3D: The segmented 2D masks are fused onto the 3D mesh using Gaussian Geodesic Reweighting, Visibility Smoothing, and Global Filtering Strategy to produce coherent and high-quality segmentation results in 3D space.

Experimental Validation

The effectiveness of Reasoning3D was evaluated using the FAUST benchmark for open-vocabulary 3D segmentation and a custom dataset of in-the-wild 3D models collected from SketchFab. The results demonstrated competitive performance in open-vocabulary segmentation compared to state-of-the-art methods such as SATR and 3DHighlighter. The method's capability in reasoning-based segmentation was qualitatively assessed through user input of implicit segmentation commands, confirming its utility in real-world applications.

Performance Metrics

• Mean Intersection over Union (mIoU): Utilized to quantify segmentation accuracy across different semantic categories and shapes in the FAUST dataset.
• Qualitative User Feedback: Used to assess the reasoning-based segmentation task, highlighting the system's ability to handle complex, implicit queries effectively.

Discussion and Implications

While Reasoning3D presents a robust foundation for future research and development in 3D part segmentation, several areas warrant further exploration. The need for comprehensive benchmarks and user studies is emphasized to validate the approach's practical applicability. Additionally, the integration of view selection strategies aligned with the pre-trained vision encoder could further enhance performance.

The implications of this research extend across multiple domains, including robotics, AR/VR, autonomous driving, and medical applications. By providing a training-free, zero-shot inference method, Reasoning3D facilitates rapid deployment and practical utilization, marking a significant milestone in the evolution of 3D segmentation techniques.

Future Directions

Future research could focus on optimizing view selection to maximize the potential of LVLMs and explore fine-tuning with larger datasets to balance generalization and specificity. Moreover, adapting the multi-view 2D segmentation and 3D projection method for scene-based contexts could unlock new applications and improve interaction dynamics in 3D environments.

Conclusion

Reasoning3D represents a pivotal step in 3D segmentation, harnessing the advanced capabilities of LVLMs to deliver nuanced, reasoning-based segmentation results with minimal data overhead. By bridging the gap between 2D pre-training and 3D real-world applications, it opens new avenues for innovation and practical implementation across diverse fields. The open-sourced code and resources aim to foster collaborative progress, positioning Reasoning3D as a foundational tool for advancing 3D computer vision.

The code and related resources for Reasoning3D can be accessed at: Reason3D Project Page.