Grounded 3D-LLM with Referent Tokens (2405.10370v2)

Abstract: Prior studies on 3D scene understanding have primarily developed specialized models for specific tasks or required task-specific fine-tuning. In this study, we propose Grounded 3D-LLM, which explores the potential of 3D large multi-modal models (3D LMMs) to consolidate various 3D vision tasks within a unified generative framework. The model uses scene referent tokens as special noun phrases to reference 3D scenes, enabling it to handle sequences that interleave 3D and textual data. Per-task instruction-following templates are employed to ensure natural and diversity in translating 3D vision tasks into language formats. To facilitate the use of referent tokens in subsequent LLMing, we provide a large-scale, automatically curated grounded scene-text dataset with over 1 million phrase-to-region correspondences and introduce Contrastive Language-Scene Pre-training (CLASP) to perform phrase-level scene-text alignment using this data. Our comprehensive evaluation covers open-ended tasks like dense captioning and 3D question answering, alongside close-ended tasks such as object detection and language grounding. Experiments across multiple 3D benchmarks reveal the leading performance and the broad applicability of Grounded 3D-LLM. Code and datasets are available at the https://groundedsceneLLM.github.io/grounded_3d-LLM.github.io.

Grounded 3D-LLM: A Unified Framework for 3D Scene Understanding

The paper "Grounded 3D-LLM" introduces an innovative approach to 3D scene understanding by proposing a unified generative framework. This framework leverages grounded phrase-level LLMing to consolidate various 3D vision tasks. By integrating scene referent tokens into LLMs, the model aims to perform tasks such as object detection, visualization grounding, and 3D QA without task-specific fine-tuning. I will provide a detailed overview of the methodology, the dataset generation, the empirical results, and the implications for future AI developments.

Methodology

The Grounded 3D-LLM model is constructed to address the limitations of existing 3D vision models, which are typically specialized for specific tasks. The core innovation lies in using referent tokens, denoted <ref>, to represent scene regions or object features as special noun phrases. To establish effective scene-text alignment, the paper introduces the Contrastive LAnguage-Scene Pre-training (CLASP) framework. This method:

1. Extracts point-level embeddings through a sparse convolutional network.
2. Employs a cross-modal interactor to couple text embeddings from BERT with visual representations.
3. Utilizes learnable queries as proxies to connect textual phrases with raw 3D point clouds.

Technical enhancements like these ensure phrase-level alignment between natural language and visual scenes, which facilitates multiple downstream tasks within a unified framework. The LLMing capability is extended using instruction templates that transform existing datasets into task-specific instructions, thus eliminating the necessity for independent detectors or task-specific tuning.

Dataset Generation

To facilitate the proposed model, the paper presents the Grounded Scene Caption (G-SceneCap) dataset. This dataset provides fine-grained scene-text correspondence necessary for phrase-level grounding. The G-SceneCap dataset was generated through a pipeline that combines:

1. Object captions derived from dense object annotations and refined using visual and textual models.
2. Condensed scene captions using GPT-4, integrating related spatial relationships programmatically.

Apart from G-SceneCap, the model utilizes transformed existing datasets like Grounded ScanRefer and Grounded Multi3DRef for broader generalization. This extensive dataset amalgamation ensures comprehensive pre-training and evaluation coverage across multiple 3D vision tasks.

Empirical Results

Evaluations demonstrate the model's superior performance as follows:

• Grounding Tasks: The model outperforms previous discriminative and generative models significantly in single-object and multi-object grounding tasks. It achieves an accuracy of 47.9% at 0.25 IoU and 44.1% at 0.5 IoU on the ScanRefer grounding task.
• 3D QA and Captioning: The model also excels in language-oriented tasks, achieving the highest CIDEr score of 70.6 in Scan2Cap and a strong BLEU-4 score of 13.4 in ScanQA.
• Detection: Unique among generative models, Grounded 3D-LLM supports 3D object detection, demonstrating its versatility.

The comparison with models like 3D-LLM, Chat-3D, and LL3DA highlights the effectiveness of phrase-level alignment facilitated through CLASP. The ablative studies underscore the critical role of diverse datasets and fine-grained scene captions in elevating the model's performance.

Implications and Future Directions

Grounded 3D-LLM opens the pathway for creating comprehensive 3D multi-modal models that can generalize across numerous tasks without the need for specialized architectures. This unified approach is particularly relevant for applications in VR/AR, robotics, interactive embodied agents, and autonomous navigation, where multifunctional understanding and interaction with 3D environments are crucial.

Future developments may explore:

1. Scaling the dataset to cover more diverse environments and objects, enhancing the model's robustness and adaptability.
2. Extending the model to incorporate dynamic environments where objects and entities are in motion.
3. Integrating more sophisticated reasoning capabilities to handle complex 3D scene interactions and higher-order question answering.

In summary, the Grounded 3D-LLM paper offers a significant advancement in the integration of language and 3D visual data, providing a versatile framework that bridges multiple vision tasks seamlessly. The implications for AI and robotics are profound, marking a step-forward in creating truly intelligent multi-modal systems capable of understanding and interacting with complex 3D environments.