Ground3D-LMM: Fine-Grained 3D Point Grounding and Spatial Reasoning with LMM

Abstract: Natural-language queries about 3D environments become actionable when responses are verifiable and metric. Verifiability requires explicit grounding to the referred 3D region, while metric answers report physical measurements in real-world units (e.g., size, thickness, clearance, and distance). Existing 3D large multimodal models (LMMs) approaches remain limited: conversational systems typically respond without explicit 3D grounding, while 3D grounding models are not designed for interactive, metric-aware dialogue. In this paper, we present Ground3D-LMM, a unified model that takes a point cloud and an optional RGB image as input and supports 3D spatial conversation with (i) point-grounded responses and (ii) metric numeric outputs at both object and part granularity, including multi-object queries. To evaluate this intersection of grounding and measurement, we define the 3D Grounded Measurement task, which requires predicting the referred 3D region and the corresponding metric quantities in real-world units. We introduce a large-scale dataset built on ScanNet and ScanNet++ datasets with dense object and part annotations and roughly 2.5M question-answer pairs spanning eight tasks, along with a manually verified test set. Extensive experiments on multiple datasets and tasks show that our proposed Ground3D-LMM model provides a strong baseline for grounded, metric-aware 3D conversational understanding. Our dataset and model are publicly available.

• The paper introduces Ground3D-LMM, a multimodal framework that unifies fine-grained 3D point segmentation, metric reasoning, and conversational interactions.
• It leverages a novel Ground3D dataset with over 104K annotated QA pairs and employs joint text-mask supervision to boost both segmentation accuracy and linguistic performance.
• Experimental results demonstrate state-of-the-art performance across multiple benchmarks, paving the way for advanced robotics, AR/VR, and interactive spatial applications.

Ground3D-LMM: Fine-Grained 3D Point Grounding and Spatial Reasoning with Large Multimodal Models

Introduction and Motivation

Ground3D-LMM addresses a critical challenge in vision-language research: unifying fine-grained 3D spatial understanding, metric reasoning, and conversational interaction within a single multimodal LLM (LMM) framework. While previous models have achieved partial successes—producing either 3D masks from language or textual answers about 3D scenes—they lack explicit metric grounding and a unified conversational approach that interleaves natural language, fine-grained point-level 3D segmentation, and the handling of both objects and parts. This paper formalizes these requirements, introducing the 3D Grounded Measurement task and the Ground3D dataset to comprehensively benchmark open-vocabulary grounding, metric reasoning, and interactive spatial conversation.

Ground3D Dataset Construction and Semantic Novelty

Ground3D is constructed from ~2.5K scenes sourced from ScanNet and ScanNet++, using a staged pipeline to sample informative frames, detect objects and parts with LMMs, generate fine 2D instance masks (SAM2), lift supervision to 3D point clouds, and synthesize diverse metric and relational queries at varying granularity.

Figure 1: Ground3D statistics and annotation pipeline, showing acquisition from ScanNet datasets, multi-level annotation, 2D-3D lifting, and QA synthesis for grounding and measurement tasks.

The resulting dataset includes over 104K high-quality object/part-level and scene-level QA pairs with aligned 3D masks and metric annotations. Novelty is rigorously evaluated by projecting text label embeddings (CLIP, UMAP) into semantic space, revealing that a substantial fraction of Ground3D labels extend beyond the ScanNet/ScanNet++ taxonomy, with approximately 25% falling below nearest-neighbor similarity thresholds defined by the native taxonomy. This empirically confirms semantic expansion into previously uncovered concept space.

Figure 3: Semantic embedding map (UMAP). CLIP embeddings show expansion of Ground3D labels (orange) beyond the ScanNet taxonomy (blue).

Figure 5: Nearest-neighbor cosine similarity distributions—Ground3D labels show heavier low-similarity tails compared to native taxonomy, quantifying semantic novelty.

Model Architecture: Ground3D-LMM

Ground3D-LMM integrates a voxel-based 3D point encoder (sparse 3D U-Net over superpoints) with a LMM backbone (Qwen3-VL-4B-Instruct) capable of ingesting both 3D features and optional 2D images. The model utilizes a multimodal alignment mechanism: 3D point tokens are linearly projected into the LMM’s token embedding space, with reserved special tokens in the input prompt for joint reasoning. Queries referencing objects or parts trigger a segmentation head, which applies attention-based decoding to predict binary 3D masks corresponding to grounded referents.

Figure 2: Ground3D-LMM overview: given colored point clouds (and optionally images), it produces joint textual and 3D mask outputs in a conversational interface.

The architecture supports multi-object queries, open-vocabulary metric estimation (e.g., dimensions, distances), and explicit part-level understanding—a critical gap in prior works. Training employs a joint loss (weighted sum of Dice+BCE for segmentation and autoregressive cross-entropy for text), optimizing both mask accuracy and linguistic quality.

Experimental Evaluation

Segmentation and Metric Reasoning

Ground3D-LMM is comprehensively benchmarked on Ground3D, Reason3D, and ScanRefer. Across object segmentation (mIoU), part segmentation, metric question answering (Absolute Percentage Error [APE], $\delta$-threshold success), and text quality (hallucination, completeness), Ground3D-LMM outperforms state-of-the-art models—including open-vocabulary segmentation (UniSeg3D), 3D reasoning (Reason3D), and image-based LMMs (SD-VLM)—with or without 2D image inputs.

Numerical results:

• On Ground3D-ScanNet, object-level segmentation mIoU = 42.22 (3D+2D).
• On Ground3D-ScanNet++, object-level segmentation mIoU = 30.42 (3D+2D).
• Part-level segmentation: mIoU = 39.07 (3D+2D), strong performance on fine-grained part tasks.
• Text metric APE lower (better), and $\delta$-success rates higher than prior baselines.

Cross-Benchmark Generalization

On Reason3D and ScanRefer, Ground3D-LMM achieves new state-of-the-art results:

• ScanRefer: mIoU = 41.30 (3D+2D), outperforming prior approaches by significant margins, especially with only point cloud input.
• Reason3D: mIoU = 41.29 (3D+2D), even surpassing methods leveraging larger LMMs (LISA-7B) or richer visual backbones.

Qualitative Analysis and Conversational Reasoning

Qualitative results illustrate robust performance on both spatial and metric queries, with visualizations revealing precise 3D masks anchored to natural-language phrases, accurate spatial relations, and consistent metric estimates.

Figure 4: Qualitative outputs for representative object- and part-level tasks in Ground3D-LMM.

The model demonstrates strong generalization to multi-turn, conversational 3D scenarios—accurately maintaining referential context and grounding across dialogue steps.

Figure 6: Multi-turn 3D conversational reasoning: Ground3D-LMM answers sequential questions, reusing prior groundings for context continuation.

Figure 7: Longer six-turn dialogue (unseen in training) showcasing robust context tracking and generalization.

Figure 8: Qualitative results for all object-level tasks—demonstrating robustness to query diversity and spatial complexity.

Figure 9: Qualitative results for all part-level tasks in complex scenes.

Implications and Future Directions

Ground3D-LMM enables an unprecedented degree of verifiable, metric-aware 3D spatial reasoning in conversational AI systems. The explicit point-level grounding with metric outputs is practically transformative for robotics (e.g., clearance checking, AR/VR scene interaction, assistive agents), where ambiguous textual responses are insufficient. The open-vocabulary, fine-grained part reasoning and semantic expansion support a broad range of downstream tasks beyond the capacities of models limited to static, closed vocabularies.

Several theoretical implications arise:

• Joint text-mask supervision (rather than text-only) is mutually beneficial, enhancing both segmentability and metric prediction.
• Incorporating optional 2D cues further boosts performance, suggesting a path towards robust cross-modal fusion strategies.
• Large-scale synthetic conversational annotation pipelines (as in Ground3D) provide a scalable path for future dataset generation.

Future work might explore scaling the backbone LMM, integrating richer physical simulation (affordance reasoning), supporting dynamic scenes, or extending to autonomous closed-loop applications. Evaluating robustness on diverse, out-of-distribution 3D domains remains critical for practical adoption.

Conclusion

Ground3D-LMM represents a unified step change in spatially grounded, metric-aware conversational LMMs. The model delivers end-to-end verifiable 3D outputs—pairing precise natural language, point-level segmentation, and physical measurements—grounded in a large-scale, semantically enriched dataset. Empirical results confirm substantial gains on multiple challenging 3D reasoning tasks, setting a strong benchmark for future research in grounded embodied AI and interactive spatial agents.