---
title: '3D-GRAND: 3D Dataset & Multi-Agent SLAM'
url: https://www.emergentmind.com/topics/3d-grand
type: topic
---

# 3D-GRAND: 3D Dataset & Multi-Agent SLAM

3D-GRAND encompasses two distinct but thematically related research contributions spanning large-scale datasets for 3D language-vision models and advanced multi-agent SLAM systems based on Gaussian splatting. In context, "3D-GRAND" refers to: (i) a million-scale dataset and benchmarking suite for instruction-tuned 3D-LLMs targeting robust 3D grounding and hallucination reduction [2406.05132]; and (ii) an abbreviation for the 3D extension of GRAND-SLAM—an architecture for scalable, globally consistent, multi-agent Gaussian SLAM [2506.18885].

## 1. 3D-GRAND: Definition and Scope

3D-GRAND, as introduced in [2406.05132], is a large-scale dataset specifically designed to bridge language and 3D perception for embodied AI. It contains 40,087 richly tagged household scenes (mainly from 3D-FRONT and Structured3D) and 6.2 million densely-grounded scene-language instruction pairs. The dataset enables instruction-tuning of 3D large language models (3D-LLMs), supporting dense and faithful grounding of noun phrases to 3D objects, and provides a rigorous evaluation benchmark (3D-POPE) for systematic hallucination assessment.

In the context of multi-agent SLAM, "3D-GRAND" (as an *Editor's term*: "Gaussian Reconstruction via Multi-Agent Dense SLAM") refers to an extension of the GRAND-SLAM framework for collaborative scene reconstruction using 3D Gaussian splatting, tailored for both indoor and large-scale outdoor environments [2506.18885].

## 2. Dataset Construction and Statistics

The 3D-GRAND dataset construction pipeline in [2406.05132] consists of several distinct stages:

- **Scene Collection:** 38,000+ scenes are drawn from the 3D-FRONT and Structured3D repositories, featuring diverse room types. Meshes/layouts are rendered into colored point clouds via multi-view fusion (Blender-based pipeline), with spatial separation enforced by layout masks.
- **Densely-grounded Annotation:** For each scene, the following procedure is executed:
  - **Object Extraction:** 2D object crops are generated; if ground truth is absent, "set-of-mark" GPT-4V prompts are used.
  - **Attribute Tagging:** Object category, color, finish, and texture are inferred using open-vocabulary GPT-4V processing.
  - **Scene-Graph Assembly:** Each object is represented as a JSON entry with (category, centroid $(x,y,z)$, extents $(w,h,d)$, attributes).
  - **Instruction Generation:** GPT-4/GPT-4-Turbo templates produce eight classes of scene-language tasks, including grounded object reference, scene description, question answering (existence, spatial, count), and multi-turn or landmark-driven variants.
  - **Filtering and Augmentation:** Low-quality or hallucinated outputs are filtered, and phrase-to-object tags are augmented via prompt-based approaches.
  - **Human Validation:** 10,200 scene-annotation pairs are validated by three raters per sample on Hive.ai ( ≥2/3 agreement); text truthfulness is 85–88% and grounding accuracy is 92–95%.

The key descriptive statistics are summarized below:

| Attribute              | Value                              |  
|------------------------|------------------------------------|  
| Scenes                 | 40,087                             |  
| Instruction pairs      | 6.20 million                       |  
| Avg. pairs per scene   | 155                                |  
| Dense grounding        | 100% of noun phrases → 3D objects  |  
| Tasks/splits           | 8 task variants                    |  
| Avg. objects per scene | 86.4                               |  
| Unique categories      | 182                                |  
| Real-scan test set     | 1,400 ScanNet scenes (zero-shot)   |

This resource aims to provide explicit, high-quality grounding information critical for instruction-tuned 3D-LLMs.

## 3. Model Architectures, Training, and Losses

The primary instruction-tuned 3D-LLM architecture described in [2406.05132] consists of:

- **Backbone:** Llama-2 (70B parameters), equipped with LoRA adapters for efficient fine-tuning.
- **Input Fusion:** Object-centric 3D scene-graph descriptions prepended as JSON tokens (e.g., `<obj id=12,cat=chair,x=1.2,...>`).
- **Query Encoding:** Instructions or questions appended to the textual context.
- **Output:** Free-form response with explicit inline grounding tags (e.g., `<ground obj=12>`).

The training objective incorporates three loss terms:
\[
\mathcal{L} = \mathcal{L}_{\text{CE}} + \lambda_{\text{ground}}\mathcal{L}_{\text{ground}} + \lambda_{\text{hall}} \mathcal{L}_{\text{hall}}
\]
- **$\mathcal{L}_{\mathrm{CE}}$:** Standard cross-entropy on the target sequence.
- **$\mathcal{L}_{\mathrm{ground}}$:** Negative log-likelihood between phrase-object pairs; $y_{ij}=1$ if phrase $i$ grounds to object $j$.
- **$\mathcal{L}_{\mathrm{hall}}$:** BCE loss to penalize hallucinated object mentions in negative examples.
Co-efficients: $\lambda_{\mathrm{ground}} = 1.0$, $\lambda_{\mathrm{hall}} = 0.1$.

Explicit phrase-to-object tags are crucial for minimizing hallucination and improving faithfulness in object reference.

## 4. 3D-POPE Benchmark and Quantitative Evaluation

3D-POPE (3D Probing Of Precision and Existence) is introduced as a benchmark for standardized, compositional hallucination evaluation in 3D-LLMs [2406.05132]. The design comprises:

- **Dataset:** Derived from the ScanNet200 validation set (141 scenes, 200 classes).
- **Task Format:** Each sample is a (scene, object category Q, binary answer) triple, balanced 1:1 positive:negative.
- **Negative Sampling:** Random, popular (top-$k$ frequent absents), and adversarial (highest co-occurrence absent).

Evaluation metrics are standard:
\[
\text{Precision} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},\quad 
\text{Recall}    = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}},\quad 
F_1   = 2\cdot \frac{\mathrm{Precision}\cdot \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}},
\\
\text{Accuracy} = \frac{\mathrm{TP}+\mathrm{TN}}{\text{Total}},\quad 
H\,(\text{Hallucination rate}) = 1 - \mathrm{Precision}
\]
All models are tested zero-shot on ScanNet to ensure strict comparability.

### Representative Results

| Model           | Precision | Recall | F1    | Accuracy | Yes (%) |  
|-----------------|-----------|--------|-------|----------|---------|  
| Random Baseline | 50.00     | 50.00  | 50.00 | 50.00    | 50.00   |  
| 3D-LLM          | 50.03     | 99.88  | 66.67 | 50.07    | 99.81   |  
| 3D-VisTA        | 50.12     | 53.58  | 51.79 | 49.66    | 53.95   |  
| LEO             | 51.95     | 77.65  | 62.25 | 52.91    | 74.73   |  
| 3D-GRAND        | 93.34     | 84.25  | 88.56 | 89.12    | 45.13   |

- Precision for 3D-GRAND exceeds 93% in random and ≥70% in all negative sampling regimes, outperforming all prior baselines (≤52%).

On ScanRefer zero-shot (sim-to-real), 3D-GRAND achieves 38.0% Acc@0.25 (compared to 30.3% for 3D-LLM and 17.1% for LLM-Grounder).

A data-scaling effect is observed: Acc@0.25 grows ~linearly from ≈20% to ≈38% as training data increases from 0.2M to 6.2M examples; hallucination drops from ≈30% to ≈7%.

## 5. 3D-GRAND in Multi-Agent Gaussian SLAM

The 3D-GRAND approach constitutes the 3D extension of GRAND-SLAM's collaborative multi-agent Gaussian splatting-based SLAM [2506.18885]. The methodology integrates:

1. **3D Gaussian Splatting for Scene Representation:** Scene geometry and appearance are modeled as collections of explicit, anisotropic Gaussians $G_i(\mu_i, \Sigma_i, o_i, c_i)$; rendering utilizes radiance accumulation along view rays.
2. **Submap-Based Local Tracking and Optimization:** Each robot maintains overlapping submaps, seeded adaptively in under-observed regions and incrementally optimized via photometric and geometric rendering losses across multiple keyframes. Frame-to-map registration uses a hybrid color-depth odometry, refined through differentiable rendering alignment.
3. **Loop Closure and Pose-Graph Optimization:** Both intra- and inter-agent loop closures are established via NetVLAD feature matching, followed by initial pose estimation, colored point cloud ICP refinement, and constraint gating (based on fitness and RMSE). The resulting pose-graph (nodes: keyframes; edges: tracking/loops) is optimized (e.g., via GTSAM Levenberg–Marquardt), updating all submap origins and globally aligning Gaussians.
4. **Performance Highlights:** 
   - On Replica indoor datasets, 3D-GRAND (as GRAND-SLAM) yields a 28% improvement in RGB PSNR over prior strong baselines, and trajectory errors down to 0.25–0.27 cm.
   - On outdoor Kimera-Multi sequences (1.85 km), it exhibits a 91% reduction in tracking error (ATE RMSE: 4.99 m vs. 60.79 m) and maintains rendering fidelity (PSNR ≈28 dB, SSIM ≈0.97, LPIPS ≈0.11), substantially surpassing Gaussian-SLAM and MAGiC-SLAM.

## 6. Key Insights, Implications, and Future Directions

- **Dense Grounding:** The explicit pairing of linguistic phrases to 3D objects is essential for minimizing hallucination and maximizing reference accuracy. Removal of grounding tokens leads to a measurable drop in evaluation metrics (see ablations in [2406.05132]).
- **Data Scaling Law:** Performance on grounding and faithfulness tasks scales nearly linearly with data volume, with no saturation observed at the 6M instruction level. Densely grounded data variants consistently outperform non-grounded counterparts.
- **Sim-to-Real Transfer:** Models trained exclusively on synthetic 3D-GRAND data transfer effectively to real scans, with strong zero-shot results and no fine-tuning on real data, indicating robust generalization.
- **Standardization:** 3D-POPE establishes a reproducible, public leaderboard and standardized benchmark for 3D-LLM hallucination and grounding, facilitating fair comparison across architectures.
- **Embodied AI Directions:** Research advocates integrating real-scan fine-tuning, multi-view LLM-vision fusion, and robotic embodiment (action grounding) leveraging 3D-GRAND pretraining as a foundation for more capable and reliable embodied agents.

3D-GRAND, as both a dataset for instruction-tuned vision-language models and as a core component in scalable multi-agent SLAM, provides critical infrastructure for achieving accurate, trustworthy 3D understanding in embodied AI systems [2406.05132][2506.18885].

Source: https://www.emergentmind.com/topics/3d-grand