---
title: 'ViGiL3D: 3D Visual Grounding Dataset'
url: https://www.emergentmind.com/topics/vigil3d
type: topic
---

# ViGiL3D: 3D Visual Grounding Dataset

Searching arXiv for ViGiL3D and closely related 3D visual grounding papers.
ViGiL3D, short for **Visual Grounding with Diverse Language in 3D**, is a compact, human-annotated diagnostic dataset for **3D visual grounding (3DVG)**, the task of localizing entities in a 3D scene referred to by natural language text. It was introduced to evaluate whether open-vocabulary 3DVG methods can robustly localize objects in 3D scenes when the language is as varied and nuanced as natural English, rather than restricted to the narrower prompt distributions typical of prior benchmarks. Built on ScanNet and ScanNet++ scenes, ViGiL3D provides **350 prompts across 35 scenes**, emphasizes out-of-distribution linguistic phenomena, and is intended as a **test-only, diagnostic benchmark** rather than a training corpus [2501.01366].

## 1. Task setting and motivation

3D visual grounding requires linking natural-language mentions—targets and context—to referents in a 3D scene. In the open-vocabulary setting, the task additionally requires generalizing beyond seen classes and phrasing. This capability underpins embodied AI, large-scale 3D scene retrieval, and assistive mixed or augmented reality systems [2501.01366].

The motivation for ViGiL3D is the mismatch between real linguistic variation and the prompt distributions represented in widely used 3DVG datasets. ScanRefer, Nr3D, Sr3D, ReferIt3D, and Multi3DRefer collectively cover roughly **700 indoor scenes** and approximately **170K grounding prompts**, but they emphasize discriminative references within a scene while often relying on direct object class names, sparse or templated relationships, and few negative or complex linguistic constructs. LLM-scaled corpora such as 3D-VisTA, 3D-GRAND, SceneVerse, and ScanScribe greatly expand volume, but they can be overspecified, unbalanced through over-reliance on easy cues such as class names, and still miss phenomena including explicit negation, agent or viewpoint conditioning, arrangement or ordinal reasoning, and non-object anchors such as regions or rooms [2501.01366].

ViGiL3D is therefore framed as a benchmark for linguistic coverage rather than scale alone. The dataset is designed around the question of whether current methods can handle prompts such as “the second mug from the left on the shelf, not the one with ‘fragile’ written on it,” where successful localization depends on compositional reasoning, exclusion constraints, and viewpoint-sensitive reference. A plausible implication is that the benchmark targets the linguistic regimes most likely to matter when 3DVG systems are deployed outside ScanRefer-like evaluation distributions.

## 2. Linguistic analysis framework

A central contribution of ViGiL3D is a framework for analyzing 3DVG prompts through a decomposition into **targets, anchors, attributes, and relationships**, together with grammar-level phenomena such as **coreference**, **not-first-noun (NFN)**, and **negation** [2501.01366]. The framework operationalizes **6 count metrics** and **24 binary metrics** across three categories.

**Language Diversity (DIV)** measures variation and coverage of linguistic forms, including lexical diversity through the unique bigram proportion over lexical tokens, denoted **2lex**. **Language Resolution (RES)** measures whether descriptors can be linked to intended referents; it includes whether the target is the first noun phrase and whether later mentions corefer to earlier text. **Understanding Attributes and Relationships (UAR)** quantifies the counts and types of attributes and relationships attached to targets and anchors [2501.01366].

The attribute taxonomy includes **color, size, shape, number, material, function, texture, style, text label, and state**. Representative prompts include “Find the object labeled ‘caution’.” for **text label** and “Find the folded chair closest to the door.” for **state**. The relationship taxonomy includes **near or proximity, far or opposite, directional or viewpoint-dependent, vertical, containment or part-of, arrangement, ordinal position, and comparison**. Representative prompts include “In the row of chairs and tables against the wall, find the third chair from the left.” for **ordinal position** and “This fancy rotating display is the one nearest to the orange carpet.” for **comparison** [2501.01366].

The target reference taxonomy distinguishes **generic**, **coarse-grained**, and **fine-grained** mentions. Thus, a target may be referred to as an “object,” a broader category such as “device” or “container,” or a canonical class name. The framework also distinguishes anchor types, including **single-object anchors**, **multi-object anchors**, **non-object anchors** such as regions or rooms, and **agent or viewpoint-based anchors**. ViGiL3D explicitly emphasizes phenomena that are underrepresented or missing in prior datasets: **negation, generic and coarse references, agent or viewpoint dependence, multi-object and arrangement-based relations, ordinal reasoning, non-object anchors, NFN, number or state or text-label attributes, and richer lexical variety** [2501.01366].

The paper also reports an automated analysis pipeline based on **GPT-4o + SpaCy**. Across **225 manually validated prompts**, it achieves **average precision 0.86** and **recall 0.91** across the **24 binary metrics**, with **median count errors 0.0** and **mean absolute error approximately 0.43** [2501.01366].

## 3. Dataset design, annotation, and composition

ViGiL3D is a **compact, human-annotated diagnostic dataset built on real indoor scenes**. The source scenes are **ScanNet** and **ScanNet++**, the latter described as higher-fidelity indoor scans. The dataset contains **350 prompts across 35 scenes**, with **vocabulary size = 942 unique words**, **average length 14.1 tokens**, and **1.2 sentences per prompt** [2501.01366].

Target cardinality is deliberately heterogeneous. Of the 350 prompts, **43 prompts have 0 targets** as diagnostic non-target cases, **275 have 1 target**, and **32 have multiple targets**. The benchmark is explicitly **test-only**; there is **no training split**, and out-of-distribution characterization is instead provided through category tags and subgroup partitions for analysis [2501.01366].

The annotation process is manual. Human annotators, specifically the authors, viewed the **3D point cloud**, **ground-truth instance segmentation**, and **RGB video** for each scene. For each prompt, they selected a target, multiple targets, or none, and then wrote diverse, natural grounding descriptions spanning the intended linguistic phenomena while maintaining balanced specificity so as to avoid both underspecification or ambiguity and overspecification. Prompts were annotated with metadata indicating which phenomena were present and were manually validated for correctness [2501.01366].

A separate human evaluation study achieved **84% accuracy in locating targets on ScanNet scenes**, which the paper interprets as confirmation that the prompts are solvable while still exceeding current model performance. ViGiL3D also shows **higher lexical diversity**, with **2lex approximately 0.45**, than prior datasets, and it exhibits more balanced descriptor counts than LLM-scaled corpora that often overspecify [2501.01366].

The data schema is organized around references to ScanNet and ScanNet++ reconstructions, including point clouds and RGB or RGB-D streams. Ground-truth instance segmentations provide referent objects, and evaluation uses **GT instance boxes** or **predicted boxes**, for example from **Mask3D**. The released metadata includes prompt text, linguistic category tags, referent IDs, and scene metadata linking objects and prompts. The project page is **https://3dlg-hcvc.github.io/vigil3d/**, while use of the underlying scene datasets remains governed by the upstream terms of ScanNet and ScanNet++ [2501.01366].

## 4. Evaluation protocol and benchmarked methods

ViGiL3D follows standard 3DVG evaluation practice but adapts it to multi-target and zero-target settings. The benchmark reports **accuracy** and **F1** at **IoU thresholds 0.25 and 0.50**, using either **ground-truth boxes** or **predicted boxes** such as Mask3D proposals. Intersection-over-Union for 3D bounding boxes is defined as

$$
\mathrm{IoU}(B_{\mathrm{pred}}, B_{\mathrm{gt}})=\frac{|B_{\mathrm{pred}} \cap B_{\mathrm{gt}}|}{|B_{\mathrm{pred}} \cup B_{\mathrm{gt}}|}.
$$

Accuracy at threshold $\tau$ is

$$
\mathrm{Acc}@\tau=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\mathrm{IoU}_i \ge \tau].
$$

For multi-target prompts, the benchmark does not prioritize mean IoU; instead, it uses **F1**, defined as the harmonic mean of precision and recall over the set of targets for a prompt:

$$
F1=\frac{2 \cdot P \cdot R}{P+R}.
$$

For multi-target prompts, **accuracy** requires all targets to be localized at the threshold, whereas **F1** captures partial localization quality [2501.01366].

The evaluation covers several families of open-vocabulary 3DVG methods. **CLIP-aligned 3D representations** include **OpenScene**, which aligns 3D point features to CLIP through 2D segmentation supervision using **MinkUNet18A**, and **LERF**, which uses language-embedded NeRF features with **CLIP ViT-B/16**. **Zero-shot 3DVG with LLMs** includes **ZSVG3D**, which uses visual program synthesis with **GPT-4o** and a localization module based on **CLIP ViT-B/16**, and **LLM-Grounder**, which reasons in **GPT-4** over candidate objects detected through CLIP-aligned features. **Models trained on large-scale 3DVG data** include **3D-VisTA**, **3D-GRAND**, and **PQ3D** [2501.01366].

The reported evaluation was run with both GT boxes and Mask3D predictions, and the total compute across methods was approximately **22 GPU-hours on an RTX 4090**. Inference times vary substantially: **LERF** is slowest because it requires per-scene optimization, whereas **PQ3D**, **3D-VisTA**, and **OpenScene** are faster. The overview states, for example, that **3D-VisTA** fine-tuned on ScanRefer achieves approximately **0.02 s per scene** with batch size 1, while **LERF** requires **minutes per scene** because of per-scene NeRF optimization [2501.01366].

## 5. Empirical results and diagnostic findings

On ScanNet scenes in ViGiL3D, performance drops sharply relative to ScanRefer-like validation distributions. Using **GT boxes**, overall accuracy on ViGiL3D is reported as **OpenScene 2.1%**, **LERF 2.5%**, **ZSVG3D 18.9%**, **LLM-Grounder 2.5%**, **3D-VisTA 14.2%**, **3D-GRAND 17.9%**, and **PQ3D 26.2%**. With **predicted boxes (Mask3D)**, performance remains low: at **Acc@25**, the results are **OpenScene 1.7%**, **LERF 2.1%**, **ZSVG3D 8.5%**, **LLM-Grounder 7.1%**, **3D-VisTA 15.8%**, **3D-GRAND 15.8%**, and **PQ3D 10.8%**; at **Acc@50**, they are **OpenScene 1.3%**, **LERF 2.1%**, **ZSVG3D 5.6%**, **LLM-Grounder 5.0%**, **3D-VisTA 13.3%**, **3D-GRAND 12.5%**, and **PQ3D 10.8%**. The paper states that **F1 mirrors Acc trends and shows the same gap** [2501.01366].

The contrast with ScanRefer validation is substantial. **PQ3D** reports **57.0%/51.2% Acc@25/@50** on ScanRefer versus **10.8%/10.8%** on ViGiL3D. **3D-VisTA** reports **50.6%/45.8%** versus **15.8%/13.3%**. **3D-GRAND** reports **38.0%/27.4%** versus **15.8%/12.5%**. **ZSVG3D** reports **36.4%/32.7%** on ScanRefer validation with GPT-3.5 versus **8.5%/5.6%** on ViGiL3D with GPT-4o [2501.01366].

Subgroup analysis with GT boxes highlights where current methods fail. **PQ3D** is best in most categories, including **Number 28.9%**, **State 28.0%**, **Far 26.7%**, **Arrangement 22.9%**, **Comparison 24.5%**, **Generic target 20.0%**, **Coarse-grained 24.5%**, **Fine-grained 28.6%**, **NFN 26.1%**, and **Non-object anchors 24.6%**. **ZSVG3D** leads on **Text label 12.0%**, **Ordinal 19.2%**, **Comparison 25.0%**, and **Agent-based anchors 23.1%**, and it also handles **State (28.0%)** well. **3D-GRAND** is relatively strong on **Negation (21.6% on GT)** and on **multi-object anchors (15.4%)** and arrangement or ordinal reasoning compared to other trained baselines. **CLIP-only methods**, namely OpenScene and LERF, struggle across the board, especially on complex relational language, negation, and non-class references [2501.01366].

An important empirical anomaly is that some methods perform **better with Mask3D predictions than with GT boxes**. This occurs for methods including **LLM-Grounder** and **3D-VisTA**. The paper attributes this to sensitivity to box generation pipelines and their auxiliary signals, possible incompleteness or mismatches between GT and candidate object sets exposed to the model, and dependence of LLM reasoning on the box dimensions fed into the prompt, for example **DBSCAN-cluster-derived** dimensions versus GT box sizes [2501.01366].

The benchmark also evaluates generalization to **ScanNet++**. On higher-fidelity ScanNet++ scenes with GT boxes, **ZSVG3D** achieves **Acc 18.3%, F1 24.5%**, **3D-VisTA** achieves **Acc 11.8%, F1 11.1%**, and **3D-GRAND** achieves **Acc 9.2%, F1 9.2%**. The paper notes that the larger spatial extent and object counts of ScanNet++ make grounding harder; **3D-GRAND** is additionally affected by token truncation, whereas **ZSVG3D** scales more gracefully, albeit with slower inference [2501.01366].

## 6. Interpretation, limitations, and subsequent developments

The main interpretive claim of ViGiL3D is that current 3DVG systems exhibit **dataset-induced brittleness**. Methods tuned to ScanRefer-like distributions rely heavily on class names and simple relations, whereas ViGiL3D requires parsing **compositional constraints**, handling **negation** and **generic or coarse references**, and reasoning over **non-object** and **agent-based** anchors. The paper further argues that **pure CLIP alignment** struggles with language that is not effectively a bag-of-words description, while **LLM-based reasoning helps but is fragile**, with failures arising from parsing, program synthesis, and sensitivity to the quality of detected candidate objects [2501.01366].

The benchmark’s broader methodological claim is that **training data distribution matters as much as scale**. Models trained on LLM-scaled corpora such as **3D-VisTA** and **3D-GRAND** outperform CLIP-only baselines, but they are not universally stronger than **PQ3D**, which is trained on manually annotated datasets. The paper explicitly concludes that simply scaling dataset size or relying on synthetic scenes does not guarantee broader language generalization, particularly when overspecified prompts bias models toward easy signals [2501.01366].

ViGiL3D also states several limitations. The dataset is **small** and **test-only**, which may constrain statistical power even though many subgroup results are statistically significant. It is **English-only**, so language and cultural scene differences are not represented. It also inherits potential biases from reliance on **ScanNet/ScanNet++ indoor scenes**, from **GT segmentation granularity** that may miss text labels or fine details, and from possible misclassifications in the LLM-based analysis pipeline, though validation mitigates this [2501.01366].

The paper identifies several future directions: scaling the benchmark while retaining **balanced, compositional language**; enriching 3D inputs with **higher-resolution RGB-D** and **multi-view images** for better text-label recognition and fine-grained attribute perception; and developing architectures that move beyond bag-of-words alignment through **graph- or program-based reasoning**, **explicit anchor modeling**, and **multi-step constraint satisfaction** across attributes and relations [2501.01366].

A direct successor, **ViGiL3D++**, positions ViGiL3D as the manually annotated benchmark that established what diversity in grounding language looks like, but not one that scales. ViGiL3D++ introduces a **scalable, scene-agnostic generation pipeline** based on **cross-referenced scene graphs**, **constraint sampling**, and **LLM rephrasing**, and it reports that models trained on it improve performance on several 3DVG benchmarks while still exposing limitations of current VLMs [2606.20946]. This suggests that ViGiL3D’s main legacy lies not only in its diagnostic test set, but also in its formalization of linguistic diversity as a measurable and model-relevant property of 3D visual grounding data.

Source: https://www.emergentmind.com/topics/vigil3d