ViGiL3D: 3D Visual Grounding Dataset

• ViGiL3D is a compact, human-annotated dataset for 3D visual grounding that evaluates open-vocabulary methods using diverse, natural language prompts.
• It leverages ScanNet and ScanNet++ scenes with 350 prompts across 35 scenes, emphasizing complex phenomena like negation, ordinal reasoning, and multi-object relationships.
• The dataset provides a diagnostic framework with detailed linguistic and relational metrics to drive advancements in embodied AI and augmented reality applications.

Searching arXiv for ViGiL3D and closely related 3D visual grounding papers. ViGiL3D, short for Visual Grounding with Diverse Language in 3D, is a compact, human-annotated diagnostic dataset for 3D visual grounding (3DVG), the task of localizing entities in a 3D scene referred to by natural language text. It was introduced to evaluate whether open-vocabulary 3DVG methods can robustly localize objects in 3D scenes when the language is as varied and nuanced as natural English, rather than restricted to the narrower prompt distributions typical of prior benchmarks. Built on ScanNet and ScanNet++ scenes, ViGiL3D provides 350 prompts across 35 scenes, emphasizes out-of-distribution linguistic phenomena, and is intended as a test-only, diagnostic benchmark rather than a training corpus (Wang et al., 2 Jan 2025).

1. Task setting and motivation

3D visual grounding requires linking natural-language mentions—targets and context—to referents in a 3D scene. In the open-vocabulary setting, the task additionally requires generalizing beyond seen classes and phrasing. This capability underpins embodied AI, large-scale 3D scene retrieval, and assistive mixed or augmented reality systems (Wang et al., 2 Jan 2025).

The motivation for ViGiL3D is the mismatch between real linguistic variation and the prompt distributions represented in widely used 3DVG datasets. ScanRefer, Nr3D, Sr3D, ReferIt3D, and Multi3DRefer collectively cover roughly 700 indoor scenes and approximately 170K grounding prompts, but they emphasize discriminative references within a scene while often relying on direct object class names, sparse or templated relationships, and few negative or complex linguistic constructs. LLM-scaled corpora such as 3D-VisTA, 3D-GRAND, SceneVerse, and ScanScribe greatly expand volume, but they can be overspecified, unbalanced through over-reliance on easy cues such as class names, and still miss phenomena including explicit negation, agent or viewpoint conditioning, arrangement or ordinal reasoning, and non-object anchors such as regions or rooms (Wang et al., 2 Jan 2025).

ViGiL3D is therefore framed as a benchmark for linguistic coverage rather than scale alone. The dataset is designed around the question of whether current methods can handle prompts such as “the second mug from the left on the shelf, not the one with ‘fragile’ written on it,” where successful localization depends on compositional reasoning, exclusion constraints, and viewpoint-sensitive reference. A plausible implication is that the benchmark targets the linguistic regimes most likely to matter when 3DVG systems are deployed outside ScanRefer-like evaluation distributions.

2. Linguistic analysis framework

A central contribution of ViGiL3D is a framework for analyzing 3DVG prompts through a decomposition into targets, anchors, attributes, and relationships, together with grammar-level phenomena such as coreference, not-first-noun (NFN), and negation (Wang et al., 2 Jan 2025). The framework operationalizes 6 count metrics and 24 binary metrics across three categories.

Language Diversity (DIV) measures variation and coverage of linguistic forms, including lexical diversity through the unique bigram proportion over lexical tokens, denoted 2lex. Language Resolution (RES) measures whether descriptors can be linked to intended referents; it includes whether the target is the first noun phrase and whether later mentions corefer to earlier text. Understanding Attributes and Relationships (UAR) quantifies the counts and types of attributes and relationships attached to targets and anchors (Wang et al., 2 Jan 2025).

The attribute taxonomy includes color, size, shape, number, material, function, texture, style, text label, and state. Representative prompts include “Find the object labeled ‘caution’.” for text label and “Find the folded chair closest to the door.” for state. The relationship taxonomy includes near or proximity, far or opposite, directional or viewpoint-dependent, vertical, containment or part-of, arrangement, ordinal position, and comparison. Representative prompts include “In the row of chairs and tables against the wall, find the third chair from the left.” for ordinal position and “This fancy rotating display is the one nearest to the orange carpet.” for comparison (Wang et al., 2 Jan 2025).

The target reference taxonomy distinguishes generic, coarse-grained, and fine-grained mentions. Thus, a target may be referred to as an “object,” a broader category such as “device” or “container,” or a canonical class name. The framework also distinguishes anchor types, including single-object anchors, multi-object anchors, non-object anchors such as regions or rooms, and agent or viewpoint-based anchors. ViGiL3D explicitly emphasizes phenomena that are underrepresented or missing in prior datasets: negation, generic and coarse references, agent or viewpoint dependence, multi-object and arrangement-based relations, ordinal reasoning, non-object anchors, NFN, number or state or text-label attributes, and richer lexical variety (Wang et al., 2 Jan 2025).

The paper also reports an automated analysis pipeline based on GPT-4o + SpaCy. Across 225 manually validated prompts, it achieves average precision 0.86 and recall 0.91 across the 24 binary metrics, with median count errors 0.0 and mean absolute error approximately 0.43 (Wang et al., 2 Jan 2025).

3. Dataset design, annotation, and composition

ViGiL3D is a compact, human-annotated diagnostic dataset built on real indoor scenes. The source scenes are ScanNet and ScanNet++, the latter described as higher-fidelity indoor scans. The dataset contains 350 prompts across 35 scenes, with vocabulary size = 942 unique words, average length 14.1 tokens, and 1.2 sentences per prompt (Wang et al., 2 Jan 2025).

Target cardinality is deliberately heterogeneous. Of the 350 prompts, 43 prompts have 0 targets as diagnostic non-target cases, 275 have 1 target, and 32 have multiple targets. The benchmark is explicitly test-only; there is no training split, and out-of-distribution characterization is instead provided through category tags and subgroup partitions for analysis (Wang et al., 2 Jan 2025).

The annotation process is manual. Human annotators, specifically the authors, viewed the 3D point cloud, ground-truth instance segmentation, and RGB video for each scene. For each prompt, they selected a target, multiple targets, or none, and then wrote diverse, natural grounding descriptions spanning the intended linguistic phenomena while maintaining balanced specificity so as to avoid both underspecification or ambiguity and overspecification. Prompts were annotated with metadata indicating which phenomena were present and were manually validated for correctness (Wang et al., 2 Jan 2025).

A separate human evaluation study achieved 84% accuracy in locating targets on ScanNet scenes, which the paper interprets as confirmation that the prompts are solvable while still exceeding current model performance. ViGiL3D also shows higher lexical diversity, with 2lex approximately 0.45, than prior datasets, and it exhibits more balanced descriptor counts than LLM-scaled corpora that often overspecify (Wang et al., 2 Jan 2025).

The data schema is organized around references to ScanNet and ScanNet++ reconstructions, including point clouds and RGB or RGB-D streams. Ground-truth instance segmentations provide referent objects, and evaluation uses GT instance boxes or predicted boxes, for example from Mask3D. The released metadata includes prompt text, linguistic category tags, referent IDs, and scene metadata linking objects and prompts. The project page is https://3dlg-hcvc.github.io/vigil3d/, while use of the underlying scene datasets remains governed by the upstream terms of ScanNet and ScanNet++ (Wang et al., 2 Jan 2025).

4. Evaluation protocol and benchmarked methods

ViGiL3D follows standard 3DVG evaluation practice but adapts it to multi-target and zero-target settings. The benchmark reports accuracy and F1 at IoU thresholds 0.25 and 0.50, using either ground-truth boxes or predicted boxes such as Mask3D proposals. Intersection-over-Union for 3D bounding boxes is defined as

$$\mathrm{IoU}(B_{\mathrm{pred}}, B_{\mathrm{gt}})=\frac{|B_{\mathrm{pred}} \cap B_{\mathrm{gt}}|}{|B_{\mathrm{pred}} \cup B_{\mathrm{gt}}|}.$$

Accuracy at threshold $\tau$ is

$$\mathrm{Acc}@\tau=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\mathrm{IoU}_i \ge \tau].$$

For multi-target prompts, the benchmark does not prioritize mean IoU; instead, it uses F1, defined as the harmonic mean of precision and recall over the set of targets for a prompt:

$F1=\frac{2 \cdot P \cdot R}{P+R}.$

For multi-target prompts, accuracy requires all targets to be localized at the threshold, whereas F1 captures partial localization quality (Wang et al., 2 Jan 2025).

The evaluation covers several families of open-vocabulary 3DVG methods. CLIP-aligned 3D representations include OpenScene, which aligns 3D point features to CLIP through 2D segmentation supervision using MinkUNet18A, and LERF, which uses language-embedded NeRF features with CLIP ViT-B/16. Zero-shot 3DVG with LLMs includes ZSVG3D, which uses visual program synthesis with GPT-4o and a localization module based on CLIP ViT-B/16, and LLM-Grounder, which reasons in GPT-4 over candidate objects detected through CLIP-aligned features. Models trained on large-scale 3DVG data include 3D-VisTA, 3D-GRAND, and PQ3D (Wang et al., 2 Jan 2025).

The reported evaluation was run with both GT boxes and Mask3D predictions, and the total compute across methods was approximately 22 GPU-hours on an RTX 4090. Inference times vary substantially: LERF is slowest because it requires per-scene optimization, whereas PQ3D, 3D-VisTA, and OpenScene are faster. The overview states, for example, that 3D-VisTA fine-tuned on ScanRefer achieves approximately 0.02 s per scene with batch size 1, while LERF requires minutes per scene because of per-scene NeRF optimization (Wang et al., 2 Jan 2025).

5. Empirical results and diagnostic findings

On ScanNet scenes in ViGiL3D, performance drops sharply relative to ScanRefer-like validation distributions. Using GT boxes, overall accuracy on ViGiL3D is reported as OpenScene 2.1%, LERF 2.5%, ZSVG3D 18.9%, LLM-Grounder 2.5%, 3D-VisTA 14.2%, 3D-GRAND 17.9%, and PQ3D 26.2%. With predicted boxes (Mask3D), performance remains low: at Acc@25, the results are OpenScene 1.7%, LERF 2.1%, ZSVG3D 8.5%, LLM-Grounder 7.1%, 3D-VisTA 15.8%, 3D-GRAND 15.8%, and PQ3D 10.8%; at Acc@50, they are OpenScene 1.3%, LERF 2.1%, ZSVG3D 5.6%, LLM-Grounder 5.0%, 3D-VisTA 13.3%, 3D-GRAND 12.5%, and PQ3D 10.8%. The paper states that F1 mirrors Acc trends and shows the same gap (Wang et al., 2 Jan 2025).

The contrast with ScanRefer validation is substantial. PQ3D reports 57.0%/51.2% Acc@25/@50 on ScanRefer versus 10.8%/10.8% on ViGiL3D. 3D-VisTA reports 50.6%/45.8% versus 15.8%/13.3%. 3D-GRAND reports 38.0%/27.4% versus 15.8%/12.5%. ZSVG3D reports 36.4%/32.7% on ScanRefer validation with GPT-3.5 versus 8.5%/5.6% on ViGiL3D with GPT-4o (Wang et al., 2 Jan 2025).

Subgroup analysis with GT boxes highlights where current methods fail. PQ3D is best in most categories, including Number 28.9%, State 28.0%, Far 26.7%, Arrangement 22.9%, Comparison 24.5%, Generic target 20.0%, Coarse-grained 24.5%, Fine-grained 28.6%, NFN 26.1%, and Non-object anchors 24.6%. ZSVG3D leads on Text label 12.0%, Ordinal 19.2%, Comparison 25.0%, and Agent-based anchors 23.1%, and it also handles State (28.0%) well. 3D-GRAND is relatively strong on Negation (21.6% on GT) and on multi-object anchors (15.4%) and arrangement or ordinal reasoning compared to other trained baselines. CLIP-only methods, namely OpenScene and LERF, struggle across the board, especially on complex relational language, negation, and non-class references (Wang et al., 2 Jan 2025).

An important empirical anomaly is that some methods perform better with Mask3D predictions than with GT boxes. This occurs for methods including LLM-Grounder and 3D-VisTA. The paper attributes this to sensitivity to box generation pipelines and their auxiliary signals, possible incompleteness or mismatches between GT and candidate object sets exposed to the model, and dependence of LLM reasoning on the box dimensions fed into the prompt, for example DBSCAN-cluster-derived dimensions versus GT box sizes (Wang et al., 2 Jan 2025).

The benchmark also evaluates generalization to ScanNet++. On higher-fidelity ScanNet++ scenes with GT boxes, ZSVG3D achieves Acc 18.3%, F1 24.5%, 3D-VisTA achieves Acc 11.8%, F1 11.1%, and 3D-GRAND achieves Acc 9.2%, F1 9.2%. The paper notes that the larger spatial extent and object counts of ScanNet++ make grounding harder; 3D-GRAND is additionally affected by token truncation, whereas ZSVG3D scales more gracefully, albeit with slower inference (Wang et al., 2 Jan 2025).

6. Interpretation, limitations, and subsequent developments

The main interpretive claim of ViGiL3D is that current 3DVG systems exhibit dataset-induced brittleness. Methods tuned to ScanRefer-like distributions rely heavily on class names and simple relations, whereas ViGiL3D requires parsing compositional constraints, handling negation and generic or coarse references, and reasoning over non-object and agent-based anchors. The paper further argues that pure CLIP alignment struggles with language that is not effectively a bag-of-words description, while LLM-based reasoning helps but is fragile, with failures arising from parsing, program synthesis, and sensitivity to the quality of detected candidate objects (Wang et al., 2 Jan 2025).

The benchmark’s broader methodological claim is that training data distribution matters as much as scale. Models trained on LLM-scaled corpora such as 3D-VisTA and 3D-GRAND outperform CLIP-only baselines, but they are not universally stronger than PQ3D, which is trained on manually annotated datasets. The paper explicitly concludes that simply scaling dataset size or relying on synthetic scenes does not guarantee broader language generalization, particularly when overspecified prompts bias models toward easy signals (Wang et al., 2 Jan 2025).

ViGiL3D also states several limitations. The dataset is small and test-only, which may constrain statistical power even though many subgroup results are statistically significant. It is English-only, so language and cultural scene differences are not represented. It also inherits potential biases from reliance on ScanNet/ScanNet++ indoor scenes, from GT segmentation granularity that may miss text labels or fine details, and from possible misclassifications in the LLM-based analysis pipeline, though validation mitigates this (Wang et al., 2 Jan 2025).

The paper identifies several future directions: scaling the benchmark while retaining balanced, compositional language; enriching 3D inputs with higher-resolution RGB-D and multi-view images for better text-label recognition and fine-grained attribute perception; and developing architectures that move beyond bag-of-words alignment through graph- or program-based reasoning, explicit anchor modeling, and multi-step constraint satisfaction across attributes and relations (Wang et al., 2 Jan 2025).

A direct successor, ViGiL3D++, positions ViGiL3D as the manually annotated benchmark that established what diversity in grounding language looks like, but not one that scales. ViGiL3D++ introduces a scalable, scene-agnostic generation pipeline based on cross-referenced scene graphs, constraint sampling, and LLM rephrasing, and it reports that models trained on it improve performance on several 3DVG benchmarks while still exposing limitations of current VLMs (Wang et al., 18 Jun 2026). This suggests that ViGiL3D’s main legacy lies not only in its diagnostic test set, but also in its formalization of linguistic diversity as a measurable and model-relevant property of 3D visual grounding data.