---
title: 'ScanRefer Benchmark: 3D Visual Grounding'
url: https://www.emergentmind.com/topics/scanrefer-benchmark
type: topic
---

# ScanRefer Benchmark: 3D Visual Grounding

The ScanRefer benchmark is a standardized evaluation suite for 3D object localization via natural language in RGB-D scans, central to the 3D visual grounding (3DVG) and referring expression comprehension (REC) research communities. Constructed atop ScanNet’s richly labeled 3D reconstructions of indoor scenes, it uniquely correlates raw, free-form language descriptions to specific object instances in real-world 3D data, driving multimodal research at the intersection of geometry, vision, and language.

## 1. Dataset Composition and Linguistic Characteristics

ScanRefer comprises 51,583 human-authored referring expressions describing 11,046 object instances across 800 unique indoor scenes sampled from ScanNet’s RGB-D corpus ([1912.08830]). The scenes are diverse (living rooms, offices, etc.) and contain a broad set of objects: the original taxonomy defines 18 coarse categories (chair, table, sofa, window, etc.), with “chair” being the most frequent. Each object is referenced on average by 4.67 distinct descriptions, with a vocabulary size of ≈4,200 and mean utterance length of ≈20 tokens.

Descriptions are highly compositional: spatial relations are present in 98.7% of queries, color in 74.7%, shape in 64.9%, and size in 14.2%. Comparative (672) and superlative (2,734) cues frequently appear. The tail of the dataset contains especially challenging cases where geometric, visual, and discourse-level cues must be reasoned jointly.

The dataset is split strictly by scenes: 562 training, 141 validation, and 97 test scenes, corresponding respectively to 7,875, 2,068, and 1,103 unique objects, and 36,665, 9,508, and 5,410 descriptions ([1912.08830], [2412.04383], [2309.05251]).

## 2. Task Definition and Annotation Protocol

The primary task in ScanRefer is single-object 3D visual grounding: Given a 3D point cloud $P=\{p_i\}_{i=1,\ldots,N_P}$ with features $f'_i$ (XYZ, RGB, normals, multi-view image features, height) and a free-form language description $d$ (token sequence mapped to 300-d GloVe and encoded by GRU), predict an axis-aligned 3D bounding box $B^{\text{pred}}$ with center and size parameters $(c_x, c_y, c_z, r_x, r_y, r_z)$ corresponding to the described object ([1912.08830], [2412.04383]).

Annotations were crowdsourced: annotators first segmented objects in ScanNet point clouds, then wrote natural language queries to unambiguously reference each object among its distractors (“the floral chair next to the three-seater couch”). The split between “unique” (one instance of class per scene) and “multiple” (several same-class distractors) is explicit and used for stratified evaluation ([2309.05251], [2410.13860]).

## 3. Evaluation Metrics and Protocol

Performance is measured using intersection-over-union ($\mathrm{IoU}$) between axis-aligned 3D boxes:

\[
\mathrm{IoU}(B_{\text{pred}}, B_{\text{gt}}) = \frac{\mathrm{vol}(B_{\text{pred}}\cap B_{\text{gt}})}{\mathrm{vol}(B_{\text{pred}}\cup B_{\text{gt}})}
\]

Top-$1$ accuracy at threshold $t$ ($\mathrm{Acc}@t$) counts the proportion of samples with $\mathrm{IoU} \ge t$, evaluated at $t\in\{0.25, 0.5\}$ ([1912.08830], [2603.08131], [2412.04383]). For segmentation settings (3DRES), mask-level $\mathrm{mIoU}$ and $\mathrm{Overall}@t$ are also employed ([2603.17753]).

For multi-object extensions ([2309.05251]), bipartite matching (Hungarian algorithm) aligns predicted and GT boxes, allowing calculation of Precision, Recall, and $F_1$ at various IoU thresholds. Special handling ensures correct scoring of “zero-target” queries.

## 4. Baseline and Advanced Methodologies

### 4.1 Classical Baselines

The canonical baseline ([1912.08830]) integrates a VoteNet-style detector to generate up to $M$ candidate bounding boxes from P, extracts 128-d proposal features $f_{3D}$, fuses each with the 256-d language embedding $e$ via MLP, and scores proposals with a linear softmax head, selecting the top-scoring one. The end-to-end loss combines localization, detection, and auxiliary language-to-object classification, with empirical weights $\alpha=0.1$, $\beta=10$, $\gamma=1$.

Voting, proposal and fusion ablations demonstrate limited performance with semantic labels alone (21.88% Acc@0.5), rising with richer geometry (+normals, multi-view image features) and full descriptions (up to 27.40% Acc@0.5 on val).

### 4.2 Classical and Panoptic Matching Advances

InstanceRefer [2103.01128] replaces proposal-centric matching with panoptic segmentation, predicting candidate instance point clouds using PointGroup. Holistic matching aggregates attributes (AP), instance-to-instance relations (RP), global localization (GLP), and uses deep co-attention, achieving 40.23%/32.93% (val) and 44.27%/35.80% (test) at Acc@0.25/0.5—substantial improvements over original baselines.

Look Around and Refer (LAR) [2211.14241] distills 2D semantics by synthetically rendering multiple views per proposal, encoding these with a Tiny-ConvNeXt, and aligning via a visual transformer. A hybrid five-term loss (object class, referring, correspondence, language class) guides joint 2D–3D multimodal learning, yielding 54.6% referring accuracy in the bounding-box setting (surpassing previous state-of-the-art by +1.0 point).

### 4.3 Vision-Language and Zero-Shot Grounding

SeeGround [2412.04383] and VLM-Grounder [2410.13860] exploit pretrained 2D vision-language models for 3DVG, eschewing direct 3D supervision. SeeGround employs a Perspective Adaptation Module (PAM) for query-aligned rendering and a Fusion Alignment Module (FAM) to inject explicit object markers into rendered images, feeding paired images and text to a VLM to score the relevant box. This achieves 44.1% Acc@0.25, outperforming prior zero-shot SOTA (ZSVG3D, 36.4%), with ablations confirming the complementary value of PAM and FAM.

VLM-Grounder dynamically stitches multi-view RGB-D images, and uses a feedback/ensemble strategy for projecting grounded 2D masks back to 3D bounding boxes, obtaining 51.6% Acc@0.25, approaching the best fully supervised approaches ([2410.13860]).

Open-vocabulary and open-world generalization have been driven by non-3D-supervised models such as UniGround [2603.08131], which achieves 46.1%/34.1% Acc@0.25/0.5 in a purely training-free, prompting-oriented scheme.

### 4.4 Unified and Dual-Task Grounding

PC-CrossDiff [2603.17753] unifies 3D bounding and segmentation via point-level and cluster-level bidirectional differential attention, coupled with a multi-task harmonized loss. On the ScanRefer test set, it achieves 58.47%/47.89% (REC, @0.25/@0.50) and 60.41%/52.52% (RES, @0.25/@0.50), with especially strong results on challenging implicit and multiple subsets. Differential attention yields notable improvements over vanilla cross-attention.

## 5. Extensions and Analysis: Multi-Target, Segmentation, Implicit Cues

Extensions such as Multi3DRefer [2309.05251] broaden ScanRefer with zero-, single- and multi-target queries, attribute-type annotations (spatial, color, texture, shape), and a new evaluation metric (F1@t) for multi-grounding. Rescored/rewritten language with increased syntactic diversity enhances the challenge. Contrastive multimodal CLIP-based methods yield +6.8 points over geometry-only baselines in Acc@0.5, and F1 improvements in the multi-target setting.

Segmentation-based benchmarks (ScanRefer 3DRES: [2603.17753]) deploy per-point binary mask tasks in parallel with bounding-box localization, revealing complementary difficulties and solution pathways.

Difficulties in parsing implicit cues (queries without explicit spatial prepositions) are systematically evaluated, with PC-CrossDiff reporting +10.16% absolute gain over previous best methods on Implicit@0.50.

## 6. Key Challenges, Open Problems, and Best Practices

Oracle baselines ([1912.08830]) show that knowledge of the semantic class yields near-perfect accuracy only for “unique” queries; performance collapses with multiple distractors. Even with access to ground-truth boxes, establishing the mapping from language to object is far from solved (~73.5% Acc@0.5 on “unique,” ~32% on “multiple”).

Error analysis identifies failure modes: suboptimal 3D proposals for thin/flat objects (pictures, sinks), ambiguity in complex spatial discourse (“third from the wall,” “shorter than…”), and occlusion. Best practices established include using full multi-sentence descriptions, enriching point features with normals and multi-view semantics, and fusing proposal and language representations with attention ([1912.08830], [2103.01128]). Future benchmarks are expected to prioritize improved spatial reasoning, zero-shot class grounding, and fully unified 2D–3D segmentation/localization.

## 7. Comparative Performance and Ongoing Evolution

Performance across ScanRefer has trended upwards from original baselines (41.2% Acc@0.25, 27.4% Acc@0.5 [1912.08830]) to state-of-the-art supervised methods (MCLN 57.2%/45.7%, TSP3D 56.5%/46.7% [2603.08131]), advanced panoptic/instance-centric (InstanceRefer 40.2%/32.9% [2103.01128]), CLIP-based hybrid models (LAR, 54.6% [2211.14241]), dual-task attention architectures (PC-CrossDiff 58.47%/47.89% [2603.17753]), and the strongest zero-shot VLM-based models (VLM-Grounder, 51.6%/32.8%; SeeGround, 44.1%/39.4% [2410.13860], [2412.04383]).

A tabular summary of selected methods’ overall val-set accuracy is given below:

| Method           | Supervision    | Acc@0.25 (%) | Acc@0.5 (%) |
|------------------|---------------|-------------|-------------|
| ScanRefer (2020) | Full          |   41.2      |   27.4      |
| InstanceRefer    | Full          |   40.2      |   32.9      |
| MCLN             | Full          |   57.2      |   45.7      |
| PC-CrossDiff     | Full          |   58.5      |   47.9      |
| VLM-Grounder     | Zero-shot     |   51.6      |   32.8      |
| SeeGround        | Zero-shot     |   44.1      |   39.4      |
| UniGround        | Zero-shot/OW  |   46.1      |   34.1      |

This persistent gap between “multiple” and “unique” splits, and the highest scores clustering well below 60%, reflect the intrinsic difficulty of holistic 3DVG under unconstrained natural language and complex indoor geometry ([2603.08131], [2412.04383], [2603.17753], [2103.01128]).

---

The ScanRefer benchmark thus defines the canonical problem and dataset for single-object 3D visual grounding via language, providing both a standard for comparative analysis and an evolving platform for new algorithmic developments in geometry–language–vision integration. Its enduring challenge has motivated substantial progress in proposal mechanisms, cross-modal fusion, panoptic reasoning, and open-set/compositional generalization.

Source: https://www.emergentmind.com/topics/scanrefer-benchmark