---
title: 'ScanRefer: 3D Object Localization Dataset'
url: https://www.emergentmind.com/topics/scanrefer-dataset
type: topic
---

# ScanRefer: 3D Object Localization Dataset

ScanRefer is a large-scale dataset designed for the task of 3D object localization in indoor RGB-D scans using free-form natural language descriptions. It provides the foundational benchmark for research in grounding complex referring expressions directly in full 3D reconstructed environments, enabling rigorous investigation of multi-modal scene understanding, embodied AI, and language-based object localization [1912.08830].

## 1. Motivation and Task Definition

The core motivation behind ScanRefer is to enable the localization of arbitrary objects within fully reconstructed 3D scenes using only natural language queries as guidance. Unlike prior efforts restricted to 2D images or those relying on projecting language references onto 3D via intermediate steps, ScanRefer targets direct linguistic grounding within 3D point clouds. The benchmark addresses practical application demands in robotics, human–computer interaction, and AR by requiring models to ground free-form referring expressions—capturing reference via spatial layout, appearance, and semantic context—against complex environments [1912.08830, 2412.04383].

The formal task input is a pair $(P, Q)$, where $P$ is the point cloud of an indoor scene and $Q$ is a natural language description mandating localization of a unique target object. The output is a tight, axis-aligned 3D bounding box predicted to match the described instance.

## 2. Data Composition and Structure

### ScanNet Foundation

ScanRefer is constructed atop ScanNet, which provides 1,613 RGB-D scans of 806 distinct indoor environments. After filtering out scenes lacking describable objects, 800 scenes are included in the final dataset [1912.08830].

### Object and Expression Statistics

- **Object Instances:** 11,046 objects across 265 ScanNet object classes, subsequently aggregated into 18 common semantic categories (e.g., chair, table, cabinet, plus an “others” class).
- **Natural Language Descriptions:** 51,583 free-form, multi-sentence descriptions, averaging 4.67 descriptions per object instance. Descriptions average 20.27 words each, yielding a vocabulary of 4,197 unique tokens.

### Dataset Splits

The scenes are partitioned into non-overlapping train/val/test splits for comparative benchmarking:

| Split       | Scenes | Objects | Descriptions |
|-------------|:------:|:-------:|:------------:|
| Train       | 562    | 7,875   | 36,665       |
| Validation  | 141    | 2,068   | 9,508        |
| Test        | 97     | 1,103   | 5,410        |

All splits are disjoint at the scene level. Each referred object is unambiguously annotated; multiple instances per class per scene may appear, necessitating nuanced referential disambiguation [1912.08830].

### File Organization and Schema

The dataset follows a rigorous directory structure:
```
ScanRefer/
  ├── pointclouds/
  │    ├── sceneXXXX_XX.ply    # reconstructed point cloud
  │    ├── sceneXXXX_XX.npz    # per-point features (RGB, normals, height, 128D multiview)
  ├── annotations/
  │    ├── train.json
  │    ├── val.json
  │    └── test.json
  └── objects/
       └── object_bboxes.json  # ground-truth boxes
```
Annotation entries are stored in JSON with the following schema:
```json
{
  "scene_id": "scene0101_00",
  "object_id": 17,
  "class_name": "nightstand",
  "bbox": [1.25,0.30,2.10, 1.75,0.60,2.40],
  "description": "This is a short nightstand next to the bed. It has two drawers and is dark brown.",
  "tokens": ["This","is","a","short","nightstand","next","to","the","bed",".","It","has","two","drawers","and","is","dark","brown","."],
  "split": "train"
}
```
Point cloud data (.ply) contain vertices with XYZRGB; per-point features are stored in .npz format for efficiency [1912.08830].

## 3. Annotation Protocol and Linguistic Phenomena

### Description Collection

Descriptions are obtained via a web-based 3D UI deployed to English-speaking crowd workers. Target objects are visually highlighted, with all other objects faded for context. Annotators observe the full 3D mesh as well as several 2D perspectives showing color and texture. Each object is described by five distinct workers, instructed to use at least two full sentences emphasizing physical appearance (color, shape, material) and spatial relations to other scene elements [1912.08830].

### Verification and Filtering

Post-collection, trained student annotators perform a verification phase:
- Presented with a scene and a referring expression, the verifier selects all matching objects.
- Descriptions matching zero or more than one object are discarded.
- Spelling and obvious wording mistakes are corrected; additional manual filtering yields a final set with ~2,823 invalid descriptions removed and ~2,129 lightly edited [1912.08830].

### Expression Types

The dataset exhibits a rich diversity of reference:
- **Spatial terms:** 98.7% of descriptions
- **Color attributes:** 74.7%
- **Shape attributes:** 64.9%
- **Explicit size terms:** 14.2%
- **Comparatives:** 672 instances (e.g., “smaller”, “bigger”)
- **Superlatives:** 2,734 (e.g., “largest”, “rightmost”)
- **Ordinal spatial:** frequent (“second chair from the door”)

This distribution directly enables fine-grained disambiguation and complex linguistic grounding [1912.08830].

## 4. Benchmarking, Metrics, and Baselines

### Localization Metrics

The principal evaluation metrics are:
- **3D Intersection-over-Union (IoU):**
  $$
  \mathrm{IoU} = \frac{\mathrm{Vol}(B_\text{pred} \cap B_\text{gt})}{\mathrm{Vol}(B_\text{pred} \cup B_\text{gt})}
  $$
- **Accuracy at k-IoU ($\text{Acc}@ k$):**
  Fraction of queries for which the top-1 predicted bounding box exceeds IoU threshold $k$ with the ground-truth. Common thresholds: $k=0.25, 0.50$.
- **Recall@5:** Proportion of samples for which any of the top 5 predictions exceed a 0.5 IoU threshold [2103.07894, 1912.08830, 2412.04383].

### Baseline Performance

On the official validation split, core baseline results are summarized as:

| Model                     | Acc@0.25 | Acc@0.50 |
|---------------------------|----------|----------|
| OracleCatRand             | 29.99%   | 29.76%   |
| OracleRefer               | 40.63%   | 40.06%   |
| VoteNetRand               | 10.00%   | 5.28%    |
| VoteNetBest               | 55.10%   | 54.33%   |
| SCRC + backprojection     | 18.70%   | 6.45%    |
| One-stage + backprojection| 20.38%   | 9.04%    |
| Ours (end-to-end, ScanRefer) | 41.19%   | 27.40%   |

Subsequent works such as 3DVG-T, InstanceRefer, BUTD-DETR, and EDA have advanced fully supervised state of the art to over 54% Acc@0.25 and 42% Acc@0.50. Zero-shot methods such as SeeGround leverage 2D vision–language models to reach 44.1%/39.4% on Acc@0.25/0.5, outperforming prior zero-shot approaches by 7.7 percentage points at Acc@0.25 [2412.04383].

### Standard Processing Pipeline

Practitioners typically:
1. Downsample to 40,000 scene points.
2. Project multi-view features or use raw RGB, normals, and height.
3. Encode the referring expression via pretrained GloVe embeddings and a GRU (256D).
4. Detect 3D proposals (e.g., M=256 via VoteNet).
5. Fuse proposal features with language embedding via MLP and score with softmax.
6. Apply 3D NMS and select top-1.
7. Evaluate against ground-truth using designated metrics [1912.08830].

## 5. Extensions and Related Developments

### Multi3DRefer

Multi3DRefer generalizes ScanRefer, addressing descriptions that refer to zero, single, or multiple target objects, thus reflecting more realistic and ambiguous reference scenarios. It encompasses 61,926 descriptions over the same 800 scenes and 11,609 objects, with:
- 6,688 zero-target descriptions
- 42,060 single-target
- 13,178 multiple-target (average 2 to 5.8 targets per prompt, max 32)

An F1-matching metric (using the Hungarian algorithm over pairwise IoU) is adopted for multi-object evaluation. Multi3DRefer also enriches vocabulary (unique word types: 7,077) and introduces controlled paraphrasing using ChatGPT for linguistic diversity [2309.05251].

### SeeGround and Zero-Shot Grounding

SeeGround exemplifies zero-shot approaches on ScanRefer, leveraging 2D vision–language models by representing 3D scenes as hybrid sets of rendered images combined with 3D-aware text prompts. This method achieves 39.4% Acc@0.5 without any ScanRefer training signal [2412.04383].

### 2D-3D Projected Tasks

Methods such as “Refer-it-in-RGBD” extract single-view RGB-D images from ScanRefer, enabling 2D-based visual grounding over projections, though such approaches are generally surpassed by direct 3D frameworks for disambiguation and localization performance [2103.07894].

## 6. Applications and Impact

ScanRefer is the canonical testbed for:
- Embodied AI research: language-guided robotic manipulation and navigation in human environments
- 3D visual grounding: disambiguating references to objects described verbally within real-world rooms
- Multi-modal understanding: aligning geometric, spatial, and linguistic information without explicit ontological constraints
- Benchmarking contrastive learning and transformer-based models for spatial language understanding

By providing diverse (in spatial relations, appearance, and semantic context) and richly annotated language–3D pairings, ScanRefer has enabled progress in open-vocabulary vision, improved 3D language grounding algorithms, and fostered the development of architectures supporting multi-reference and ambiguity-tolerant reasoning [1912.08830, 2412.04383, 2309.05251].

Source: https://www.emergentmind.com/topics/scanrefer-dataset