---
title: 'ScAN: Fine-Grained Scan-to-CAD Retrieval'
url: https://www.emergentmind.com/topics/scan-dataset
type: topic
---

# ScAN: Fine-Grained Scan-to-CAD Retrieval

ScAN is a dataset of fine-grained, human-annotated similarity rankings between real-world 3D scan objects and synthetic CAD models, introduced with the cross-domain retrieval framework described in "Joint Embedding of 3D Scan and CAD Objects" [1908.06989]. It is designed as a benchmark for CAD model retrieval from cluttered, noisy, and partial scans, and specifically targets the problem of learning a shared representation in which semantically similar objects from scan and CAD domains lie close together despite substantial lower-level geometric discrepancies [1908.06989].

## 1. Research setting and intended role

ScAN was introduced to support a joint embedding framework for 3D scan geometry and CAD geometry. The motivating premise is that 3D scan geometry and CAD models contain complementary information toward understanding environments, but that establishing a mapping between the two domains is difficult because scan objects are noisy, partial, and embedded in clutter, whereas CAD models are clean synthetic abstractions [1908.06989].

Within that setting, ScAN serves two functions. First, it provides supervision for learning a shared embedding space between scan objects and CAD models. Second, it provides a benchmark for evaluating CAD retrieval under fine-grained similarity judgments rather than coarse category matching alone. This is important because the target task is not merely to recover an object class such as chair or cabinet, but to retrieve CAD objects judged by human annotators to be geometrically similar to a specific scanned instance.

A common misconception is to treat ScAN as a generic 3D recognition dataset. Its actual design is narrower and more specialized: it is a ranked scan-to-CAD similarity dataset for retrieval from partial scans, not a full pipeline benchmark for simultaneous detection, segmentation, and retrieval. The underlying framework assumes pre-segmented scan objects, and the dataset is organized around that assumption.

## 2. Composition, scale, and geometric representation

ScAN contains 5,102 ranked similarity judgments over 3,979 unique scan objects and 7,650 unique CAD models spanning 31 ShapeNet-derived object categories, including chair, table, sofa, cabinet, bookshelf, bathtub, and trash bin [1908.06989]. The scan objects are segmented instances drawn from ScanNet, and the CAD models are drawn from ShapeNetCore.

| Split | Annotations | Unique scans / unique CADs |
|---|---:|---:|
| Train | 2,466 | 1,923 / 3,697 |
| Validation | 907 | 713 / 1,830 |
| Test | 1,729 | 1,343 / 3,113 |

The dataset stores both scan objects and CAD models as $32 \times 32 \times 32$ binary occupancy grids. Preprocessing centers and scales each object to fit within the unit cube, then voxelizes it by sampling surfaces and filling interior occupancy [1908.06989]. This representation standardizes the input format across the two domains and permits direct use of 3D CNN encoders.

The use of occupancy alone is methodologically significant. It makes the benchmark explicitly geometric, excluding appearance-based cues such as color and texture. This design isolates the cross-domain geometry problem, but it also constrains the information available to retrieval models.

## 3. Annotation procedure and ranking semantics

Scan objects are automatically segmented from ScanNet reconstructions using the provided instance labels. For each scan object, six CAD candidates are proposed by first finding nearest neighbors in a 128-d autoencoder latent space trained on ShapeNetCore and then randomly sampling 6 of the nearest 30 [1908.06989]. This candidate-generation step narrows annotation to plausible matches while retaining variation within the proposal set.

Annotation is performed through a web-based interface that displays a voxelized scan instance alongside the six CAD candidates. Annotators may rotate and zoom each model and may optionally click a hint to view the original RGB image and bounding box associated with the scan instance. Three trained annotators select up to three CADs per query and order them by perceived geometric similarity, with rank 1 denoting the most similar and rank 3 the least similar among the selected matches. All annotations were reviewed for consistency to yield the final 5,102 rankings.

The ranking protocol is deliberately finer-grained than binary relevance. For a given scan object, annotators are not only identifying plausible CAD analogues but also imposing an ordinal structure over them. This makes ScAN suitable for evaluating retrieval systems that must recover both relevant instances and their relative ordering.

Annotations are distributed in JSON or CSV form. The reported JSON structure is:

```json
{
  "scan_id": "string",
  "proposals": ["cad_id_1", "...", "cad_id_6"],
  "ranks": [{"cad_id": "cad_id_x", "rank": 1}]
}
```

An example entry includes a scan identifier such as `scene0101_00_obj012`, a six-element proposal list, and a ranked list such as `{cad_id:"02818832_0174", rank:1}, {cad_id:"03001627_0001", rank:2}` [1908.06989]. The release also includes train/validation/test split files and scripts to load voxel grids and annotation files.

## 4. Joint embedding formulation and evaluation measures

The dataset was introduced together with a 3D CNN-based method for learning a joint embedding between scans and CAD models. The framework uses separate 3D-CNN encoders for scan and CAD inputs and optimizes a triplet loss so that the embedded feature of a scan $S$ is closer to a matching CAD $C_p$ than to a nonmatching CAD $C_n$:

$$
L = \max\bigl(d(f(S), g(C_p)) - d(f(S), g(C_n)) + \mathrm{margin}, 0\bigr)
$$

where $f$ and $g$ are the scan and CAD encoders, $d(\cdot,\cdot)$ is Euclidean distance, and $\mathrm{margin} = 0.2$ [1908.06989].

To create a shared space in which scan objects and CAD models can interlace, the method uses a stacked hourglass approach to separate foreground and background from a scan object and transform it to a complete, CAD-like representation. This architectural choice addresses the domain gap between partial real-world scans and complete synthetic geometry.

ScAN supports evaluation at both embedding and retrieval levels. For embedding-space evaluation, the paper defines a confusion score that measures how well scan and CAD points locally mix among the $k$ nearest neighbors:

$$
\mathrm{confusion} = 0.5 \times \left[
\frac{1}{k \cdot |\mathrm{scans}|} \sum_{\mathrm{Scans}} |\mathrm{CAD\text{-}nbrs}|
+
\frac{1}{k \cdot |\mathrm{CADs}|} \sum_{\mathrm{CADs}} |\mathrm{scan\text{-}nbrs}|
\right]
$$

A value of $0.5$ indicates perfect domain mixing among the $k$ nearest neighbors [1908.06989].

For retrieval evaluation, ScAN uses two task-specific criteria. Top-1 retrieval accuracy is counted as correct when the retrieved CAD belongs to the human-annotated similar set. Ranking quality is defined, for each query with $n$ annotated matches where $n \leq 3$, as the fraction of correctly ordered CADs in the top-$n$ predictions. These measures separate coarse relevance from ordinal fidelity.

## 5. Benchmark results and empirical behavior

On the embedding confusion measure with $k=10$, the reported joint embedding achieves approximately $0.50$, corresponding to perfect mix, whereas prior descriptors including FPFH, SHOT, and PointNet lie below $0.30$ [1908.06989]. In the intended interpretation of the metric, this means that scan and CAD embeddings are locally interleaved rather than segregated by domain.

On ranked retrieval, the full model achieves top-1 retrieval accuracy of $0.39$ class-average and $0.43$ instance-average, compared with $0.33$ and $0.31$ for a 3D-CNN baseline. Top-1 category accuracy is reported as $0.68$ for the proposed method versus $0.57$ for the prior best 3D-CNN. Ranking quality at the instance level is reported as $0.16$ versus $0.13$ for the 3D-CNN baseline [1908.06989].

These results are consistent with the abstract’s statement that the learned joint embedding outperforms the previous state of the art for CAD model retrieval by 12% in instance retrieval accuracy. The quantitative pattern also indicates that the benchmark distinguishes between several levels of success: category recognition, instance-level retrieval, and within-match ranking quality. A plausible implication is that ScAN is not reducible to category retrieval alone; the strongest improvements are meaningful precisely because the benchmark encodes human judgments at instance granularity.

Another interpretive point concerns the confusion score. A score near $0.5$ reflects strong local domain mixing in the learned embedding, but retrieval performance is still measured separately. This suggests that cross-domain interlacing is necessary but not, by itself, a complete account of retrieval quality.

## 6. Uses, constraints, and scope of the benchmark

The reported applications of ScAN include fine-grained CAD retrieval from real-world scans for augmented reconstruction, robotic grasp planning, and virtual reality scene modeling [1908.06989]. It also functions as a benchmark for learning cross-domain embeddings between noisy partial scans and clean synthetic geometry, and as a component for systems that complete or replace scanned objects with CAD proxies.

Its limitations are explicit. Only geometry in the form of occupancy is used, so color and texture cues are ignored. The voxel resolution of $32^3$ limits the capture of fine detail. The benchmark assumes pre-segmented object proposals, meaning that joint detection and retrieval are not addressed. In addition, ranking is limited to relatively small candidate pools, specifically 6 proposals plus 100 distractors [1908.06989].

These constraints define the dataset’s proper scope. ScAN is most suitable for studying representation learning and retrieval across the scan/CAD domain boundary under controlled voxelized inputs. It is less suitable as a benchmark for appearance-aware retrieval, fine-scale geometric discrimination beyond $32^3$ occupancy, or end-to-end scene understanding that jointly localizes and retrieves objects.

Researchers can download the voxelized scan and CAD data, annotation files, and evaluation scripts to reproduce benchmarks or train new embedding and retrieval models. In that role, ScAN occupies a specific niche within 3D vision: a supervised, ranked, human-annotated benchmark for fine-grained scan-to-CAD similarity under real-world partiality and synthetic completeness [1908.06989].

Source: https://www.emergentmind.com/topics/scan-dataset