OpenScan Benchmark

• OpenScan Benchmark is a large-scale testbed for evaluating 3D scene understanding through free-form linguistic queries that capture visual, functional, and commonsense attributes.
• It expands traditional 3D benchmarks by incorporating detailed attribute annotations from indoor scenes, enabling analysis beyond discrete class labels.
• Baseline analyses reveal marked performance drops on abstract attribute queries, highlighting the need for improved multimodal and semantic reasoning in current models.

OpenScan is a large-scale benchmark specifically designed for evaluating the capabilities of machine learning systems in Generalized Open-Vocabulary 3D Scene Understanding (GOV-3D). Unlike traditional open-vocabulary 3D (OV-3D) benchmarks that restrict queries to discrete object class names, OpenScan expands the task to include free-form linguistic queries that target not only class labels but also fine-grained, object-specific attributes across a diverse array of semantic, functional, and commonsense dimensions. This benchmark exposes significant limitations in current 3D scene-understanding systems by requiring a more holistic understanding of 3D environments, thereby facilitating the development and assessment of models with deeper semantic and commonsense reasoning abilities (Zhao et al., 2024).

1. Problem Formulation: GOV-3D versus OV-3D

The conventional OV-3D task provides a model with a 3D point-cloud scene $P$, associated RGB frames $I$, and a vocabulary $\{c_t\}$ of object class names. The model predicts a set of 3D masks $Q = \mathbb{M}(P, I, \{c_t\})$, mapping queries to semantic segmentations interpreted as class-based object detections.

GOV-3D generalizes this paradigm by introducing an attribute vocabulary $\{a_k\}$, with each $a_k$ formulated as a free-form natural-language description—e.g., “is made of wood,” “can be worn on the head.” The model's objective is to return $Q = \mathbb{N}(P, I, \{a_k\})$: a set of 3D masks corresponding to entity subsets best matching each attribute query. This task necessitates reasoning over abstract, visual, and functional attributes that transcend mere visual class boundaries and require multimodal and commonsense understanding (Zhao et al., 2024).

2. Dataset Characteristics and Annotation Protocol

OpenScan is constructed atop the ScanNet200 dataset, comprising 1,513 richly annotated indoor scenes. It introduces 153,644 attribute annotations encompassing 341 distinct attributes, averaging 3.15 attributes per object and 101.6 per scene. Attributes are systematically organized into eight linguistic aspects:

Aspect	Count	Example
Affordance	104	"is used for sitting"
Property	19	"is bright"
Type	96	"is a kitchen appliance"
Manner	21	"can be played"
Synonym	16	"is related to image"
Requirement	28	"requires balance to ride"
Element	47	"has 88 keys"
Material	10	"is made of wood"

The multi-stage annotation process involves:

1. Automated extraction of commonsense relations from ConceptNet for each ScanNet200 class (edges $(c,r,w,v)$), followed by selection of top-weighted attributes per relation.
2. Manual labeling of purely visual/material attributes via a web interface displaying both 3D meshes and 2D frames.
3. Grouping of each phrase under a predefined aspect, with subsequent human verification and pruning—removing ambiguous or near-duplicate entries and restricting to one attribute per aspect per object, reducing the initial candidate pool from 528 to 341.
4. Attribute query generation by replacing object class in each phrase with “this term” to create compositional queries (e.g., “this term is used for sitting”) (Zhao et al., 2024).

3. Evaluation Protocols and Metrics

OpenScan evaluates GOV-3D models through both semantic and instance segmentation tasks:

• Semantic Segmentation:
  • For each attribute $a$, performance is quantified by:

  $$\mathrm{IoU}_a = \frac{\mathrm{TP}_a}{\mathrm{TP}_a + \mathrm{FP}_a + \mathrm{FN}_a}$$

  $$\mathrm{Acc}_a = \frac{\mathrm{TP}_a}{\mathrm{TP}_a + \mathrm{FN}_a}$$ - Macro-averages over all attributes yield mean IoU (mIoU) and mean accuracy (mAcc).

• Instance Segmentation:

  • Detections are matched to ground truth masks at multiple IoU thresholds. Average Precision (AP) is computed as:

  $$\mathrm{AP}_t = \int_0^1 \mathrm{Precision}(\mathrm{Recall}=r) \, dr$$ - AP values at thresholds 0.25 ($AP_{25}$), 0.50 ($AP_{50}$), and mean AP over $t=50:5:95$ are reported.

This rigorous, multi-faceted evaluation probes not only visual but also functional and commonsense scene understanding capabilities (Zhao et al., 2024).

4. Baseline Methods and Quantitative Analysis

OpenScan provides extensive baselines across both instance and semantic segmentation modalities:

• 3D Instance Segmentation Baselines:

  • OpenMask3D, SAI3D, MaskClustering, Open3DIS.
• 3D Semantic Segmentation Baselines:
  • OpenScene, PLA, RegionPLC.

Performance drops significantly when moving from class labels to attribute-based queries. For all 341 attributes (mean over aspects):

Model	AP (Attributes)	AP (Classes)
OpenMask3D	9.9%	15.4%
SAI3D	7.7%	12.7%
MaskClustering	8.1%	12.0%
Open3DIS	15.8%	23.7%

Aspect-wise, Open3DIS achieves the highest AP for material (28.3%) and synonym (26.7%), but lowest for affordance (11.9%) and property (12.8%). Semantic segmentation performance on attributes nearly collapses (OpenScene: mIoU = 0.45%, mAcc = 1.87% vs. mIoU = 47.5%, mAcc = 70.7% on classes) (Zhao et al., 2024).

Qualitatively, baseline models can localize the correct object class but fail to segment for abstract attribute queries (e.g., “this term has 88 keys”). Vision-and-LLMs such as CLIP demonstrate lower image-text similarity for attribute phrases compared to class names, indicating underdeveloped attribute representations within contemporary visual-LLMs.

5. Empirical Findings and Diagnostic Insights

Key diagnostic results from OpenScan reveal core limitations of current methods:

• Abstract Attribute Comprehension: State-of-the-art OV-3D methods see substantial performance degradation when evaluated on attribute queries—especially functional or commonsense (affordance, requirement)—while visual attributes (material) and class-name synonyms are less affected.
• Vocabulary Scaling Ineffectiveness: Scaling the class vocabulary during pre-training (tested with RegionPLC, varying $S=\{10,12,15,150,170\}$) yields negligible improvements in attribute segmentation performance, up to slight gains for material. This suggests that expanding class names does not impart the required attribute-level knowledge or reasoning.
• Impact of Query Formulation: Using compositional templates (e.g., “this term is made of wood”) increases AP by 1–2 points over using bare attribute keywords, highlighting the value of explicit relational context for CLIP-style representations.

These findings indicate that current systems implicitly rely on visual pretraining biases and lack the abstraction necessary for robust attribute-grounded 3D understanding (Zhao et al., 2024).

6. Prospects for Future Research

OpenScan highlights several promising research avenues:

• Attribute-Aware 3D Proposals: Advances are needed in developing class-agnostic mask generation mechanisms capable of abstract attribute query handling.
• Commonsense Integration: Incorporating external knowledge graphs or prompting LLMs to map abstract attribute queries into likely object class sets represents a plausible path forward (e.g., mapping “has 88 keys” to {“piano”}).
• Richer Training Resources: Extending attribute annotation to new or more complex 3D datasets (ScanNet++, Matterport3D) or capturing richer real-world scenes can foster more comprehensive learning and evaluation environments.
• Unified 2D–3D–Language Reasoning: Leveraging advances in multimodal transformers and unified architectures holds potential for bridging the current reasoning gap by more effectively integrating cross-modal cues.

In total, OpenScan establishes a rigorous, attribute-centered testbed and surfaces the current gap between class-level object detection and the nuanced, knowledge-driven 3D reasoning needed for generalized open-vocabulary scene understanding (Zhao et al., 2024).