SpaCeFormer-3M: Open-Vocabulary 3D Dataset

• The paper presents SpaCeFormer-3M, a comprehensive dataset leveraging multi-view mask clustering and vision-language captioning for open-vocabulary 3D instance segmentation.
• The dataset aggregates 7,361 scenes, 604,127 3D instances, and 3,020,635 multi-view captions, achieving up to 21× higher mask recall than prior single-view methods.
• The work underpins proposal-free architectures, enabling zero-shot segmentation with enhanced geometric and semantic consistency critical for efficient end-to-end training.

Searching arXiv for the main paper and a few directly related works mentioned in the provided material. SpaCeFormer-3M is a large-scale dataset for open-vocabulary 3D instance segmentation introduced alongside SpaCeFormer in “SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation” (Choy et al., 22 Apr 2026). It is defined by multi-view-consistent 3D instance masks and associated natural-language captions, and is designed to address deficiencies in prior pseudo-label pipelines that rely on single-view 2D-to-3D lifting, fragmented masks, inconsistent captions, and external region proposals. The dataset contains 7,361 scenes, 604,127 3D instances, and 3,020,635 captions, with an average of 5 captions per instance and 46,571,420 total words; its stated role is to enable end-to-end, proposal-free training for open-vocabulary 3D instance segmentation at substantially higher mask recall than prior single-view pipelines (Choy et al., 22 Apr 2026).

1. Definition and scope

SpaCeFormer-3M is presented as “the largest open-vocabulary 3D instance segmentation dataset,” with “3.0M multi-view-consistent captions over 604K instances from 7.4K scenes” (Choy et al., 22 Apr 2026). In the detailed statistics, these totals are specified as 7,361 scenes, 604,127 instances, and 3,020,635 captions, with an average caption length of 15.5 words and an average of 82.07 masks per scene (Choy et al., 22 Apr 2026). The dataset aggregates scenes from ScanNet, ScanNet++, ARKitScenes, and Matterport3D.

Its central design premise is that open-vocabulary 3D instance segmentation requires consistency in both geometry and semantics. Geometric consistency is implemented through multi-view mask aggregation into complete 3D instances. Semantic consistency is implemented through multi-view captioning that emphasizes intrinsic properties such as shape, color, material, and size, as well as consistent spatial relationships across views (Choy et al., 22 Apr 2026).

The dataset is positioned against prior pseudo-label datasets such as RegionPLC and Mosaic3D, which are described as single-view methods producing fragmented masks and inconsistent captions. A plausible implication is that SpaCeFormer-3M is not merely larger in volume, but structurally different in how it couples 3D mask construction with language supervision (Choy et al., 22 Apr 2026).

2. Data sources and composition

SpaCeFormer-3M aggregates data from four RGB-D corpora: ScanNet with 1,201 scenes, ScanNet++ with 223 scenes, ARKitScenes with 4,497 scenes, and Matterport3D with 1,440 scenes (Choy et al., 22 Apr 2026). Preprocessing includes pose and intrinsic conversion, depth backprojection, and quality filtering for multi-view consistency.

The scale statistics reported for the dataset are as follows:

Quantity	Value
Scenes	7,361
Instances (3D masks)	604,127
Captions	3,020,635
Words	46,571,420
Average captions per instance	5
Average caption length	15.5 words
Average masks per scene	82.07

The mean masks-per-scene value is reported as variable by source, with ScanNet++ given as an example at approximately 122 masks per scene (Choy et al., 22 Apr 2026). This suggests substantial scene-level object density and correspondingly broad instance coverage.

Because the dataset spans multiple source domains, it is also used to support evaluation under held-out scenes and new taxonomies. The paper characterizes the resulting system as fully open-vocabulary and reports zero-shot benchmarking on Replica as well as transfer to ScanNet++ and Matterport3D settings (Choy et al., 22 Apr 2026). A plausible implication is that the diversity of source data is intended to reduce overfitting to a single annotation ontology or capture regime.

3. Construction pipeline

The dataset construction procedure has two primary stages: multi-view mask clustering and multi-view vision-LLM captioning (Choy et al., 22 Apr 2026).

Multi-view mask clustering

The first stage addresses the problem that prior pseudo-label pipelines use single-view 2D-to-3D lifting, producing fragmented and incomplete 3D instance masks. SpaCeFormer-3M instead uses MaskClustering to aggregate 2D masks detected by foundation models, such as SAM2, from multiple views into complete, geometry-consistent 3D instances (Choy et al., 22 Apr 2026).

The procedure described in the paper includes:

• aggregating RGB-D frames from ScanNet, ScanNet++, ARKitScenes, and Matterport3D;
• filtering and sampling diverse, representative views to maximize coverage and minimize redundancy through overlap-based greedy selection;
• detecting 2D masks in all views with SAM2;
• lifting 2D masks to 3D using camera extrinsics and intrinsics;
• grouping mask fragments into instances using multi-view spatial clustering based on proximity, containment, and visibility, with strict consensus for high precision (Choy et al., 22 Apr 2026).

This construction strategy is explicitly meant to yield near-complete, geometry-consistent instances suitable for proposal-free end-to-end training.

Multi-view VLM captioning

The second stage addresses the problem that single-view captions are often inconsistent and view-dependent. SpaCeFormer-3M therefore uses a structured multi-view prompting strategy to generate captions that are multi-view-consistent and focused on intrinsic object properties (Choy et al., 22 Apr 2026).

For each 3D instance, the method selects up to $K$ diverse views via k-medoids and mask visibility scores. It prepares both contextual and masked crops for each selected view, then prompts a vision-LLM, exemplified by Gemma-3, with all selected images simultaneously. The prompt is designed to elicit descriptions of intrinsic properties—shape, color, material, and size—and consistent spatial relationships. Multiple diverse captions are generated per instance, with a default of 5 captions (Choy et al., 22 Apr 2026).

The dataset thus combines geometric consolidation and language normalization. This pairing is central to its function in open-vocabulary supervision: language is not attached to arbitrary 2D fragments, but to multi-view 3D instances.

4. Mask quality and comparison to prior datasets

The paper reports a mask-quality analysis on ScanNet that compares SpaCeFormer-3M to Mosaic3D (Choy et al., 22 Apr 2026). The comparison is summarized below.

Dataset	Masks/scene	Mean Max IoU	Precision @0.5	Recall @0.5	IoU > 0.5
Mosaic3D	16.1	0.247	4.8%	2.5%	3.8%
SpaCeFormer-3M	65.2	0.251	33.6%	54.3%	25.9%

The most emphasized result is “21× higher mask recall at IoU>0.5 (54.3% vs 2.5%)” relative to Mosaic3D (Choy et al., 22 Apr 2026). Precision at 0.5 also rises from 4.8% to 33.6%, while masks per scene increase from 16.1 to 65.2.

The mean max IoU changes only modestly, from 0.247 to 0.251. This pattern suggests that the principal gain lies less in marginal improvements to best-overlap quality than in much broader coverage of valid instances. The paper interprets this as evidence that SpaCeFormer-3M achieves near-complete, geometry-consistent instances, a property it identifies as necessary for effective training of proposal-free architectures (Choy et al., 22 Apr 2026).

Relative to RegionPLC and Mosaic3D, the stated advantages of SpaCeFormer-3M are: largest scale to date in terms of 3D mask-caption pairs; significantly higher mask recall and precision; multi-view consistency in geometry and captions; and support for proposal-free architectures and true open-vocabulary segmentation (Choy et al., 22 Apr 2026).

5. Role in the SpaCeFormer model

SpaCeFormer-3M is used to train SpaCeFormer end-to-end without ground-truth 3D labels. The paper states that the architecture uses only SpaCeFormer-3M pseudo-labels and requires no GT 3D mask supervision (Choy et al., 22 Apr 2026).

SpaCeFormer itself is described as a proposal-free space-curve transformer with four relevant components:

1. a Space-Curve Transformer backbone combining spatial window attention with Morton-curve serialization;
2. 3D Rotary Positional Embeddings (RoPE) in both backbone and decoder;
3. a proposal-free decoder that predicts instance masks directly from learned queries;
4. end-to-end training on SpaCeFormer-3M pseudo-labels (Choy et al., 22 Apr 2026).

The interaction between dataset and architecture is explicit. The dataset is intended to provide sufficiently complete instance masks to make direct query-based mask prediction feasible. Prior fragmented pseudo-labels are described as poor for end-to-end training, whereas SpaCeFormer-3M is said to enable “proposal-free architectures and true open-vocabulary segmentation” (Choy et al., 22 Apr 2026).

The model’s mask prediction is given by

$$\mathbf{M} = \sigma(\mathbf{F}(\mathbf{W}_M \mathbf{Q}^{(T)})^\top),$$

where the decoder iteratively refines learned queries over $T=3$ iterations using cross-attention with RoPE, self-attention, and a feed-forward network (Choy et al., 22 Apr 2026). The decoder also predicts CLIP-aligned text embeddings and foregroundness logits for open-vocabulary classification and instance confidence.

A plausible implication is that the dataset’s multi-view caption consistency is particularly important for the CLIP-aligned component, because noisy or view-specific captions would weaken the semantic target for open-vocabulary alignment.

6. Architectural couplings enabled by the dataset

The paper links the dataset to several architectural choices in SpaCeFormer, especially those relevant to mask fidelity and spatial reasoning (Choy et al., 22 Apr 2026).

Spatial window attention and Morton serialization

The backbone combines spatial window attention with Morton-curve serialization. Spatial window attention divides 3D space into cubic windows that preserve spatial locality, which the paper describes as critical for sharp masks. Morton-curve serialization converts voxels into 1D segments for scalable attention, providing diversity in attention patterns and helping at coarser levels or when sparsity leaves windows under-filled (Choy et al., 22 Apr 2026).

An ablation table reports:

Attention type	AP	mAP
Window Only	0.2547	0.095
Morton Only	0.2331	0.094
Window + Morton	0.2517	0.111

The paper further states that the combined scheme yields “28.6% lower within-window pairwise distance compared to Morton-only,” which it associates with more local features and improved mask boundary accuracy (Choy et al., 22 Apr 2026).

RoPE-enhanced decoder

The decoder uses 3D RoPE, with the inner-product relation written as

$$\langle\mathbf{q}_i, \mathbf{k}_j\rangle = \mathbf{q}_i \mathbf{R}_{\Theta, \mathbf{p}_j - \mathbf{p}_i} \mathbf{k}_j^\top$$

where $\mathbf{R}_{\Theta, \Delta\mathbf{p}}$ is a block-diagonal rotation matrix and the angle is given per dimension and frequency by

$$\theta_d^{(m)} = \frac{\Delta p_d}{\text{base}^{2m/d_h}}.$$

The paper states that RoPE is important for distinguishing instances at different locations and reports that it outperforms standard sinusoidal or relative positional encodings, with “+11–28% improvement in mAP” (Choy et al., 22 Apr 2026). The specific decoder positional-encoding ablation is:

Decoder positional encoding	mAP
None	5.97
Absolute PE	5.95
Positional Bias	6.46
RoPE	7.60

These results concern the model, but they also clarify the dataset’s function: SpaCeFormer-3M supplies pseudo-labels of sufficient geometric integrity for the model to benefit from fine-grained spatial inductive biases rather than from proposal heuristics.

7. Empirical impact and interpretation

Using SpaCeFormer-3M, SpaCeFormer reports 11.1 zero-shot mAP on ScanNet200, characterized as a 2.8× improvement over the prior best proposal-free method; 22.9 mAP on ScanNet++; and 24.1 mAP on Replica (Choy et al., 22 Apr 2026). The paper states that these results surpass prior methods, including methods using multi-view 2D inputs, while operating at 0.14 seconds per scene and running “2–3 orders of magnitude faster than multi-stage 2D+3D pipelines” (Choy et al., 22 Apr 2026).

The quantitative summary reported in the paper is:

Benchmark	mAP	Comparison	Latency
Replica (ZS)	24.1	Surpasses Open-YOLO 3D (23.7 mAP); approaches SOLE (24.7 mAP, but with GT)	0.14s
ScanNet++	22.9	Exceeds OpenTrack3D (20.6 mAP, multi-view 2D/3D)	0.14s
ScanNet200	11.1	2.8× better than Mosaic3D+Decoder (3.9 mAP)	0.14s

These outcomes are reported for SpaCeFormer, not for the dataset in isolation. Nevertheless, the paper explicitly attributes the viability of proposal-free end-to-end training to the pseudo-label quality of SpaCeFormer-3M (Choy et al., 22 Apr 2026). This suggests that the dataset is not merely a benchmarking resource but a training substrate that changes the feasible model class: proposal-free learned-query decoders become practical when pseudo-labels are sufficiently complete and caption supervision is multi-view-consistent.

A common misconception in this area is that open-vocabulary performance depends primarily on stronger 2D foundation models or multi-stage 2D+3D aggregation. The paper argues instead that the central bottleneck is the quality and consistency of 3D pseudo-labels, and presents SpaCeFormer-3M as evidence that higher-recall multi-view 3D supervision can support a faster end-to-end architecture without external proposals (Choy et al., 22 Apr 2026). Another plausible implication is that improvements in dataset construction may have system-level effects comparable to, or larger than, changes in decoder design alone.

In that sense, SpaCeFormer-3M occupies a specific place in open-vocabulary 3D instance segmentation research: it formalizes a shift from fragmentary single-view pseudo-labeling toward multi-view-consistent 3D masks and captions, with the stated consequence of enabling accurate, zero-shot, proposal-free segmentation at low latency (Choy et al., 22 Apr 2026).