SpaCeFormer: Open-Vocabulary 3D Segmentation

• The paper introduces SpaCeFormer, a proposal-free 3D instance segmentation framework that uses space-curve transformers and RoPE-enhanced decoding for direct mask prediction from point clouds.
• It leverages a large pseudo-labeled 3D-text training corpus (SpaCeFormer-3M) to ensure multi-view consistency and robust natural language alignment.
• The system outperforms multi-stage 2D+3D pipelines in both speed and accuracy, enabling interactive segmentation for robotics, AR/VR, and spatial computing applications.

Searching arXiv for the relevant SpaCeFormer paper and closely related naming variants to ground the article. SpaCeFormer is a proposal-free open-vocabulary 3D instance segmentation system for reconstructed 3D scenes, designed to predict instance masks directly from 3D input while aligning instance features with natural-language descriptions (Choy et al., 22 Apr 2026). It combines a space-curve transformer backbone, a RoPE-enhanced query decoder, and a large pseudo-labeled 3D-text training corpus called SpaCeFormer-3M. In the reported formulation, the method operates on point clouds or sparse voxel grids, uses no 2D RGB input at inference, and targets the central bottlenecks identified in prior open-vocabulary 3D instance segmentation: multi-stage 2D+3D pipelines with high latency, fragmented pseudo-labels from single-view lifting, and loss of local spatial coherence under purely serialization-based 3D attention (Choy et al., 22 Apr 2026).

1. Problem setting and motivation

Open-vocabulary 3D instance segmentation requires a model to detect object instances in 3D, predict a 3D mask for each instance, and assign each instance a semantic label from arbitrary natural language rather than a fixed taxonomy (Choy et al., 22 Apr 2026). In the formulation reported for SpaCeFormer, the model outputs instance masks $\mathbf{m}_q \in [0,1]^N$ and language-aligned instance features $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$, which are matched to text embeddings at inference by similarity.

The paper situates the task in robotics, embodied AI, AR/VR, and spatial computing, where category vocabularies are not closed and queries may be compositional or descriptive, such as object references involving appearance or location. Prior methods are divided into two broad families. The first is multi-stage 2D+3D pipelines, exemplified in the paper by OpenMask3D, Open3DIS, OpenTrack3D, Any3DIS, and SAI3D, which run 2D foundation models on multiple RGB views, lift 2D proposals into 3D, cluster them, and classify them using CLIP or VLM features. The second is pseudo-labeled end-to-end 3D methods, such as RegionPLC, Mosaic3D, and OpenIns3D-style systems, which train 3D models from pseudo-labels lifted from 2D. The paper identifies three recurring limitations: heavy multi-stage heuristics, high latency that is often hundreds of seconds per scene, and fragmentation or inconsistency in pseudo-labels when 3D masks and captions are derived from single views (Choy et al., 22 Apr 2026).

SpaCeFormer is presented as a response to those limitations. Its central claim is that an effective open-vocabulary 3D instance segmentation system should be 3D-only at inference, proposal-free, and trained from pseudo-labels that are multi-view consistent rather than single-view fragmented. A plausible implication is that the design prioritizes end-to-end geometric coherence and deployment-time efficiency over dependence on external proposal generators.

2. System organization and formal outputs

SpaCeFormer consists of three coupled components: the SpaCeFormer-3M dataset, a space-curve transformer backbone, and a RoPE-enhanced proposal-free decoder (Choy et al., 22 Apr 2026). The dataset provides pseudo-labeled 3D instances and multi-view captions; the backbone produces spatially coherent voxel features and a CLIP-aligned feature map; the decoder uses learned queries to predict masks, foregroundness, and language-aligned instance features without external proposals.

The input is a 3D point cloud $\mathcal{P} = \{(p_i,f_i)\}_{i=1}^M$, where $p_i \in \mathbb{R}^3$ and $f_i$ contains point features such as color. The point cloud is voxelized into a sparse grid $\mathcal{V}$. The backbone yields per-voxel features at the finest decoder level, which are interpolated back to points to produce $\mathbf{F}\in\mathbb{R}^{M\times D}$, together with CLIP-aligned voxel features $$\mathbf{Z} \in \mathbb{R}^{N \times D_{\text{CLIP}}}$$. The decoder initializes $Q$ learned queries $\mathbf{Q}^{(0)} \in \mathbb{R}^{Q\times D}$, with the paper giving $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$0 as an example, and iteratively refines them through RoPE-enhanced cross-attention, self-attention, and feed-forward blocks (Choy et al., 22 Apr 2026).

At the output stage, the final queries $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$1 drive three heads. The mask head predicts

$\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$2

so that each query defines a soft mask over points. The foreground/background head predicts $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$3, and the instance language head predicts

$\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$4

This architecture is explicitly proposal-free: the learned queries themselves act as instance slots rather than consuming region proposals from a separate network (Choy et al., 22 Apr 2026).

The term “space-curve” refers to the joint use of spatial window attention and Morton-curve serialization. The former preserves local geometric neighborhoods through fixed 3D windows; the latter preserves the scalable token ordering strategies used in sparse voxel transformers. This dual structure is the defining organizational principle of the model.

3. Backbone and decoder architecture

The backbone is a U-Net-like sparse voxel transformer with a sparse Conv3D stem and transformer blocks. For each non-empty voxel $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$5, the input feature is formed by average pooling point features inside the voxel,

$\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$6

followed by a learned embedding $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$7 (Choy et al., 22 Apr 2026). Each block applies a sparse convolutional residual stem, multi-head self-attention, and an FFN: $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$8

$\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$9

$\mathcal{P} = \{(p_i,f_i)\}_{i=1}^M$0

Encoder and decoder stages follow a standard hierarchical U-Net pattern with skip connections and up/down convolution.

The distinctive part of the backbone is the coexistence of two attention schemes. Spatial window attention partitions voxels into fixed 3D windows of size $\mathcal{P} = \{(p_i,f_i)\}_{i=1}^M$1 and computes MHSA within each window. Because the windows are defined in coordinate space, they have fixed geometric extent but variable occupancy under sparsity. The paper reports that spatial window attention reduces average within-window pairwise distance by $\mathcal{P} = \{(p_i,f_i)\}_{i=1}^M$2 compared to Morton-only attention, indicating improved spatial coherence (Choy et al., 22 Apr 2026). To avoid hard boundary artifacts, the model uses randomized shifted windows, offset by half the window size along combinations of axes.

Morton-curve serialization provides the complementary “curve” component. Voxels are serialized by Morton or Z-order indexing and divided into fixed-length segments; attention is computed within each segment. This yields fixed token length but variable spatial extent, introducing longer-range structured mixing than purely windowed attention. The reported ablation indicates that neither window-only nor Morton-only attention is optimal; their combination produces the best class-specific performance (Choy et al., 22 Apr 2026).

RoPE supplies the positional mechanism in both the backbone and decoder. For voxel positions $\mathcal{P} = \{(p_i,f_i)\}_{i=1}^M$3 and $\mathcal{P} = \{(p_i,f_i)\}_{i=1}^M$4, attention depends on a relative-position rotation: $\mathcal{P} = \{(p_i,f_i)\}_{i=1}^M$5 The rotation matrix $\mathcal{P} = \{(p_i,f_i)\}_{i=1}^M$6 is block-diagonal across $\mathcal{P} = \{(p_i,f_i)\}_{i=1}^M$7, $\mathcal{P} = \{(p_i,f_i)\}_{i=1}^M$8, and $\mathcal{P} = \{(p_i,f_i)\}_{i=1}^M$9, so the encoding is naturally decomposed into spatial axes. The paper attributes two properties to this design: relative positional encoding and translation invariance (Choy et al., 22 Apr 2026).

The decoder is DETR-style and proposal-free. At each iteration, it samples $p_i \in \mathbb{R}^3$0 scene points for efficiency, applies RoPE-enhanced cross-attention from learned queries to sampled scene features, then RoPE-enhanced self-attention among queries, and finally an FFN. The decoder weights are shared across $p_i \in \mathbb{R}^3$1 iterations, with $p_i \in \mathbb{R}^3$2 given as an example. The use of all points is deferred until the final mask projection step. This suggests that computational savings are concentrated in the iterative refinement phase rather than in the final mask evaluation.

4. SpaCeFormer-3M dataset and pseudo-label construction

SpaCeFormer-3M is described as the largest open-vocabulary 3D instance segmentation dataset in the paper’s setting, containing $p_i \in \mathbb{R}^3$3 3D instances and $p_i \in \mathbb{R}^3$4 captions over $p_i \in \mathbb{R}^3$5 scenes from ScanNet, ScanNet++, ARKitScenes, and Matterport3D (Choy et al., 22 Apr 2026). The reported aggregate statistics are $p_i \in \mathbb{R}^3$6 masks, $p_i \in \mathbb{R}^3$7 captions, about $p_i \in \mathbb{R}^3$8 masks per scene, and average caption length of about $p_i \in \mathbb{R}^3$9 words.

The dataset is built in two stages: multi-view mask clustering and multi-view VLM captioning. In the first stage, geometrically diverse video frames are selected using an overlap criterion between reconstructed point sets from two frames: $f_i$0 A new frame is selected only when overlap falls below a threshold. SAM2 is then run on selected RGB frames to obtain 2D masks, which are lifted into 3D using depth and camera pose. Those lifted fragments are aggregated by MaskClustering using spatial proximity, visibility, and consensus, with strict filters of visibility $f_i$1 and consensus $f_i$2 (Choy et al., 22 Apr 2026).

The reported comparison to Mosaic3D on ScanNet train scenes shows why this matters. Mosaic3D yields $f_i$3 masks per scene, mean best IoU $f_i$4, [email protected] of $f_i$5, and recall@0.5 of $f_i$6. SpaCeFormer-3M yields $f_i$7 masks per scene, mean best IoU $f_i$8, [email protected] of $f_i$9, and [email protected] of $\mathcal{V}$0, which the paper summarizes as $\mathcal{V}$1 higher recall at IoU $\mathcal{V}$2 (Choy et al., 22 Apr 2026). This is a central empirical premise of the method: proposal-free decoding benefits from pseudo-labels that are substantially less fragmented.

In the second stage, captions are generated by multi-view prompting rather than single-view captioning. For each 3D instance, the system retrieves all source views, selects up to $\mathcal{V}$3 representative views using k-medoids over time with visibility weighting, constructs two image formats per view—cropped context and background-removed object—and queries a vision-LLM with a structured prompt encouraging intrinsic, view-consistent descriptions (Choy et al., 22 Apr 2026). The resulting captions are intended to emphasize material, shape, size, and stable relational properties rather than camera-specific phrasing. The dataset includes five captions per mask on average.

5. Training objectives and open-vocabulary alignment

Training uses pseudo-label supervision from SpaCeFormer-3M and no ground-truth 3D annotations (Choy et al., 22 Apr 2026). The total objective is

$\mathcal{V}$4

Predicted queries and ground-truth pseudo-instances are first paired through bipartite Hungarian matching with a cost that combines classification and mask similarity terms. The mask loss is the sum of BCE and Dice losses over matched pairs: $\mathcal{V}$5 Foregroundness is trained with a standard binary cross-entropy over whether a query is matched to any object.

The open-vocabulary component is provided by a CLIP-style contrastive alignment between predicted instance features and caption embeddings. For a matched query-caption pair $\mathcal{V}$6, similarity is defined as

$\mathcal{V}$7

and optimized with an InfoNCE objective: $\mathcal{V}$8 This makes the predicted instance representation directly comparable with arbitrary text embeddings at inference (Choy et al., 22 Apr 2026).

Zero-shot labeling then follows a simple retrieval rule. Candidate class names or free-form prompts are embedded once; each predicted instance feature $\mathcal{V}$9 is matched to the text embedding with maximal cosine similarity,

$\mathbf{F}\in\mathbb{R}^{M\times D}$0

The paper explicitly notes that this permits both taxonomy-based evaluation, such as ScanNet200 class names, and arbitrary natural-language queries, such as descriptive or compositional references (Choy et al., 22 Apr 2026).

The reported training setup uses Muon for matrix weights, AdamW for non-matrix parameters, $\mathbf{F}\in\mathbb{R}^{M\times D}$1 nodes with $\mathbf{F}\in\mathbb{R}^{M\times D}$2 H100 80GB per node, $\mathbf{F}\in\mathbb{R}^{M\times D}$3 iterations over $\mathbf{F}\in\mathbb{R}^{M\times D}$4 GPUs, batch size $\mathbf{F}\in\mathbb{R}^{M\times D}$5, and total training time of about $\mathbf{F}\in\mathbb{R}^{M\times D}$6 days (Choy et al., 22 Apr 2026).

6. Empirical performance, efficiency, and ablations

The paper reports end-to-end runtime of $\mathbf{F}\in\mathbb{R}^{M\times D}$7 seconds per scene on an H100 GPU, including sparse voxelization, backbone forward, decoder forward, and mask extraction (Choy et al., 22 Apr 2026). This efficiency is attributed to sparse voxel processing, windowed and curve attention with approximately linear scaling in scene size, proposal-free decoding, and random key sampling in the decoder.

The core benchmark results are summarized below.

Benchmark	SpaCeFormer result	Runtime
Replica	24.1 mAP	0.14 s
ScanNet++	22.9 mAP	0.14 s
ScanNet200	11.1 mAP	0.14 s

On Replica, SpaCeFormer reaches $\mathbf{F}\in\mathbb{R}^{M\times D}$8 mAP, $\mathbf{F}\in\mathbb{R}^{M\times D}$9 mAP@50, and $$\mathbf{Z} \in \mathbb{R}^{N \times D_{\text{CLIP}}}$$0 mAP@25 at $$\mathbf{Z} \in \mathbb{R}^{N \times D_{\text{CLIP}}}$$1 seconds per scene. The same table reports SOLE at $$\mathbf{Z} \in \mathbb{R}^{N \times D_{\text{CLIP}}}$$2 mAP but slower by about $$\mathbf{Z} \in \mathbb{R}^{N \times D_{\text{CLIP}}}$$3, Open-YOLO 3D at $$\mathbf{Z} \in \mathbb{R}^{N \times D_{\text{CLIP}}}$$4 mAP and $$\mathbf{Z} \in \mathbb{R}^{N \times D_{\text{CLIP}}}$$5 seconds, OpenTrack3D at $$\mathbf{Z} \in \mathbb{R}^{N \times D_{\text{CLIP}}}$$6 mAP and about $$\mathbf{Z} \in \mathbb{R}^{N \times D_{\text{CLIP}}}$$7 seconds, Open3DIS at $$\mathbf{Z} \in \mathbb{R}^{N \times D_{\text{CLIP}}}$$8 mAP and $$\mathbf{Z} \in \mathbb{R}^{N \times D_{\text{CLIP}}}$$9 seconds, and OpenMask3D at $Q$0 mAP and $Q$1 seconds (Choy et al., 22 Apr 2026). The paper therefore characterizes SpaCeFormer as Pareto-optimal among methods without ground-truth 3D supervision and notes that it reaches interactive latency below one second.

On ScanNet++, SpaCeFormer reports $Q$2 mAP, $Q$3 mAP@50, and $Q$4 mAP@25 at $Q$5 seconds, compared with $Q$6 mAP for OpenTrack3D at about $Q$7 seconds and $Q$8 mAP or lower for Open3DIS, Any3DIS, and OpenMask3D (Choy et al., 22 Apr 2026). The paper states that SpaCeFormer surpasses all multi-view pipelines on this benchmark while being more than $Q$9 faster.

On ScanNet200, the focus is the 3D-only, proposal-free, no-ground-truth regime. SpaCeFormer reports $\mathbf{Q}^{(0)} \in \mathbb{R}^{Q\times D}$0 mAP, $\mathbf{Q}^{(0)} \in \mathbb{R}^{Q\times D}$1 mAP@50, and $\mathbf{Q}^{(0)} \in \mathbb{R}^{Q\times D}$2 mAP@25, versus $\mathbf{Q}^{(0)} \in \mathbb{R}^{Q\times D}$3 mAP for Mosaic3D with decoder. The paper summarizes this as a $\mathbf{Q}^{(0)} \in \mathbb{R}^{Q\times D}$4 improvement over the prior best proposal-free method in that regime (Choy et al., 22 Apr 2026). For class-agnostic mask quality on the same benchmark, SpaCeFormer reports $\mathbf{Q}^{(0)} \in \mathbb{R}^{Q\times D}$5 AP, $\mathbf{Q}^{(0)} \in \mathbb{R}^{Q\times D}$6 AP@50, and $\mathbf{Q}^{(0)} \in \mathbb{R}^{Q\times D}$7 AP@25 at $\mathbf{Q}^{(0)} \in \mathbb{R}^{Q\times D}$8 seconds, exceeding the corresponding AP@50 and AP@25 numbers of OVIR-3D and MaskClustering despite using only 3D input.

The ablation results isolate the role of the architecture. For space-curve attention, window only yields mAP $\mathbf{Q}^{(0)} \in \mathbb{R}^{Q\times D}$9, Morton only yields mAP $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$00, and window plus Morton yields mAP $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$01. For positional encoding, no PE gives $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$02 mAP, absolute PE $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$03, learnable bias $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$04, and RoPE $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$05. For query initialization, learned queries give mAP $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$06 and AP $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$07, FPS gives mAP $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$08 and AP $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$09, and random gives mAP $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$10 and AP $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$11; the final model uses learned queries (Choy et al., 22 Apr 2026). The backbone-only comparison against PTv3 also reports gains on ScanNet200, ScanNet20, ScanNet++, and Matterport3D, which the paper interprets as evidence that spatial window attention, Morton segments, and RoPE improve the underlying representation.

7. Scope, limitations, applications, and nomenclature

The reported qualitative behavior includes tight masks around object boundaries, separation of adjacent objects in cluttered indoor scenes, and reasonable recovery of small objects such as cups and bowls (Choy et al., 22 Apr 2026). The paper also presents open-vocabulary examples on Matterport3D with categories outside the ScanNet200 taxonomy, including “Snoopy toy,” “Christmas tree,” “ceiling fan,” “cistern,” “flower,” “globe,” “sewing machine,” and “stroller.” This suggests that the CLIP-aligned instance features capture semantic structure that transfers beyond the training label space.

The authors identify several limitations. SpaCeFormer still trails methods that use multi-view RGB-D at inference or ground-truth-trained proposals, such as systems based on Mask3D or OneFormer3D. Evaluation is restricted to indoor scenes, so generalization to outdoor or unstructured environments is not established. The fixed query count $\mathbf{z}_q \in \mathbb{R}^{D_{\text{CLIP}}}$12 may limit recall in scenes with many small objects, motivating adaptive query allocation. The paper also identifies optional future directions, including outdoor deployment, dynamic query allocation, optional hybrid 2D guidance for difficult cases, and improved pseudo-label generation (Choy et al., 22 Apr 2026).

The practical deployment profile emphasized in the paper is straightforward. At inference, the system requires a reconstructed 3D point cloud or mesh with color features, followed by sparse voxelization and coordinate normalization. It does not require streaming RGB images, SAM, YOLO-World, Grounded-SAM, or any external region proposal network (Choy et al., 22 Apr 2026). This makes the method directly relevant to robotics pipelines that maintain 3D maps and to AR/VR systems that project 3D instance masks back into the current view.

A final point of terminology is necessary because the literature in the data block contains two similarly named systems. “SpaCeFormer” denotes the open-vocabulary 3D instance segmentation framework discussed here (Choy et al., 22 Apr 2026). It should be distinguished from “SpaceFormer,” a molecular pretrained representation architecture that models 3D space around atoms for molecular property prediction (Lu et al., 13 Mar 2025). The similarity in naming reflects a shared emphasis on spatial modeling, but the problem settings, architectures, and application domains are entirely different.