SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation

Abstract: Open-vocabulary 3D instance segmentation is a core capability for robotics and AR/VR, but prior methods trade one bottleneck for another: multi-stage 2D+3D pipelines aggregate foundation-model outputs at hundreds of seconds per scene, while pseudo-labeled end-to-end approaches rely on fragmented masks and external region proposals. We present SpaCeFormer, a proposal-free space-curve transformer that runs at 0.14 seconds per scene, 2-3 orders of magnitude faster than multi-stage 2D+3D pipelines. We pair it with SpaCeFormer-3M, the largest open-vocabulary 3D instance segmentation dataset (3.0M multi-view-consistent captions over 604K instances from 7.4K scenes) built through multi-view mask clustering and multi-view VLM captioning; it reaches 21x higher mask recall than prior single-view pipelines (54.3% vs 2.5% at IoU > 0.5). SpaCeFormer combines spatial window attention with Morton-curve serialization for spatially coherent features, and uses a RoPE-enhanced decoder to predict instance masks directly from learned queries without external proposals. On ScanNet200 we achieve 11.1 zero-shot mAP, a 2.8x improvement over the prior best proposal-free method; on ScanNet++ and Replica, we reach 22.9 and 24.1 mAP, surpassing all prior methods including those using multi-view 2D inputs.

• The paper presents a novel, proposal-free framework that combines multi-view mask clustering and view-consistent captioning to generate high-quality 3D supervision.
• The SpaCeFormer-3M pipeline achieves a 21× improvement in mask recall by aggregating partial 2D masks into coherent 3D instances over large-scale indoor datasets.
• The architecture leverages space-curve attention and RoPE-enhanced decoding to boost spatial fidelity and inference speed, running up to 3900× faster than multi-stage baselines.

SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation

Problem Setting and Motivation

Open-vocabulary 3D instance segmentation is pivotal for real-world robotics, AR/VR, and embodied reasoning tasks, requiring fine-grained object delineation across unconstrained categories described in natural language. Existing methods cluster into two camps: multi-stage pipelines fusing 2D foundation model outputs in 3D—suffering from high computational cost and degraded generalization—or pseudo-label–driven approaches where 2D-based pseudo-masks and captions are lifted to 3D, often yielding fragmented and inconsistent supervision. Critically, state-of-the-art 3D transformer backbones—e.g., Point Transformer v3 (PTv3), OctFormer—enforce serialization heuristics that degrade local spatial coherence, impairing boundary precision and hampering the generalization capabilities essential for open-vocabulary reasoning.

SpaCeFormer-3M: Data Generation Pipeline

This work introduces SpaCeFormer-3M, a large-scale open-vocabulary 3D instance segmentation dataset. The pipeline synthesizes high-quality 3D mask-caption pairs through two main innovations:

1. Multi-view Mask Clustering: Partial 2D masks—obtained via segmentation foundation models—are aggregated into complete 3D instances employing training-free spatial clustering across multiple views, resolving previously intractable mask fragmentation issues.
2. View-consistent Captioning: For each instance, diverse captions emphasizing intrinsic object properties are generated using structured multi-view VLM prompting, increasing semantic consistency relative to single-view pipelines.

   Figure 1: Illustration of the dataset generation pipeline, with multi-view mask clustering and multi-view VLM prompting yielding geometry- and caption-consistent 3D supervision.

Quantitative analysis demonstrates a 21× improvement in mask recall (54.3% vs 2.5% at IoU $>$ 0.5) over prior single-view pseudo-label pipelines such as Mosaic3D, and an order-of-magnitude increase in scale—encompassing 604K multi-view–consistent masks and 3.0M captions over 7.4K indoor scenes from ScanNet, ScanNet++, ARKitScenes, and Matterport3D.

Architecture: Space-Curve Transformer with RoPE Decoding

SpaCeFormer’s model architecture is explicitly tailored for open-vocabulary, proposal-free 3D instance segmentation. It is based on two architectural advances:

Space-Curve Attention Backbone

The backbone hybridizes spatial window attention—which preserves neighborhood coherence critical for precise mask prediction—with Morton curve serialization—which injects structured diversity, enabling efficient scalability. This is in contrast to pure Morton or windowed approaches, and is shown to yield a 28.6% reduction in intra-window spatial distance.

This space-curve hybridization significantly boosts the spatial fidelity of point embeddings, which is measured and ablated in the manuscript.

RoPE-Enhanced Proposal-Free Instance Decoder

SpaCeFormer eliminates all region proposal networks, replacing them with a RoPE-enhanced query-based decoder. Learned queries interact with scene point features through cross-attention equipped with 3D rotary positional embeddings (RoPE), directly encoding geometric relationships required for instance-level discrimination and open-vocabulary alignment (via CLIP). The decoder architecture iteratively refines queries, and training is performed end-to-end using composite mask, foregroundness, and CLIP-alignment losses tuned with advanced optimizers (Muon for 2D parameters, AdamW for vectors).

Experimental Results

Extensive experiments validate SpaCeFormer as Pareto-optimal along the accuracy-latency tradeoff boundary.

Figure 2: Accuracy versus latency on Replica zero-shot open-vocabulary 3D instance segmentation. SpaCeFormer is Pareto-optimal in the regime with no ground-truth 3D supervision and interactive latency (<1s).

• Replica (zero-shot, 8 scenes): SpaCeFormer achieves 24.1 mAP at 0.14s per scene—comparable to the best proposal-based models (SOLE: 24.7 mAP), running up to 3900× faster than multi-stage baselines.
• ScanNet++ (100 classes): Delivers 22.9 mAP, surpassing all prior methods including those leveraging 2D multi-view input (OpenTrack3D: 20.6 mAP, requires YOLO-World + SAM2, 320s/scene).
• ScanNet200 (200 classes): Under the proposal-free, 3D-only, GT-agnostic regime, SpaCeFormer attains 11.1 mAP, a 2.8× gain over the prior best proposal-free method (Mosaic3D+Decoder: 3.9 mAP), while executing at 0.14s per scene.

Qualitative analysis highlights the model’s ability to segment diverse objects—furniture, equipment, and novel categories—beyond closed-taxonomy limitations.

Implications, Limitations, and Future Directions

From a practical standpoint, SpaCeFormer pushes open-vocabulary 3D reasoning toward real-time, interactive operation on commodity hardware without reliance on geometric ground-truth supervision or costly multi-view pipelines. The design principles—especially spatially coherent window attention and geometry-aware RoPE decoding—set a foundation for unified scene and instance segmentation in robotics, AR/VR, and embodied perception.

On the theoretical side, the space-curve hybrid attention indicates the necessity of preserving local geometric context for dense prediction in 3D transformer backbones. The successful application of 3D RoPE in both backbone and decoder blocks establishes rotary encodings as an effective, scalable alternative for geometry-aware positional encoding in sparse 3D domains.

Nevertheless, several limitations persist:

• Taxonomic Generalization: Although SpaCeFormer advances open-category recognition, overall mAP lags closed-vocabulary, GT-trained proposal-based models, especially in cluttered scenes with many small or occluded objects.
• Scope: Evaluation is limited to indoor, RGB-D–derived environments; adaptation to outdoor or multi-modal lidar scenes remains unaddressed.
• Query Scalability: A fixed number of learned queries ($Q=200$) introduces an inherent cap on instance granularity in highly cluttered scenes.

Future progress will likely focus on adaptive query mechanisms, unified multi-modal (2D-3D-text) representations, domain-invariant geometric pretraining, and scaling consistency-aware supervision pipelines.

Conclusion

SpaCeFormer provides a formal advancement in open-vocabulary 3D instance segmentation through innovations in both data curation (SpaCeFormer-3M) and model architecture (space-curve attention, RoPE-decoder, proposal-free querying). Achieving state-of-the-art accuracy and orders-of-magnitude speedup over prior multi-stage approaches, SpaCeFormer establishes a new baseline for fast, robust, proposal-free open-vocabulary 3D scene understanding, catalyzing future research in real-time embodied AI and language-grounded 3D perception.