---
title: 'OV-3DIS: Open-Vocabulary 3D Instance Segmentation'
url: https://www.emergentmind.com/topics/open-vocabulary-3d-instance-segmentation-ov-3dis
type: topic
---

# OV-3DIS: Open-Vocabulary 3D Instance Segmentation

Open-vocabulary 3D instance segmentation (OV-3DIS) is the task of segmenting and classifying object instances in 3D scenes beyond a fixed annotated label space, typically from point clouds, RGB-D sequences, or learned 3D scene representations, while supporting arbitrary text queries or free-form language descriptions [2312.10671]. In the literature summarized here, OV-3DIS has evolved from object-centric retrieval over class-agnostic 3D masks toward broader formulations that include fine-grained parts, material or attribute regions, open-world unknown-object handling, and even vocabulary-free or open-ended naming at test time [2409.18431].

## 1. Task definition and conceptual boundaries

The canonical OV-3DIS setting assumes that a system produces 3D instance masks and associates them with text prompts or query embeddings so that previously unseen categories can be retrieved at inference time [2406.02548]. This distinguishes it from closed-vocabulary 3D instance segmentation, which is trained and evaluated on a fixed semantic taxonomy, and from purely semantic 3D labeling, which does not require instance separation.

Several adjacent formulations clarify the boundary of the task. Open-world 3D indoor instance segmentation permits the model to distinguish known classes while identifying an unknown object as unknown, and then later incrementally learning the semantic category of the unknown when the corresponding category labels are available [2309.14338]. Open-ended 3D instance segmentation removes the necessity for predefined class names during testing and introduces a setting in which the system must generate both masks and class names automatically [2408.11747]. Vocabulary-free 3D instance segmentation goes further by operating in a setting void of any vocabulary prior, using a large vision-language assistant and an open-vocabulary 2D instance segmenter to discover and ground semantic categories on posed images [2408.10652].

A second boundary concerns granularity. Classic OV-3DIS methods such as OpenMask3D and Open3DIS are described as object-centric: they extract object instances in 3D, aggregate features per object, and allow querying for objects using arbitrary text [2409.18431]. Search3D explicitly argues that this object-level focus is insufficient for object parts, composite queries such as “wooden,” or regions spanning multiple objects or material segments, and therefore constructs hierarchical open-vocabulary 3D scene representations that support fine-grained object parts, entire objects, and attribute-defined regions [2409.18431]. This suggests that OV-3DIS is no longer restricted to object retrieval alone, but is increasingly treated as a language-grounded 3D search problem over multiple levels of scene structure.

## 2. Core computational pipeline

Despite substantial methodological variation, most OV-3DIS systems decompose into proposal formation, multimodal feature construction, and language-driven assignment or retrieval.

A prominent family first builds class-agnostic 3D proposals and then labels them with 2D vision-language evidence. Open3DIS exemplifies this pattern: it generates 2D instance masks on RGB-D frames, over-segments the 3D scene into superpoints, lifts 2D masks into geometrically coherent 3D regions, hierarchically merges regions across views, combines these proposals with native 3D class-agnostic proposals, and then performs open-vocabulary classification with pointwise CLIP feature fusion [2312.10671]. Open-YOLO 3D preserves the two-stage structure but replaces SAM- and CLIP-heavy 2D segmentation with open-vocabulary 2D object detection, arguing that the projection of class-agnostic 3D point cloud instances already holds instance information and that Multi-View Prompt Distribution can assign prompts efficiently from bounding-box-derived label maps [2406.02548].

A second pattern keeps the 3D proposal stage but alters the bridge to 2D. OpenIns3D uses only 3D point clouds as input, then renders synthetic scene-level images at multiple scales, applies 2D open-vocabulary detectors to those renderings, and performs category assignment through a “Mask-Snap-Lookup” scheme [2309.00616]. The same general principle appears in methods that prioritize tracking or online aggregation rather than offline clustering. “Details Matter for Indoor Open-vocabulary 3D Instance Segmentation” employs robust 3D tracking-based proposal aggregation and iterative merging/removal of overlapped or partial proposals, then classifies proposals with Alpha-CLIP and a standardized maximum similarity score [2507.23134]. OpenTrack3D constructs cross-view consistent object proposals online from an RGB-D stream with a visual-spatial tracker, then replaces CLIP with a multi-modal large language model to improve compositional reasoning for complex user queries [2512.03532].

A third pattern is to reduce or eliminate the 2D-to-3D semantic bottleneck at inference. FOLK designs a teacher model that extracts high-quality instance embeddings from 2D CLIP features and distills that knowledge into a 3D student model, so that the distilled model can directly classify instances from the 3D point cloud during inference [2510.08849]. SpaCeFormer is proposal-free: it predicts instance masks directly from learned queries with a RoPE-enhanced decoder and uses a proposal-free space-curve transformer for open-vocabulary 3D instance segmentation using only 3D point cloud input [2604.20395]. SOLE also pushes semantics into the 3D model itself by generating semantic-related masks directly from 3D point clouds through multimodal fusion in both backbone and decoder, supervised by mask-visual, mask-caption, and mask-entity associations [2404.02157].

At the feature level, most methods rely on some form of multi-view aggregation. Search3D constructs object features from multi-view image crops encoded with SigLIP and segment features from 2D segments backprojected to 3D, with semantic merging of adjacent, feature-similar segments [2409.18431]. OpenInsGaussian identifies context loss and multi-view inconsistency as major weaknesses of Gaussian-based pipelines, and responds with context-aware feature extraction plus attention-driven feature aggregation [2510.18253]. These designs indicate that the central technical issue in OV-3DIS is not merely obtaining masks, but maintaining semantic consistency across occlusion, scale variation, and cross-view incompleteness.

## 3. Major architectural families

The recent literature can be organized into several recurring paradigms.

| Paradigm | Representative methods | Hallmark |
|---|---|---|
| Proposal fusion from 2D and 3D | Open3DIS, Open-YOLO 3D, OpenIns3D, Details Matter | Combine class-agnostic 3D proposals with 2D masks, boxes, renderings, or tracking cues |
| Training-free or weakly supervised aggregation | MaskClustering, PoVo | Cluster masks or superpoints with multi-view consensus, spectral clustering, or vision-language assistants |
| 3D-native embedding or proposal-free decoding | SOLE, FOLK, SpaCeFormer | Push open-vocabulary supervision into a 3D backbone, distill from 2D teachers, or decode masks directly from learned queries |
| Gaussian/NeRF-based scene representations | OpenSplat3D, OpenInsGaussian, DiSCO-3D | Attach semantic features to Gaussians or neural fields and query them with language |
| Hierarchical and fine-grained search | Search3D | Represent scene, objects, and parts in a tree structure for multi-granularity retrieval |

Proposal-fusion systems remain attractive because 2D foundation models often supply stronger open-vocabulary semantics than 3D-native backbones. Open3DIS explicitly motivates 2D-guided 3D proposals as a remedy for small-scale and geometrically ambiguous objects that native 3D proposal networks struggle to identify [2312.10671]. Open-YOLO 3D argues that using SAM may be redundant if 3D masks already provide instance information, and that faster 2D object detectors can improve matching while reducing inference time [2406.02548]. Box-guided retrieval extends this logic further by generating 3D masks for novel objects from RGB images guided by a 2D open-vocabulary detector, avoiding SAM and CLIP altogether while improving long-tail recognition [2512.19088].

Training-free and weakly supervised methods instead emphasize robust aggregation. MaskClustering introduces a global mask graph whose edge weights are determined by a view consensus rate rather than only local neighboring-frame metrics, and iteratively clusters masks showing high view consensus [2401.07745]. PoVo partitions the point cloud into dense superpoints and merges them by spectral clustering using both mask coherence and semantic coherence estimated from 2D object instance masks, while a vision-language assistant discovers scene vocabulary automatically [2408.10652]. A plausible implication is that aggregation quality, rather than the specific choice of 2D detector alone, is often the limiting factor for zero-shot 3D instance quality.

3D-native approaches target the opposite failure mode: expensive and noisy 2D lifting. FOLK states that previous methods usually introduce noise from 2D occlusions and incur substantial computational and memory costs during inference, and therefore distills open-vocabulary knowledge into a 3D student model [2510.08849]. SpaCeFormer similarly critiques both multi-stage 2D+3D pipelines and pseudo-labeled end-to-end approaches, proposing a proposal-free transformer trained on a large multi-view-consistent captioned dataset [2604.20395]. SOLE argues that directly generating generalizable but semantic-related masks from 3D point clouds should outperform purely class-agnostic or purely projected-mask strategies [2404.02157].

Gaussian and neural-field methods replace point-cloud proposals with explicit scene representations. OpenSplat3D extends 3D Gaussian Splatting with per-Gaussian instance feature embeddings learned from SAM masks and contrastive plus variance-regularized objectives, then clusters Gaussians with HDBSCAN for open-vocabulary querying [2506.07697]. OpenInsGaussian adds context-aware mask feature extraction and attention-driven multi-view aggregation to address contextual information loss and cross-view inconsistencies in semantic Gaussian Splatting [2510.18253]. DiSCO-3D operates on NeRF feature fields and frames the broader problem of 3D Open-Vocabulary Sub-concepts Discovery, unifying open-vocabulary segmentation and unsupervised semantic segmentation under query-adaptive sub-concept discovery [2507.14596].

## 4. Benchmarks, annotations, and evaluation protocols

The benchmark ecosystem for OV-3DIS is diverse but heavily concentrated on indoor scene understanding. Frequently used datasets include ScanNet200, Replica, ScanNet++, S3DIS, MatterPort3D, MultiScan, 3RScan, LERF-mask, LERF-OVS, and SceneFun3D [2312.10671]. Some works also report outdoor results in broader open-vocabulary 3D scene understanding settings, such as nuScenes in UniM-OV3D [2401.11395].

Several papers contribute new annotations rather than only reporting on existing benchmarks. Search3D adapts MultiScan into a scene-scale open-vocabulary 3D part segmentation benchmark with 155 object and 15 part categories, and evaluates either part-category-only labels or 47 joint object-part labels such as \((\text{“cabinet”}, \text{“door”})\) [2409.18431]. The same work adds fine-grained object and part-level annotations on 8 ScanNet++ scenes with 14 object and 20 part classes [2409.18431]. SpaCeFormer introduces SpaCeFormer-3M, described as the largest open-vocabulary 3D instance segmentation dataset, with 3.0M multi-view-consistent captions over 604K instances from 7.4K scenes, built through multi-view mask clustering and multi-view VLM captioning [2604.20395].

The reported metrics vary with task formulation. Standard OV-3DIS papers commonly report AP, AP\(_{50}\), AP\(_{25}\), mAP, class-agnostic AP, mIoU, and mAcc [2406.02548]. Open-world work adds WI, A-OSE, U-Recall, and mAP to evaluate known/unknown discrimination and incremental learning [2309.14338]. Open-ended evaluation introduces the Open-Ended score, which jointly measures geometric and semantic quality through one-to-one Hungarian matching of predicted masks and generated class names [2408.11747]. Its cost matrix is defined as
$$
\mathbf{C}(k, j) = -\sqrt[3]{o_{k,j} \cdot \rho_k \cdot \max(0, s_{k,j})},
$$
and the OE score is
$$
\text{OE score} = 100 \times \frac{\sum_{(k, j)\text{ matches}} -\mathbf{C}(k, j)}{J},
$$
where \(o_{k,j}\) is mask IoU, \(\rho_k\) is confidence, \(s_{k,j}\) is the cosine similarity between predicted and ground-truth class name embeddings, and \(J\) is the number of ground-truth masks [2408.11747].

A recurrent methodological point is that reported numbers are not always directly comparable across papers, because some evaluate with oracle masks or ground-truth proposals, some use only 3D point clouds, some require aligned RGB-D streams, and some optionally assume meshes or superpoints [2406.02548]. This suggests that OV-3DIS results must be interpreted in the context of input modality, proposal supervision, and whether classification is evaluated over fixed queries, discovered vocabularies, or generated names.

## 5. Empirical trends in accuracy, granularity, and efficiency

The field shows a clear trade-off between proposal quality, semantic expressivity, and inference speed, but recent work narrows this trade-off from multiple directions.

In the two-stage proposal-fusion regime, Open3DIS reported ScanNet200 AP of 23.7 for its combined 2D+3D setting, compared with 18.2 for its 2D-only variant and 18.6 for its 3D-only variant, while emphasizing particularly strong gains on tail categories [2312.10671]. Open-YOLO 3D then showed that detector-based prompt assignment can improve both runtime and accuracy, reporting 24.7% mAP on the ScanNet200 validation set at 22 seconds per scene and up to \(\sim 16\times\) speedup compared to the best existing method in literature [2406.02548]. The box-guided retrieval method of late 2025 retained this efficiency-oriented direction while improving rare-category retrieval, reporting 24.9 mAP, 32.1 mAP\(_{50}\), and 36.8 mAP\(_{25}\) on ScanNet200, with 22.4 tail mAP and 55.9 seconds per scene [2512.19088].

Tracking and proposal refinement have also produced strong gains. “Details Matter for Indoor Open-vocabulary 3D Instance Segmentation” reported 25.8 mAP on ScanNet200 in its Top-1, 2D+3D setting and 32.7 in Top-K, while also improving class-agnostic proposal quality to 46.6% AP [2507.23134]. OpenTrack3D, which is mesh-free at its core and uses an MLLM for proposal classification, reported 26.0 AP, 37.7 AP\(_{50}\), and 45.4 AP\(_{25}\) on ScanNet200, together with 23.9 AP on Replica and 20.6 AP on ScanNet++ [2512.03532].

A separate line of work aims to bypass slow multi-view semantic aggregation. FOLK reports state-of-the-art performance on ScanNet200 with an AP50 score of 35.7 while running approximately \(6.0\times\) to \(152.2\times\) faster than previous methods, using label-guided knowledge distillation so that inference proceeds directly from the 3D point cloud [2510.08849]. SpaCeFormer pushes this logic to a proposal-free extreme: it runs at 0.14 seconds per scene, reports 11.1 zero-shot mAP on ScanNet200, and reaches 22.9 and 24.1 mAP on ScanNet++ and Replica, respectively [2604.20395]. The reported numbers indicate that proposal-free speed can be achieved, but that the highest absolute accuracy on ScanNet200 in these summaries still often comes from stronger proposal or tracking pipelines.

Fine-grained segmentation and alternative scene representations show another empirical frontier. Search3D reported scene-scale open-vocabulary 3D part segmentation AP of 7.9, AP\(_{50}\) of 14.5, and AP\(_{25}\) of 31.5 on its MultiScan benchmark, as well as AP 17.0 and AP\(_{50}\) 32.4 on ScanNet++ fine-grained annotations, while maintaining object-level AP 18.1 on MultiScan and AP 23.0 on ScanNet200 with SigLIP features [2409.18431]. OpenSplat3D reported ScanNet++ AP@0.5 of 29.7 for open-vocabulary instance segmentation and 41.7 for post-processed class-agnostic segmentation, as well as strong results on LERF-mask and LERF-OVS [2506.07697]. These results suggest that the gap between point-cloud pipelines and Gaussian-based scene representations is narrowing, particularly when dense differentiable rendering improves 2D-to-3D supervision.

## 6. Limitations, misconceptions, and research directions

A common misconception is that open-vocabulary implies open-ended or vocabulary-free operation. The literature explicitly distinguishes these settings: OE-3DIS removes the requirement for predefined class names during testing, and vocabulary-free segmentation operates without any vocabulary prior, whereas many OV-3DIS systems still assume user-provided queries, prompted text lists, or class-name sets at inference [2408.11747]. Similarly, open-world segmentation is not identical to open-vocabulary segmentation, because it focuses on identifying unknown objects as unknown and incrementally learning them later rather than directly grounding arbitrary free-form language [2309.14338].

Another misconception is that open-vocabulary capability automatically implies fine-grained or compositional understanding. Search3D shows that classic OV-3DIS methods are limited to object-level entities and are not designed for parts or material regions, motivating hierarchical object-part-scene representations [2409.18431]. OpenTrack3D attributes weak handling of compositional and functional user queries to CLIP-based classifiers and replaces them with an MLLM for stronger textual reasoning [2512.03532]. DiSCO-3D makes a related point in neural fields: open-vocabulary segmentation and unsupervised segmentation capture different edge cases, and a broader formulation may require discovery of query-relevant sub-concepts rather than only direct object-name retrieval [2507.14596].

The technical limitations cited across papers are consistent. Search3D identifies suboptimal geometric over-segmentation for parts with similar geometry, noisy multi-view fusion of fine 2D masks, a restriction to only objects and parts in the hierarchy, and non-real-time feature extraction [2409.18431]. OpenInsGaussian identifies contextual information loss during local mask cropping and inconsistencies in naïve multi-view averaging [2510.18253]. FOLK targets occlusion noise and the computational and memory cost of 2D rendering and matching [2510.08849]. OpenTrack3D points to dataset-specific proposal networks and mesh-based superpoints as barriers to generalization in mesh-free environments [2512.03532]. SpaCeFormer criticizes both slow multi-stage pipelines and pseudo-labeled end-to-end approaches that rely on fragmented masks and external region proposals [2604.20395].

The main research directions follow directly from these limitations. One direction is richer structure: additional hierarchical levels, sub-parts, functional regions, and material-aware representations [2409.18431]. A second is more reliable multimodal fusion, including context-aware extraction, attention-driven cross-view aggregation, or multi-view-consistent supervision [2510.18253]. A third is stronger 3D-native inference through distillation or proposal-free decoding, reducing dependence on heavy 2D pipelines at test time [2510.08849]. A fourth is broader autonomy: open-ended naming, vocabulary-free discovery, and query-adaptive sub-concept discovery push OV-3DIS from text-conditioned retrieval toward more general 3D scene understanding [2408.11747].

Taken together, these developments indicate that OV-3DIS is no longer a single narrowly defined benchmark problem. It has become a family of language-grounded 3D segmentation tasks spanning object retrieval, part segmentation, material and attribute grounding, unknown-object handling, autonomous naming, and scene-adaptive concept discovery.

Source: https://www.emergentmind.com/topics/open-vocabulary-3d-instance-segmentation-ov-3dis