OV-3DIS: Open-Vocabulary 3D Instance Segmentation

• OV-3DIS is a segmentation technique that segments and classifies 3D object instances using open-vocabulary queries, enabling retrieval of unseen categories.
• It integrates multimodal pipelines by fusing 2D detection with 3D proposals and hierarchical representations to address challenges like occlusion and scale variation.
• Research in OV-3DIS highlights trade-offs between proposal quality, semantic expressivity, and inference speed, driving advancements in both accuracy and efficiency.

Open-vocabulary 3D instance segmentation (OV-3DIS) is the task of segmenting and classifying object instances in 3D scenes beyond a fixed annotated label space, typically from point clouds, RGB-D sequences, or learned 3D scene representations, while supporting arbitrary text queries or free-form language descriptions (Nguyen et al., 2023). In the literature summarized here, OV-3DIS has evolved from object-centric retrieval over class-agnostic 3D masks toward broader formulations that include fine-grained parts, material or attribute regions, open-world unknown-object handling, and even vocabulary-free or open-ended naming at test time (Takmaz et al., 2024).

1. Task definition and conceptual boundaries

The canonical OV-3DIS setting assumes that a system produces 3D instance masks and associates them with text prompts or query embeddings so that previously unseen categories can be retrieved at inference time (Boudjoghra et al., 2024). This distinguishes it from closed-vocabulary 3D instance segmentation, which is trained and evaluated on a fixed semantic taxonomy, and from purely semantic 3D labeling, which does not require instance separation.

Several adjacent formulations clarify the boundary of the task. Open-world 3D indoor instance segmentation permits the model to distinguish known classes while identifying an unknown object as unknown, and then later incrementally learning the semantic category of the unknown when the corresponding category labels are available (Boudjoghra et al., 2023). Open-ended 3D instance segmentation removes the necessity for predefined class names during testing and introduces a setting in which the system must generate both masks and class names automatically (Nguyen et al., 2024). Vocabulary-free 3D instance segmentation goes further by operating in a setting void of any vocabulary prior, using a large vision-language assistant and an open-vocabulary 2D instance segmenter to discover and ground semantic categories on posed images (Mei et al., 2024).

A second boundary concerns granularity. Classic OV-3DIS methods such as OpenMask3D and Open3DIS are described as object-centric: they extract object instances in 3D, aggregate features per object, and allow querying for objects using arbitrary text (Takmaz et al., 2024). Search3D explicitly argues that this object-level focus is insufficient for object parts, composite queries such as “wooden,” or regions spanning multiple objects or material segments, and therefore constructs hierarchical open-vocabulary 3D scene representations that support fine-grained object parts, entire objects, and attribute-defined regions (Takmaz et al., 2024). This suggests that OV-3DIS is no longer restricted to object retrieval alone, but is increasingly treated as a language-grounded 3D search problem over multiple levels of scene structure.

2. Core computational pipeline

Despite substantial methodological variation, most OV-3DIS systems decompose into proposal formation, multimodal feature construction, and language-driven assignment or retrieval.

A prominent family first builds class-agnostic 3D proposals and then labels them with 2D vision-language evidence. Open3DIS exemplifies this pattern: it generates 2D instance masks on RGB-D frames, over-segments the 3D scene into superpoints, lifts 2D masks into geometrically coherent 3D regions, hierarchically merges regions across views, combines these proposals with native 3D class-agnostic proposals, and then performs open-vocabulary classification with pointwise CLIP feature fusion (Nguyen et al., 2023). Open-YOLO 3D preserves the two-stage structure but replaces SAM- and CLIP-heavy 2D segmentation with open-vocabulary 2D object detection, arguing that the projection of class-agnostic 3D point cloud instances already holds instance information and that Multi-View Prompt Distribution can assign prompts efficiently from bounding-box-derived label maps (Boudjoghra et al., 2024).

A second pattern keeps the 3D proposal stage but alters the bridge to 2D. OpenIns3D uses only 3D point clouds as input, then renders synthetic scene-level images at multiple scales, applies 2D open-vocabulary detectors to those renderings, and performs category assignment through a “Mask-Snap-Lookup” scheme (Huang et al., 2023). The same general principle appears in methods that prioritize tracking or online aggregation rather than offline clustering. “Details Matter for Indoor Open-vocabulary 3D Instance Segmentation” employs robust 3D tracking-based proposal aggregation and iterative merging/removal of overlapped or partial proposals, then classifies proposals with Alpha-CLIP and a standardized maximum similarity score (Jung et al., 30 Jul 2025). OpenTrack3D constructs cross-view consistent object proposals online from an RGB-D stream with a visual-spatial tracker, then replaces CLIP with a multi-modal LLM to improve compositional reasoning for complex user queries (Zhou et al., 3 Dec 2025).

A third pattern is to reduce or eliminate the 2D-to-3D semantic bottleneck at inference. FOLK designs a teacher model that extracts high-quality instance embeddings from 2D CLIP features and distills that knowledge into a 3D student model, so that the distilled model can directly classify instances from the 3D point cloud during inference (Wu et al., 9 Oct 2025). SpaCeFormer is proposal-free: it predicts instance masks directly from learned queries with a RoPE-enhanced decoder and uses a proposal-free space-curve transformer for open-vocabulary 3D instance segmentation using only 3D point cloud input (Choy et al., 22 Apr 2026). SOLE also pushes semantics into the 3D model itself by generating semantic-related masks directly from 3D point clouds through multimodal fusion in both backbone and decoder, supervised by mask-visual, mask-caption, and mask-entity associations (Lee et al., 2024).

At the feature level, most methods rely on some form of multi-view aggregation. Search3D constructs object features from multi-view image crops encoded with SigLIP and segment features from 2D segments backprojected to 3D, with semantic merging of adjacent, feature-similar segments (Takmaz et al., 2024). OpenInsGaussian identifies context loss and multi-view inconsistency as major weaknesses of Gaussian-based pipelines, and responds with context-aware feature extraction plus attention-driven feature aggregation (Huang et al., 21 Oct 2025). These designs indicate that the central technical issue in OV-3DIS is not merely obtaining masks, but maintaining semantic consistency across occlusion, scale variation, and cross-view incompleteness.

3. Major architectural families

The recent literature can be organized into several recurring paradigms.

Paradigm	Representative methods	Hallmark
Proposal fusion from 2D and 3D	Open3DIS, Open-YOLO 3D, OpenIns3D, Details Matter	Combine class-agnostic 3D proposals with 2D masks, boxes, renderings, or tracking cues
Training-free or weakly supervised aggregation	MaskClustering, PoVo	Cluster masks or superpoints with multi-view consensus, spectral clustering, or vision-language assistants
3D-native embedding or proposal-free decoding	SOLE, FOLK, SpaCeFormer	Push open-vocabulary supervision into a 3D backbone, distill from 2D teachers, or decode masks directly from learned queries
Gaussian/NeRF-based scene representations	OpenSplat3D, OpenInsGaussian, DiSCO-3D	Attach semantic features to Gaussians or neural fields and query them with language
Hierarchical and fine-grained search	Search3D	Represent scene, objects, and parts in a tree structure for multi-granularity retrieval

Proposal-fusion systems remain attractive because 2D foundation models often supply stronger open-vocabulary semantics than 3D-native backbones. Open3DIS explicitly motivates 2D-guided 3D proposals as a remedy for small-scale and geometrically ambiguous objects that native 3D proposal networks struggle to identify (Nguyen et al., 2023). Open-YOLO 3D argues that using SAM may be redundant if 3D masks already provide instance information, and that faster 2D object detectors can improve matching while reducing inference time (Boudjoghra et al., 2024). Box-guided retrieval extends this logic further by generating 3D masks for novel objects from RGB images guided by a 2D open-vocabulary detector, avoiding SAM and CLIP altogether while improving long-tail recognition (Nguyen et al., 22 Dec 2025).

Training-free and weakly supervised methods instead emphasize robust aggregation. MaskClustering introduces a global mask graph whose edge weights are determined by a view consensus rate rather than only local neighboring-frame metrics, and iteratively clusters masks showing high view consensus (Yan et al., 2024). PoVo partitions the point cloud into dense superpoints and merges them by spectral clustering using both mask coherence and semantic coherence estimated from 2D object instance masks, while a vision-language assistant discovers scene vocabulary automatically (Mei et al., 2024). A plausible implication is that aggregation quality, rather than the specific choice of 2D detector alone, is often the limiting factor for zero-shot 3D instance quality.

3D-native approaches target the opposite failure mode: expensive and noisy 2D lifting. FOLK states that previous methods usually introduce noise from 2D occlusions and incur substantial computational and memory costs during inference, and therefore distills open-vocabulary knowledge into a 3D student model (Wu et al., 9 Oct 2025). SpaCeFormer similarly critiques both multi-stage 2D+3D pipelines and pseudo-labeled end-to-end approaches, proposing a proposal-free transformer trained on a large multi-view-consistent captioned dataset (Choy et al., 22 Apr 2026). SOLE argues that directly generating generalizable but semantic-related masks from 3D point clouds should outperform purely class-agnostic or purely projected-mask strategies (Lee et al., 2024).

Gaussian and neural-field methods replace point-cloud proposals with explicit scene representations. OpenSplat3D extends 3D Gaussian Splatting with per-Gaussian instance feature embeddings learned from SAM masks and contrastive plus variance-regularized objectives, then clusters Gaussians with HDBSCAN for open-vocabulary querying (Piekenbrinck et al., 9 Jun 2025). OpenInsGaussian adds context-aware mask feature extraction and attention-driven multi-view aggregation to address contextual information loss and cross-view inconsistencies in semantic Gaussian Splatting (Huang et al., 21 Oct 2025). DiSCO-3D operates on NeRF feature fields and frames the broader problem of 3D Open-Vocabulary Sub-concepts Discovery, unifying open-vocabulary segmentation and unsupervised semantic segmentation under query-adaptive sub-concept discovery (Petit et al., 19 Jul 2025).

4. Benchmarks, annotations, and evaluation protocols

The benchmark ecosystem for OV-3DIS is diverse but heavily concentrated on indoor scene understanding. Frequently used datasets include ScanNet200, Replica, ScanNet++, S3DIS, MatterPort3D, MultiScan, 3RScan, LERF-mask, LERF-OVS, and SceneFun3D (Nguyen et al., 2023). Some works also report outdoor results in broader open-vocabulary 3D scene understanding settings, such as nuScenes in UniM-OV3D (He et al., 2024).

Several papers contribute new annotations rather than only reporting on existing benchmarks. Search3D adapts MultiScan into a scene-scale open-vocabulary 3D part segmentation benchmark with 155 object and 15 part categories, and evaluates either part-category-only labels or 47 joint object-part labels such as $(\text{“cabinet”}, \text{“door”})$ (Takmaz et al., 2024). The same work adds fine-grained object and part-level annotations on 8 ScanNet++ scenes with 14 object and 20 part classes (Takmaz et al., 2024). SpaCeFormer introduces SpaCeFormer-3M, described as the largest open-vocabulary 3D instance segmentation dataset, with 3.0M multi-view-consistent captions over 604K instances from 7.4K scenes, built through multi-view mask clustering and multi-view VLM captioning (Choy et al., 22 Apr 2026).

The reported metrics vary with task formulation. Standard OV-3DIS papers commonly report AP, AP$_{50}$, AP$_{25}$, mAP, class-agnostic AP, mIoU, and mAcc (Boudjoghra et al., 2024). Open-world work adds WI, A-OSE, U-Recall, and mAP to evaluate known/unknown discrimination and incremental learning (Boudjoghra et al., 2023). Open-ended evaluation introduces the Open-Ended score, which jointly measures geometric and semantic quality through one-to-one Hungarian matching of predicted masks and generated class names (Nguyen et al., 2024). Its cost matrix is defined as

$$\mathbf{C}(k, j) = -\sqrt[3]{o_{k,j} \cdot \rho_k \cdot \max(0, s_{k,j})},$$

and the OE score is

$$\text{OE score} = 100 \times \frac{\sum_{(k, j)\text{ matches}} -\mathbf{C}(k, j)}{J},$$

where $o_{k,j}$ is mask IoU, $\rho_k$ is confidence, $s_{k,j}$ is the cosine similarity between predicted and ground-truth class name embeddings, and $J$ is the number of ground-truth masks (Nguyen et al., 2024).

A recurrent methodological point is that reported numbers are not always directly comparable across papers, because some evaluate with oracle masks or ground-truth proposals, some use only 3D point clouds, some require aligned RGB-D streams, and some optionally assume meshes or superpoints (Boudjoghra et al., 2024). This suggests that OV-3DIS results must be interpreted in the context of input modality, proposal supervision, and whether classification is evaluated over fixed queries, discovered vocabularies, or generated names.

5. Empirical trends in accuracy, granularity, and efficiency

The field shows a clear trade-off between proposal quality, semantic expressivity, and inference speed, but recent work narrows this trade-off from multiple directions.

In the two-stage proposal-fusion regime, Open3DIS reported ScanNet200 AP of 23.7 for its combined 2D+3D setting, compared with 18.2 for its 2D-only variant and 18.6 for its 3D-only variant, while emphasizing particularly strong gains on tail categories (Nguyen et al., 2023). Open-YOLO 3D then showed that detector-based prompt assignment can improve both runtime and accuracy, reporting 24.7% mAP on the ScanNet200 validation set at 22 seconds per scene and up to $\sim 16\times$ speedup compared to the best existing method in literature (Boudjoghra et al., 2024). The box-guided retrieval method of late 2025 retained this efficiency-oriented direction while improving rare-category retrieval, reporting 24.9 mAP, 32.1 mAP$_{50}$0, and 36.8 mAP$_{50}$1 on ScanNet200, with 22.4 tail mAP and 55.9 seconds per scene (Nguyen et al., 22 Dec 2025).

Tracking and proposal refinement have also produced strong gains. “Details Matter for Indoor Open-vocabulary 3D Instance Segmentation” reported 25.8 mAP on ScanNet200 in its Top-1, 2D+3D setting and 32.7 in Top-K, while also improving class-agnostic proposal quality to 46.6% AP (Jung et al., 30 Jul 2025). OpenTrack3D, which is mesh-free at its core and uses an MLLM for proposal classification, reported 26.0 AP, 37.7 AP$_{50}$2, and 45.4 AP$_{50}$3 on ScanNet200, together with 23.9 AP on Replica and 20.6 AP on ScanNet++ (Zhou et al., 3 Dec 2025).

A separate line of work aims to bypass slow multi-view semantic aggregation. FOLK reports state-of-the-art performance on ScanNet200 with an AP50 score of 35.7 while running approximately $_{50}$4 to $_{50}$5 faster than previous methods, using label-guided knowledge distillation so that inference proceeds directly from the 3D point cloud (Wu et al., 9 Oct 2025). SpaCeFormer pushes this logic to a proposal-free extreme: it runs at 0.14 seconds per scene, reports 11.1 zero-shot mAP on ScanNet200, and reaches 22.9 and 24.1 mAP on ScanNet++ and Replica, respectively (Choy et al., 22 Apr 2026). The reported numbers indicate that proposal-free speed can be achieved, but that the highest absolute accuracy on ScanNet200 in these summaries still often comes from stronger proposal or tracking pipelines.

Fine-grained segmentation and alternative scene representations show another empirical frontier. Search3D reported scene-scale open-vocabulary 3D part segmentation AP of 7.9, AP$_{50}$6 of 14.5, and AP$_{50}$7 of 31.5 on its MultiScan benchmark, as well as AP 17.0 and AP$_{50}$8 32.4 on ScanNet++ fine-grained annotations, while maintaining object-level AP 18.1 on MultiScan and AP 23.0 on ScanNet200 with SigLIP features (Takmaz et al., 2024). OpenSplat3D reported ScanNet++ [email protected] of 29.7 for open-vocabulary instance segmentation and 41.7 for post-processed class-agnostic segmentation, as well as strong results on LERF-mask and LERF-OVS (Piekenbrinck et al., 9 Jun 2025). These results suggest that the gap between point-cloud pipelines and Gaussian-based scene representations is narrowing, particularly when dense differentiable rendering improves 2D-to-3D supervision.

6. Limitations, misconceptions, and research directions

A common misconception is that open-vocabulary implies open-ended or vocabulary-free operation. The literature explicitly distinguishes these settings: OE-3DIS removes the requirement for predefined class names during testing, and vocabulary-free segmentation operates without any vocabulary prior, whereas many OV-3DIS systems still assume user-provided queries, prompted text lists, or class-name sets at inference (Nguyen et al., 2024). Similarly, open-world segmentation is not identical to open-vocabulary segmentation, because it focuses on identifying unknown objects as unknown and incrementally learning them later rather than directly grounding arbitrary free-form language (Boudjoghra et al., 2023).

Another misconception is that open-vocabulary capability automatically implies fine-grained or compositional understanding. Search3D shows that classic OV-3DIS methods are limited to object-level entities and are not designed for parts or material regions, motivating hierarchical object-part-scene representations (Takmaz et al., 2024). OpenTrack3D attributes weak handling of compositional and functional user queries to CLIP-based classifiers and replaces them with an MLLM for stronger textual reasoning (Zhou et al., 3 Dec 2025). DiSCO-3D makes a related point in neural fields: open-vocabulary segmentation and unsupervised segmentation capture different edge cases, and a broader formulation may require discovery of query-relevant sub-concepts rather than only direct object-name retrieval (Petit et al., 19 Jul 2025).

The technical limitations cited across papers are consistent. Search3D identifies suboptimal geometric over-segmentation for parts with similar geometry, noisy multi-view fusion of fine 2D masks, a restriction to only objects and parts in the hierarchy, and non-real-time feature extraction (Takmaz et al., 2024). OpenInsGaussian identifies contextual information loss during local mask cropping and inconsistencies in naïve multi-view averaging (Huang et al., 21 Oct 2025). FOLK targets occlusion noise and the computational and memory cost of 2D rendering and matching (Wu et al., 9 Oct 2025). OpenTrack3D points to dataset-specific proposal networks and mesh-based superpoints as barriers to generalization in mesh-free environments (Zhou et al., 3 Dec 2025). SpaCeFormer criticizes both slow multi-stage pipelines and pseudo-labeled end-to-end approaches that rely on fragmented masks and external region proposals (Choy et al., 22 Apr 2026).

The main research directions follow directly from these limitations. One direction is richer structure: additional hierarchical levels, sub-parts, functional regions, and material-aware representations (Takmaz et al., 2024). A second is more reliable multimodal fusion, including context-aware extraction, attention-driven cross-view aggregation, or multi-view-consistent supervision (Huang et al., 21 Oct 2025). A third is stronger 3D-native inference through distillation or proposal-free decoding, reducing dependence on heavy 2D pipelines at test time (Wu et al., 9 Oct 2025). A fourth is broader autonomy: open-ended naming, vocabulary-free discovery, and query-adaptive sub-concept discovery push OV-3DIS from text-conditioned retrieval toward more general 3D scene understanding (Nguyen et al., 2024).

Taken together, these developments indicate that OV-3DIS is no longer a single narrowly defined benchmark problem. It has become a family of language-grounded 3D segmentation tasks spanning object retrieval, part segmentation, material and attribute grounding, unknown-object handling, autonomous naming, and scene-adaptive concept discovery.