3D Open-Vocabulary Sub-Concept Discovery

• The paper DiSCO-3D pioneers a generalized task integrating open-vocabulary segmentation with unsupervised clustering for query-aware 3D semantic decomposition.
• It leverages diverse 3D representations—from neural fields to voxel grids—to capture fine-grained sub-concepts beyond simple object-level grouping.
• Empirical results demonstrate enhanced segmentation accuracy and improved long-tail generalization, paving the way for more layered 3D semantic models.

Searching arXiv for papers on 3D open-vocabulary sub-concepts discovery and closely related 3D open-vocabulary scene understanding. 3D Open-Vocabulary Sub-concepts Discovery denotes a 3D semantic segmentation setting in which the output must adapt both to scene content and to open-vocabulary user intent: given a 3D scene, typically represented as a neural field or another language-queryable 3D representation, and optionally one or more user queries, the system should produce a segmentation that is query-relevant, scene-adaptive, and more fine-grained than the query itself (Petit et al., 19 Jul 2025). In the surrounding literature, this goal appears in both direct and qualified forms. Some methods explicitly target broader query-conditioned sub-concept decomposition, while many others discover object-level or region-level semantic units—instance masks, sparse voxel groups, superpoints, occupancy voxels, or compact region memories—that can be interpreted as partial substrates for sub-concept discovery rather than full part-attribute-relation ontologies (Huang et al., 14 Jan 2026).

1. Task scope and conceptual boundaries

The most explicit task definition appears in DiSCO-3D, which introduces 3D Open-Vocabulary Sub-concepts Discovery as a problem that generalizes both open-vocabulary segmentation and unsupervised semantic segmentation. In this formulation, the system should discover scene-present semantic classes that are semantically related to a broader query concept, rather than merely returning a single mask for the queried concept itself (Petit et al., 19 Jul 2025). The motivating examples are intentionally query-broader-than-label: a query such as “tools” should ignore irrelevant content and separate the scene-present tool types that actually occur.

This boundary matters because a common misconception is to equate any open-vocabulary 3D system with sub-concept discovery. Much of the earlier literature is more accurately described as open-vocabulary retrieval or localization over 3D regions. OpenMask3D, for example, introduces open-vocabulary 3D instance segmentation and explicitly supports free-form queries involving geometry, affordances, materials, colors, object state, and situational context, but it does not cluster emergent sub-concepts, induce taxonomies, infer attribute axes, or discover parts automatically (Takmaz et al., 2023). The same distinction appears across related methods: a system may support query-driven access to fine-grained meanings without autonomously discovering them as structured 3D concepts.

A second boundary concerns granularity. Several methods are explicitly instance-level. OpenVoxel states that it “clusters sparse voxels into instance-level 3D masks” and aims for “object-level groups,” while acknowledging that some resulting groups may correspond to background materials or region-like units rather than canonical objects (Huang et al., 14 Jan 2026). This suggests that the term “sub-concept” is used across the literature in at least two senses: a strong sense involving systematic part-level or hierarchical decomposition, and a weaker but practically important sense involving semantically coherent 3D units below the whole-scene level.

2. Representation substrates for discoverable 3D semantics

The literature uses several distinct 3D substrates, and the representation often determines what kinds of sub-concepts can be discovered. Neural-field methods use continuous feature fields. OpenOcc represents geometry, appearance, and semantics with separate multi-resolution feature grids, decoding occupancy, color, and a continuous 3D semantic field from distinct latent structures. Its occupancy rendering follows a first-hit formulation,

$$\hat{S}(\mathbf{r}) = \sum_{i=1}^N o(\mathbf{x}_i) \prod_{j<i}(1-o(\mathbf{x}_j)) s(\mathbf{x}_i),$$

so semantic embeddings are anchored to geometrically plausible surfaces rather than an unconstrained radiance-only field (Jiang et al., 2024). POP-3D adopts a dense voxel field instead: a 2D-to-3D encoder predicts a $100\times100\times8$ voxel grid, and each occupied voxel carries a $512$-dimensional language-aligned embedding, enabling zero-shot segmentation, grounding, and retrieval from arbitrary text prompts (Vobecky et al., 2024).

Other systems are explicitly discrete. OpenVoxel starts from a reconstructed Sparse Voxel Rasterization scene $\{V_i\}_{i=1}^N$, where volumetric alpha compositing defines rendering weights

$$\mathbf{C}(\mathbf{r}) = \sum_{i<N} w_i(\mathbf{r}) \cdot c_i,\qquad w_i(\mathbf{r}) = \alpha_i(\mathbf{r}) \cdot \prod_{j<i}(1-\alpha_j(\mathbf{r})).$$

Those same weights are then reused during grouping, so the scene is not only rendered but also aggregated as explicit sparse voxels with attachable attributes (Huang et al., 14 Jan 2026). Open-Fusion instead uses a TSDF volume with dictionary-linked region semantics: each surface-near voxel stores a key into a compact embedding dictionary plus a confidence score, yielding a hybrid structure in which geometry is per-voxel while semantics are region-referenced and query cost scales with the number of stored regions rather than all points (Yamazaki et al., 2023).

A third substrate is the 3D point or mask representation aligned to language. OpenMask3D aggregates CLIP image embeddings over class-agnostic 3D instance masks, producing one language-aligned feature vector per mask (Takmaz et al., 2023). PGOV3D and related point-based systems instead align point embeddings directly with text embeddings learned from grounded 2D supervision, so the fundamental unit is the point rather than the mask or voxel (Zhang et al., 30 Jun 2025). These representational choices constrain granularity: dense voxel or point fields can in principle localize finer semantics, while object-level masks and boxes are structurally biased toward object categories.

3. Discovery mechanisms: grouping, clustering, and pseudo-label expansion

A central line of work discovers semantic units by aggregating multi-view evidence into 3D groups without training. OpenVoxel is the clearest example. Its grouping stage is bottom-up, view-aggregative, centroid-voting-based, and training-free. Each voxel stores a 3D vote $\mathcal{F}_{1:N}\in\mathbb{R}^{N\times 3}$ toward an instance centroid and a confidence weight $W_{1:N}\in\mathbb{R}^N$. Given a 2D segmentation map $M$, the centroid of each 2D instance is computed by masked averaging, voxel votes are accumulated with rendering weights $w_{ij}$, and final IDs are assigned by nearest-centroid matching: $$\mathrm{ID}_i = \arg\min_j \left\| \frac{\mathcal{F}_i}{W_i} - G_j \right\|_2.$$ Existing groups are matched to new SAM2 masks by highest IoU, unmatched masks become new groups, and SAM2 is re-prompted to merge overlapping or nested masks (Huang et al., 14 Jan 2026). Meaningfulness in this system is therefore consensus-based and view-consistent rather than learned through an explicit semantic purity objective.

OV-SAM3D uses a different training-free route. It begins with superpoints as initial 3D prompts, projects them into images as point prompts for SAM, back-projects the resulting masks into 3D, and accumulates overlap evidence in an overlapping score table $100\times100\times8$0. Coarse masks are then merged using the similarity score

$100\times100\times8$1

with $100\times100\times8$2 in the reported experiments (Tai et al., 2024). This discovers and labels 3D instances in a training-free manner, but the method intentionally filters non-instance tags and outputs a single flat label per instance.

A learned alternative is pseudo-label expansion. CoDA and CoDAv2 couple class-agnostic 3D detection with open-vocabulary semantics to discover novel 3D objects during training. A predicted 3D box is projected into 2D, CLIP produces a semantic distribution over a super-category vocabulary, and a box is accepted as a discovered novel object if it satisfies geometric confidence, semantic confidence, non-overlap with base boxes, and a label outside the seen set. CoDAv2 formalizes this as

$100\times100\times8$3

and updates the novel label pool by union (Cao et al., 2024). This mechanism is category-level rather than sub-concept-level, but it is an explicit example of iterative discovery and alignment in 3D.

DiSCO-3D is the first method in the provided literature that directly frames the broader OV-SD problem. It samples DINO and CLIP features from a frozen NeRF-style feature field, projects DINO features through a learnable projector, and computes soft assignments to $100\times100\times8$4 prototypes: $100\times100\times8$5 The unsupervised clustering is then weakly steered by a query-derived relevance mask from the CLIP field and an irrelevance-guided loss that separates relevant from irrelevant prototypes (Petit et al., 19 Jul 2025). In this formulation, sub-concepts are not predefined categories but query-relevant prototypes that emerge from clustering the relevant regions of the scene.

4. Language interfaces, semantic normalization, and query handling

Open-vocabulary usability depends not only on discovering 3D units but also on how those units are named, normalized, and queried. OpenVoxel is the strongest example of a text-centric design. After grouping, each discovered voxel group is rendered across views and captioned with the Describe Anything Model; Qwen3-VL then canonicalizes the caption into a one-line schema beginning with a concrete noun and ordered as category noun, appearance details, function or part-of, and placement or relation. The scene map stores tuples $100\times100\times8$6, and query-time retrieval is performed by direct text-to-text reasoning over canonicalized captions rather than CLIP/BERT embedding similarity (Huang et al., 14 Jan 2026). This directly refutes the assumption that open-vocabulary 3D retrieval must use a shared latent text-embedding space.

Other systems retain embedding-based retrieval but enrich the language channel. MVOV3D corrects noisy multi-view 2D features by combining region-level CLIP vision features with text proposals from RAM++, DeCap, or BLIP, selecting the text proposal whose CLIP text embedding has the highest cosine similarity to the region embedding subject to threshold $100\times100\times8$7 (Yin et al., 28 Jun 2025). PGOV3D uses LLaVA-NeXT to produce per-view object vocabularies, grounds them with Grounded-SAM, and aligns 3D point features to CLIP text embeddings through cosine similarity; it then aggregates all per-view grounded vocabularies into a scene-level vocabulary for pseudo-labeling full scenes (Zhang et al., 30 Jun 2025).

Language can also be adapted rather than merely consumed. OpenDAS modifies the CLIP matcher used inside open-vocabulary segmentation systems by multimodal prompt tuning and a triplet-loss strategy with auxiliary negative queries. In the 3D setting, it plugs directly into OpenMask3D’s mask-query matching stage, improving the semantic discrimination of existing 3D proposals without changing the proposal mechanism itself (Yilmaz et al., 2024). This suggests that sub-concept discovery pipelines can be divided into a discovery component and a language-matching component, and that the latter can be specialized independently.

5. Empirical evidence and task-specific performance

Direct OV-SD evaluation is reported by DiSCO-3D on LeRF and OpenNeRF feature fields. Under CLIP-matching evaluation on LeRF, DiSCO-3D reaches $100\times100\times8$8, $100\times100\times8$9, and $512$0, compared with $512$1, $512$2, and $512$3 for the OVS$512$4USS baseline and $512$5, $512$6, and $512$7 for USS$512$8OVS. On OpenNeRF, it reaches $512$9, $\{V_i\}_{i=1}^N$0, and $\{V_i\}_{i=1}^N$1. The same paper also shows that OV-SD includes open-vocabulary segmentation and unsupervised segmentation as edge cases: on LeRF class-level OV-Seg, performance rises from $\{V_i\}_{i=1}^N$2 to $\{V_i\}_{i=1}^N$3 in $\{V_i\}_{i=1}^N$4, and in the no-query USS regime DiSCO-3D reaches $\{V_i\}_{i=1}^N$5 $\{V_i\}_{i=1}^N$6 and $\{V_i\}_{i=1}^N$7 $\{V_i\}_{i=1}^N$8 (Petit et al., 19 Jul 2025).

Several precursor systems provide strong empirical evidence for object-centric and region-centric discovery. OpenVoxel reports $\{V_i\}_{i=1}^N$9 $$\mathbf{C}(\mathbf{r}) = \sum_{i<N} w_i(\mathbf{r}) \cdot c_i,\qquad w_i(\mathbf{r}) = \alpha_i(\mathbf{r}) \cdot \prod_{j<i}(1-\alpha_j(\mathbf{r})).$$0 on Ref-LeRF for referring expression segmentation, $$\mathbf{C}(\mathbf{r}) = \sum_{i<N} w_i(\mathbf{r}) \cdot c_i,\qquad w_i(\mathbf{r}) = \alpha_i(\mathbf{r}) \cdot \prod_{j<i}(1-\alpha_j(\mathbf{r})).$$1 average $$\mathbf{C}(\mathbf{r}) = \sum_{i<N} w_i(\mathbf{r}) \cdot c_i,\qquad w_i(\mathbf{r}) = \alpha_i(\mathbf{r}) \cdot \prod_{j<i}(1-\alpha_j(\mathbf{r})).$$2 on LeRF-OVS, and $$\mathbf{C}(\mathbf{r}) = \sum_{i<N} w_i(\mathbf{r}) \cdot c_i,\qquad w_i(\mathbf{r}) = \alpha_i(\mathbf{r}) \cdot \prod_{j<i}(1-\alpha_j(\mathbf{r})).$$3 $$\mathbf{C}(\mathbf{r}) = \sum_{i<N} w_i(\mathbf{r}) \cdot c_i,\qquad w_i(\mathbf{r}) = \alpha_i(\mathbf{r}) \cdot \prod_{j<i}(1-\alpha_j(\mathbf{r})).$$4 / $$\mathbf{C}(\mathbf{r}) = \sum_{i<N} w_i(\mathbf{r}) \cdot c_i,\qquad w_i(\mathbf{r}) = \alpha_i(\mathbf{r}) \cdot \prod_{j<i}(1-\alpha_j(\mathbf{r})).$$5 $$\mathbf{C}(\mathbf{r}) = \sum_{i<N} w_i(\mathbf{r}) \cdot c_i,\qquad w_i(\mathbf{r}) = \alpha_i(\mathbf{r}) \cdot \prod_{j<i}(1-\alpha_j(\mathbf{r})).$$6 on LeRF-Mask. It also reports practical efficiency: roughly $$\mathbf{C}(\mathbf{r}) = \sum_{i<N} w_i(\mathbf{r}) \cdot c_i,\qquad w_i(\mathbf{r}) = \alpha_i(\mathbf{r}) \cdot \prod_{j<i}(1-\alpha_j(\mathbf{r})).$$7 minutes per scene for grouping plus scene-map construction on a single RTX 5090, query inference in less than $$\mathbf{C}(\mathbf{r}) = \sum_{i<N} w_i(\mathbf{r}) \cdot c_i,\qquad w_i(\mathbf{r}) = \alpha_i(\mathbf{r}) \cdot \prod_{j<i}(1-\alpha_j(\mathbf{r})).$$8 second, and better RES performance than several embedding-aligned baselines despite using no ground-truth description-mask annotations (Huang et al., 14 Jan 2026). OpenMask3D, in turn, establishes the strength of mask-centric CLIP alignment: on ScanNet200 it reaches $$\mathbf{C}(\mathbf{r}) = \sum_{i<N} w_i(\mathbf{r}) \cdot c_i,\qquad w_i(\mathbf{r}) = \alpha_i(\mathbf{r}) \cdot \prod_{j<i}(1-\alpha_j(\mathbf{r})).$$9, $\mathcal{F}_{1:N}\in\mathbb{R}^{N\times 3}$0, $\mathcal{F}_{1:N}\in\mathbb{R}^{N\times 3}$1, with tail $\mathcal{F}_{1:N}\in\mathbb{R}^{N\times 3}$2, and on Replica it reports $\mathcal{F}_{1:N}\in\mathbb{R}^{N\times 3}$3, $\mathcal{F}_{1:N}\in\mathbb{R}^{N\times 3}$4, $\mathcal{F}_{1:N}\in\mathbb{R}^{N\times 3}$5 (Takmaz et al., 2023).

Point- and voxel-field systems emphasize large-vocabulary and long-tail generalization. MVOV3D reaches $\mathcal{F}_{1:N}\in\mathbb{R}^{N\times 3}$6 $\mathcal{F}_{1:N}\in\mathbb{R}^{N\times 3}$7 on ScanNet200 and $\mathcal{F}_{1:N}\in\mathbb{R}^{N\times 3}$8 $\mathcal{F}_{1:N}\in\mathbb{R}^{N\times 3}$9 on Matterport160 for open-vocabulary semantic segmentation, with notable tail-class results such as $W_{1:N}\in\mathbb{R}^N$0 $W_{1:N}\in\mathbb{R}^N$1 on Matterport160 tail classes (Yin et al., 28 Jun 2025). PGOV3D reports $W_{1:N}\in\mathbb{R}^N$2 $W_{1:N}\in\mathbb{R}^N$3 and $W_{1:N}\in\mathbb{R}^N$4 $W_{1:N}\in\mathbb{R}^N$5 on ScanNet, $W_{1:N}\in\mathbb{R}^N$6 $W_{1:N}\in\mathbb{R}^N$7 and $W_{1:N}\in\mathbb{R}^N$8 $W_{1:N}\in\mathbb{R}^N$9 on ScanNet200, and $M$0 $M$1 in ScanNet$M$2S3DIS zero-shot transfer (Zhang et al., 30 Jun 2025). OpenOcc reports $M$3 $M$4 on Replica, $M$5 $M$6 on ScanNet-200, and strong improvements on long-tail categories such as clock, light, fan, and book (Jiang et al., 2024). These results do not directly measure sub-concept discovery, but they indicate that preserving richer open-vocabulary semantics in 3D materially improves long-tail and fine-grained behavior.

6. Limitations, misconceptions, and open directions

The most persistent limitation is granularity. OpenVoxel explicitly notes that if a query refers to a part of a larger object, such as the “flash light of the camera,” it tends to return the whole camera because the grouping stage bundles small parts into the object-level group (Huang et al., 14 Jan 2026). OV-SAM3D likewise targets whole-instance masks with single labels and deliberately filters colors, material properties, and other non-instance tags, which limits compositional output (Tai et al., 2024). Open-Fusion stores one semantic key and confidence per voxel, without overlapping semantic layers for object, part, and attribute, so richly layered semantics are not explicitly represented (Yamazaki et al., 2023).

A second limitation is that many systems are discovery-capable only in a weak sense. OpenMask3D is highly effective for language-driven retrieval over 3D instances, but it does not cluster emergent sub-concepts or discover latent taxonomies (Takmaz et al., 2023). POP-3D can localize fine-grained phrases such as “building door,” “tire,” “stairs,” and “zebra crossing,” yet its coarse $M$7 voxel grid and reliance on MaskCLIP+ constrain small objects and fine semantic detail (Vobecky et al., 2024). PGOV3D introduces scene-adaptive vocabularies and dense partial-scene supervision, but each point is still aligned to a single textual entity and there is no explicit part-level supervision, hierarchy, or multi-label semantics (Zhang et al., 30 Jun 2025).

A third issue is dependence on upstream foundation models and priors. OpenVoxel depends on SAM2 quality, scene reconstruction quality, and MLLM caption quality (Huang et al., 14 Jan 2026). MVOV3D’s performance depends on the quality of region masks, captions or tags, and dense VLM features (Yin et al., 28 Jun 2025). DiSCO-3D is query-specific and requires optimization per query; it is fast enough for many uses but not a purely feed-forward retrieval model, and its failures often trace to the quality of the underlying CLIP/OpenSeg relevance field or DINO segmentation tendency (Petit et al., 19 Jul 2025).

These patterns suggest a sober interpretation of the field. A strong form of 3D Open-Vocabulary Sub-concepts Discovery would require scene-adaptive yet query-aware decomposition, explicit support for parts and attributes, multi-label or hierarchical semantics, and robust naming of discovered units. Current methods already provide several essential ingredients—training-free 3D grouping, dense 3D language fields, scene-adaptive vocabularies, multi-view semantic fusion, prototype-based unsupervised decomposition, and text-query interfaces—but most existing systems remain biased toward object-level or region-level semantic units rather than systematic part-level conceptual structure. A plausible implication is that future progress will come less from a single monolithic architecture than from tighter coupling of three components that already exist separately in the literature: precise 3D over-segmentation or grouping, open-vocabulary language grounding or canonicalization, and query-conditioned clustering or prototype discovery over scene-consistent feature fields.