DiSCO-3D: 3D Open-Vocabulary Sub-Concept Discovery

• DiSCO-3D is a NeRF-based approach that discovers fine-grained semantic sub-concepts in 3D scenes guided by open-vocabulary queries.
• It integrates unsupervised segmentation with weak open-vocabulary guidance to adaptively segment scene content without explicit class enumeration.
• The method employs prototype-based clustering and NeRF-aware optimization, achieving state-of-the-art performance on benchmarks like Extended Replica.

DiSCO-3D is a NeRF-based method for 3D Open-Vocabulary Sub-Concepts Discovery (OV-SD), a setting in which a 3D scene and one or more open-vocabulary queries are used to automatically discover and segment the most relevant and fine-grained semantic sub-concepts present in the scene, without requiring explicit enumeration of all classes. In this formulation, a query such as “furniture,” “eat,” or “soft” is not treated as a terminal label; instead, the method seeks scene-adapted sub-concepts such as sofa, chair, and table, while also separating irrelevant content. The method is built on neural fields, combining unsupervised segmentation with weak open-vocabulary guidance, and is presented as the first method addressing the broader problem of 3D OV-SD while also obtaining state-of-the-art results in the edge cases of both open-vocabulary and unsupervised segmentation (Petit et al., 19 Jul 2025).

1. Problem setting and semantic scope

DiSCO-3D formalizes a setting that differs from both standard 3D open-vocabulary segmentation and unsupervised semantic segmentation. In conventional open-vocabulary segmentation, the model labels queried concepts but does not automatically discover finer subclasses. In unsupervised semantic segmentation, the model discovers groups without labels, but the resulting clusters may not align with a user’s task. DiSCO-3D bridges these two regimes by adapting segmentation simultaneously to scene content and to user queries (Petit et al., 19 Jul 2025).

The scene is represented by a neural radiance field $F_\theta$. A standard NeRF maps a 3D location $\mathbf{x}\in\mathbb{R}^3$ and view direction $\mathbf{d}\in S^2$ to color $\mathbf{c}$ and density $\sigma$:

$$F_\theta(\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma).$$

Rendering along a camera ray $r$ follows standard volume compositing:

$$\hat{C}(r) = \sum_{i=0}^{N-1} w_i \, \mathbf{c}_i, \quad w_i = T_i \big(1 - e^{-\sigma_i \delta_i} \big), \quad T_i = \exp\!\Big(-\sum_{j=0}^{i-1}\sigma_j \delta_j\Big),$$

with photometric supervision

$$\mathcal{L}_{\text{photo}} = \|\hat{C}(r) - C(r)\|_2^2.$$

Beyond RGB, the method assumes frozen feature fields. The scene contains a queryable open-vocabulary field, such as CLIP or OpenSeg, and a spatially precise semantic field, such as DINO. In the LeRF setting, CLIP is built as a multi-scale patch pyramid and DINO is dense per-pixel; in the OpenNeRF setting, OpenSeg provides dense CLIP features and is also used as the precise field for projection and clustering. If $f_k$ denotes the semantic feature at sample $\mathbf{x}\in\mathbb{R}^3$0 and $\mathbf{x}\in\mathbb{R}^3$1 the associated open-vocabulary feature, then a text query $\mathbf{x}\in\mathbb{R}^3$2 is encoded as an embedding $\mathbf{x}\in\mathbb{R}^3$3 (Petit et al., 19 Jul 2025).

The OV-SD objective is to discover $\mathbf{x}\in\mathbb{R}^3$4 relevant sub-concepts for a query and segment the scene into those sub-concepts plus an irrelevant class. The output consists of learned sub-concept prototypes in a projector latent space, CLIP prototypes $\mathbf{x}\in\mathbb{R}^3$5 summarizing the discovered sub-concepts, and a per-3D-point assignment to at most one sub-concept or the irrelevant class. Multiple queries $\mathbf{x}\in\mathbb{R}^3$6 are allowed, and their relevant sub-concepts may be overlapping, disjoint, or nested.

2. Core architecture and prototype-based discovery

The method takes as input a frozen pre-trained NeRF, specifically Nerfacto, together with frozen feature fields from either LeRF or OpenNeRF. On top of these fields, DiSCO-3D learns a non-linear projector $\mathbf{x}\in\mathbb{R}^3$7 that maps semantic features into a latent space in which prototypes act as semantic centroids (Petit et al., 19 Jul 2025).

The projector is an MLP with SiLU activations plus a linear residual branch, following the style of SmooSeg, with dropout $\mathbf{x}\in\mathbb{R}^3$8. Prototypes are initialized to zero and divided into relevant and irrelevant sets. The method typically uses $\mathbf{x}\in\mathbb{R}^3$9 prototypes, including several irrelevant prototypes. At each sampled point $\mathbf{d}\in S^2$0, the projected feature is

$\mathbf{d}\in S^2$1

Assignments to prototypes are computed with a softmax over cosine similarities:

$\mathbf{d}\in S^2$2

The sharpness parameter $\mathbf{d}\in S^2$3 is scheduled linearly, for example from $\mathbf{d}\in S^2$4 to $\mathbf{d}\in S^2$5, so that assignments become progressively sharper during optimization.

Prototype updates use an exponential moving average weighted by both NeRF density weights $\mathbf{d}\in S^2$6 and assignment confidences $\mathbf{d}\in S^2$7. With $\mathbf{d}\in S^2$8 denoting the set of samples associated with prototype $\mathbf{d}\in S^2$9, the semantic prototype update is

$\mathbf{c}$0

and the CLIP companion prototype is updated analogously:

$\mathbf{c}$1

The paper reports $\mathbf{c}$2, and samples with low confidence or low density weight can be filtered, for example when $\mathbf{c}$3 or $\mathbf{c}$4.

This prototype mechanism is designed to be NeRF-aware. Density weighting suppresses free-space and inside-object artifacts, while assignment-weighted EMA stabilizes cluster evolution. The presence of irrelevant prototypes is central: rather than forcing every region into a query-aligned class, the method explicitly models semantic remainder.

3. Optimization objectives and query guidance

DiSCO-3D optimizes the projector using three coupled losses: a correlation-preserving unsupervised segmentation objective, an open-vocabulary irrelevance guidance term, and a CLIP-prototype regularizer (Petit et al., 19 Jul 2025).

The unsupervised semantic segmentation term preserves correlations in the original semantic feature space:

$\mathbf{c}$5

where $\mathbf{c}$6 is a margin threshold, $\mathbf{c}$7 is batch size, and the dot product is taken over class posteriors. This loss keeps similar features close in posterior space and discourages identical assignments for dissimilar features.

Open-vocabulary guidance is introduced through a query-dependent relevance mask. For query embedding $\mathbf{c}$8, the method defines

$\mathbf{c}$9

where $\sigma$0 is a threshold tuned per feature field, for example $\sigma$1 for LeRF and $\sigma$2 for OpenNeRF. A binary vector $\sigma$3 marks the prototypes intended to be relevant for query $\sigma$4. The irrelevance-guidance loss is

$\sigma$5

This penalizes relevant-prototype assignments outside the query-relevant region and penalizes irrelevant-prototype assignments inside it. Functionally, it pushes the unsupervised clustering process toward discovering sub-concepts of the query rather than arbitrary latent partitions.

The third term regularizes DINO-space clustering with CLIP-space structure. For each sample,

$\sigma$6

and with $\sigma$7 the one-hot vector at $\sigma$8, the regularizer is

$\sigma$9

This reduces over-segmentation and introduces object-level semantics from the CLIP branch into DINO-driven clustering.

The total objective is

$$F_\theta(\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma).$$0

Typical weights are reported as $$F_\theta(\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma).$$1, $$F_\theta(\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma).$$2, and $$F_\theta(\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma).$$3. Optimization uses Adam with a decaying learning rate, for example $$F_\theta(\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma).$$4.

4. Inference, outputs, and operating modes

At inference time, DiSCO-3D returns both semantic structure and usable scene annotations. The learned relevant prototypes $$F_\theta(\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma).$$5 define the discovered sub-concepts, and their CLIP companions $$F_\theta(\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma).$$6 provide a retrieval-oriented summary of each sub-concept (Petit et al., 19 Jul 2025).

Per-view class distributions are rendered from the neural field, then back-projected onto a common 3D point cloud. Probabilities are aggregated across views and a final class is assigned by $$F_\theta(\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma).$$7. This yields 3D masks for each discovered sub-concept and for the irrelevant class. Because CLIP prototypes are maintained throughout optimization, discovered clusters can optionally be linked post hoc to textual labels by nearest-neighbor comparison in CLIP space.

The method supports multiple operating modes. It can take a text query encoded by CLIP or OpenSeg, a visual example encoded by CLIP, or a user click mapped to a feature embedding. Multiple simultaneous queries are handled by defining one relevance mask $$F_\theta(\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma).$$8 per query and adding the corresponding $$F_\theta(\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma).$$9 terms linearly. The paper states that disjoint, overlapping, and nested semantics are all supported.

Implementation is reported in Nerfstudio, using Nerfacto with Mip-NeRF-360 improvements. Optimization requires 100–200 epochs per query on a single RTX 4090, with approximately 20–22 ms per epoch and roughly 2–4 s total per query. This makes the method short-horizon rather than instantaneous: it is fast enough for interactive analysis in many research settings, but it remains a per-query optimization procedure rather than a feed-forward predictor.

5. Evaluation protocol and empirical performance

The principal dataset is Extended Replica, comprising 8 indoor scenes and 40 query concepts designed with robotics in mind. Queries include categories, properties, and actions, with examples such as “furniture,” “seating,” “sleeping,” “light,” “eat,” “soft,” and “ventilation.” Evaluation is performed on a 3D point cloud obtained by aggregating per-view predictions, and discovered sub-concepts are matched to ground-truth sub-concepts by CLIP embedding distance, with Hungarian matching also reported as an upper-bound semantic matching analysis (Petit et al., 19 Jul 2025).

The reported metrics are mean Intersection-over-Union (mIoU), mean Accuracy (mAcc), and Panoptic Quality (PQ), together with Recognition Quality (RQ) and Segmentation Quality (SQ), where

$r$0

Successive baselines for OV-SD are constructed by combining open-vocabulary segmentation and unsupervised segmentation in sequence: USS$r$1OVS and OVS$r$2USS, with K-Means variants. The architecture and feature fields are kept fixed across DiSCO-3D and these baselines so that the principal variable is joint versus successive integration.

Feature field	DiSCO-3D (PQ / mIoU / mAcc)	Successive baselines
LeRF	8.13 / 10.79 / 33.39	USS→OVS: 4.76 / 6.52 / 22.54; OVS→USS: 5.99 / 8.71 / 21.44
OpenNeRF	8.65 / 10.82 / 19.24	USS→OVS: 4.97 / 6.08 / 13.98; OVS→USS: 5.47 / 8.94 / 13.56

Averaged across feature fields, the gains over successive baselines are reported as PQ +47–72%, mIoU +22–71%, and mAcc +42–44%. Under Hungarian matching, DiSCO-3D reaches PQ 10.19, mIoU 12.77, mAcc 44.29 with LeRF, and PQ 10.49, mIoU 12.69, mAcc 29.06 with OpenNeRF. The paper notes that about 76% of matches coincide with CLIP-based matching, with the remaining mismatches typically involving semantically close labels such as “blanket” versus “comforter.”

The method is also evaluated on the edge cases that motivated it. For single-query open-vocabulary segmentation, DiSCO-3D improves LeRF from mIoU 8.79 to 12.42 and mAcc 84.53 to 87.93, and improves concept-level LeRF scores from mIoU 10.42 to 15.78 and mAcc 37.79 to 46.73. With OpenNeRF, class-level performance changes from mIoU 21.60 to 21.87 and mAcc 91.87 to 92.66, while concept-level performance changes from mIoU 15.59 to 16.69 and mAcc 42.69 to 58.17. In the mono-label paradigm, LeRF improves from mIoU 10.49 to 13.43 and mAcc 22.02 to 28.37; OpenNeRF changes from mIoU 19.08 to 20.76 and mAcc 31.96 to 30.19, which the paper describes as a trade-off between better boundaries and minor misclassification of small classes.

For unsupervised semantic segmentation, evaluated on the top-10 classes per scene with Hungarian matching, DiSCO-3D reports mIoU 27.47 and mAcc 51.99, compared with K-Means on NeRF features at 27.00 / 50.14, GrowSP at 21.62 / 34.24, and SmooSeg (2D+NeRF) at 10.49 / 31.07. This places DiSCO-3D at the top of the reported USS evaluations while preserving the broader OV-SD capability.

Ablation studies attribute performance gains to several specific components. Removing $r$3 scheduling, removing the density or assignment weights in the EMA updates, or replacing the projector with a simpler linear alternative all reduce performance. The paper also reports that $r$4 regularizes the number of prototypes used, reducing unused or over-splitting behavior, and that a fixed prototype budget of $r$5 performs well without prior knowledge of the true number of classes.

6. Relation to adjacent methods, limitations, and research significance

DiSCO-3D is positioned relative to three neighboring research lines. First, NeRF-based open-vocabulary methods such as LeRF, OpenNeRF, and LEGaussians distill 2D encoders into 3D representations and segment queried concepts, but do not automatically discover sub-concepts and often output relevancy heatmaps rather than hard, query-adaptive semantic groups. Second, 2D unsupervised semantic segmentation methods such as STEGO, ACSeg, EAGLE, and SmooSeg discover semantic clusters but are not query-conditioned and, in 2D form, do not enforce multi-view consistency. Third, 3D unsupervised segmentation on point clouds, including GrowSP and U3DS3, is also task-agnostic. Against this background, the paper characterizes DiSCO-3D as the first method to formulate and solve OV-SD in 3D neural fields, the first USS variant tailored to NeRF, and the first to jointly integrate USS with weak open-vocabulary guidance rather than composing them successively (Petit et al., 19 Jul 2025).

The reported qualitative behavior is consistent with that positioning. DiSCO-3D is described as mitigating the “holes” and “spilling” common in open-vocabulary segmentation, supporting text, visual-example, and click-based queries, and handling multiple simultaneous queries with disjoint, overlapping, or nested semantics. This suggests that its primary contribution is not merely a new loss function, but a redefinition of the task boundary between discovery and query-driven retrieval.

The paper also states several limitations. Performance depends on the quality of the feature fields: inaccurate open-vocabulary fields can misguide discovery, and noisy DINO features may over-segment at the part level. The CLIP-prototype regularizer helps but does not fully correct failures inherited from the inputs. The method also depends on camera calibration and NeRF quality, since poor poses or reconstruction quality degrade 3D consistency and feature fidelity. Finally, although per-query optimization is short, it is still not instantaneous; the reported runtime is about 2 seconds per query, and the authors identify amortization, meta-learning, or real-time variants as future directions.

Proposed extensions include stronger geometry-aware regularization such as CRF-like consistency or surface smoothness, and adaptation to other representations, including SAM-based semantic feature fields, Garfield instance masks, 3D Gaussians, or point clouds, provided that a queryable feature space and a precise segmentation feature field are available. In that sense, DiSCO-3D is both a concrete method and a task definition: it reframes 3D semantic segmentation as query-adaptive sub-concept discovery rather than as either fixed-label recognition or unconstrained clustering alone.