SegDINO3D: 2D-Driven 3D Instance Segmentation

• The paper introduces a novel Transformer encoder-decoder that fuses pre-trained 2D detection features with 3D point cloud data to enhance instance segmentation.
• It employs a dual-level design by enriching 3D points with image-level features in the encoder and refining object queries in the decoder with 2D object-level cues.
• The approach achieves state-of-the-art performance on ScanNetV2 and ScanNet200 benchmarks while offering faster training convergence and improved memory efficiency.

Searching arXiv for SegDINO3D and closely related 2D–3D segmentation work to ground the article in papers. SegDINO3D is a Transformer encoder-decoder framework for 3D instance segmentation that is designed to fully leverage 2D representation from a pre-trained 2D detection model, including both image-level and object-level features, for improving 3D representation. It takes both a point cloud and its associated 2D images as input, enriches 3D points with retrieved 2D image features in the encoder, and formulates 3D object queries as 3D anchor boxes in the decoder, where they cross-attend to 2D object queries obtained from 2D images using the 2D detection model. The method is presented as a response to the scarcity of annotated 3D data relative to 2D training images, and it reports state-of-the-art performance on the ScanNetV2 and ScanNet200 3D instance segmentation benchmarks (Qu et al., 19 Sep 2025).

1. Concept and motivation

SegDINO3D is introduced as a method for 3D instance segmentation under the premise that 3D training data is generally not as sufficient as 2D training images. The central design objective is therefore to exploit the semantic knowledge encoded in pre-trained 2D detection models while preserving the geometric structure needed for 3D mask prediction (Qu et al., 19 Sep 2025).

The framework is described as leveraging both image-level and object-level 2D features extracted from powerful, pre-trained 2D detection models such as Grounding DINO or DINO-X. Image-level features provide pixel-wise semantic evidence aligned to 3D points, whereas object-level features provide instance-wise summaries that can be used more compactly in the decoder. This dual use is intended to avoid two limitations that recur in 2D–3D transfer: first, loss of semantic discrimination when only local image features are transferred; second, excessive memory demand when thousands of image feature maps must be retained throughout the model (Qu et al., 19 Sep 2025).

A plausible implication is that SegDINO3D occupies a specific position within the broader 2D-to-3D transfer literature. Compared with semantic segmentation approaches that inject 2D foundation-model features into 3D backbones, such as DITR, SegDINO3D is oriented toward instance segmentation and makes object-level 2D representations explicit in the decoder rather than limiting transfer to projected per-point image features (Zeid et al., 24 Mar 2025). Relative to language-conditioned systems such as OpenMaskDINO3D, it does not foreground natural-language reasoning or a special segmentation token, but instead emphasizes structured cross-attention between 3D object queries and 2D object queries (Zhang, 5 Jun 2025).

2. Input representation and encoder-stage fusion

SegDINO3D takes as input a 3D point cloud $\mathbf{P} \in \mathbb{R}^{N \times 3}$, $V$ posed RGB images $$\mathbf{I} \in \mathbb{R}^{V \times H \times W \times 3}$$, and corresponding depth maps $\mathbf{D} \in \mathbb{R}^{V \times H \times W}$. The encoder stage begins with 2D feature extraction using a frozen, pre-trained 2D DETR-based detection model, for example DINO-X or Grounding DINO, from which image-level feature maps $$\mathbf{X} \in \mathbb{R}^{V \times h \times w \times C}$$ are obtained (Qu et al., 19 Sep 2025).

Each 3D point is projected into all camera views where it is visible using intrinsic and extrinsic parameters. For each point, the top-$k$ nearest views, with default $k=3$, are selected by camera center distance. The image-level feature for point $i$ is then formed by averaging bilinearly sampled features from the selected views: $$\mathbf{f}^{2D}_i = \frac{1}{k} \sum_{v \in \mathrm{Nst}(\bm{\rho}_i, \bm{\delta}_i, k)} \operatorname{Bili}(\mathbf{X}_v, [u^v_i, v^v_i]).$$ This procedure is described as using the most detailed, least distorted image evidence for each point (Qu et al., 19 Sep 2025).

After retrieval, each 3D point’s semantic feature $\mathbf{f}_i^{2D}$ is concatenated with its geometric coordinate and passed to a 3D encoder: $V$0 The 3D encoder is described as a UNet-like sparse ConvNet and serves for global contextualization. The encoded point features are then grouped into superpoints, defined as clusters of geometrically homogeneous points, and mean-pooled into $V$1 (Qu et al., 19 Sep 2025).

This encoder-stage design parallels a broader trend in recent 3D methods that project 2D features to 3D points and then fuse them with a 3D backbone. In DITR, for example, frozen DINOv2 features are projected to visible 3D points and injected into a 3D point cloud segmentation model, with performance improvements attributed to semantically rich 2D features transferred through calibrated image–point correspondences (Zeid et al., 24 Mar 2025). SegDINO3D extends this pattern by coupling per-point image-level transfer with a second, object-level transfer pathway in the decoder.

3. Decoder architecture and object-query design

The decoder stage centers on 3D object queries that are explicitly formulated as 3D anchor boxes. At training time, $V$2 superpoints are randomly sampled, with $V$3, as initial 3D object queries. Each query consists of a content term $V$4 initialized from superpoint features and a position term $V$5 encoding box center $V$6 and box size $V$7 (Qu et al., 19 Sep 2025).

The decoder layer is repeated $V$8 times. Within each layer, a first component, Box-Modulated Cross-Attention (BMCA-3D), lets the 3D object queries attend to superpoint features. This mechanism combines content similarity with positional similarity modulated by box size. The paper summary gives the content and positional similarity as

$V$9

and

$$\mathbf{I} \in \mathbb{R}^{V \times H \times W \times 3}$$0

where

$$\mathbf{I} \in \mathbb{R}^{V \times H \times W \times 3}$$1

Here, $$\mathbf{I} \in \mathbb{R}^{V \times H \times W \times 3}$$2 denotes a sinusoidal position encoding. Attention weights are further masked by predicted 3D masks in order to avoid cross-instance confusion (Qu et al., 19 Sep 2025).

A second component, Distance-Aware Cross-Attention to 2D Object Queries (DACA-2D), performs cross-attention from 3D queries to 2D object queries. The attention update is

$$\mathbf{I} \in \mathbb{R}^{V \times H \times W \times 3}$$3

The attention mask is distance-aware: a 3D query attends only to 2D objects mapped near the 3D query region, determined via the assignment of superpoints to both 3D and 2D objects. After these cross-attention steps, self-attention and feed-forward layers further refine the 3D object queries (Qu et al., 19 Sep 2025).

The box state is iteratively refined according to

$$\mathbf{I} \in \mathbb{R}^{V \times H \times W \times 3}$$4

and each query produces a 3D segmentation mask over all superpoints via similarity with $$\mathbf{I} \in \mathbb{R}^{V \times H \times W \times 3}$$5. The introduction of 3D box queries is described as enabling the model to modulate cross-attention using the predicted boxes for more precise querying (Qu et al., 19 Sep 2025).

4. Image-level and object-level 2D knowledge transfer

A defining characteristic of SegDINO3D is its dual-level use of 2D information. In the encoder, image-level features are retrieved per 3D point from visible image views and incorporated before 3D context fusion. In the decoder, object-level features are represented by 2D object queries from the pre-trained 2D detection model and serve as a compact object-level representation of 2D images (Qu et al., 19 Sep 2025).

For 2D object query extraction, the 2D DETR-based model outputs a set of query embeddings representing object proposals and semantics. For each high-confidence 2D object, its mask is projected into 3D using depth and camera parameters, and its center is robustly estimated using Partitioning Around Medoids to remove outlier depth errors. The resulting set is downsampled to $$\mathbf{I} \in \mathbb{R}^{V \times H \times W \times 3}$$6 spatially even objects per scene with Farthest Point Sampling, yielding query features $$\mathbf{I} \in \mathbb{R}^{V \times H \times W \times 3}$$7 (Qu et al., 19 Sep 2025).

This design is motivated in memory terms as well as representational terms. The 2D object queries are described as effectively avoiding the challenge of keeping thousands of image feature maps in the memory while faithfully preserving the knowledge of the pre-trained 2D model. The method therefore attempts to preserve dense semantic evidence through image-level transfer while simultaneously providing a compressed object-centric summary for query refinement (Qu et al., 19 Sep 2025).

The distinction between image-level and object-level transfer situates SegDINO3D within a wider set of 2D-assisted 3D approaches. DITR uses patch-wise features from a frozen DINOv2 ViT, projects them to 3D, and injects them into a Point Transformer V3-based decoder through skip connections, demonstrating that direct injection of semantically rich 2D features significantly boosts 3D segmentation performance (Zeid et al., 24 Mar 2025). OpenMaskDINO3D also combines 3D and 2D evidence, extracting 2D features from multi-view RGB images using a pretrained DINOv2 vision transformer and feeding fused object and text representations to an LLM-guided mask decoder (Zhang, 5 Jun 2025). SegDINO3D differs in that the object-level representation is not primarily language-mediated; instead, it remains embedded in the DETR-style query mechanism of the decoder.

5. Training objectives, efficiency, and ablation findings

The training objective combines standard detection segmentation losses—classification, BCE/Dice mask, and semantic losses—with an additional L1 loss for box regression. During training, random dropout is used for 2D object queries as augmentation (Qu et al., 19 Sep 2025).

Several ablations are reported. The dual use of image-level and object-level 2D features yields a gain of +3.1 mAP over using either alone. Box-modulated attention yields +1.5 mAP over using only content or center similarity. Distance-aware 2D query masking yields +3.5 mAP. Global 3D context fusion yields +3.9 mAP over mere MLP fusion (Qu et al., 19 Sep 2025).

The paper summary also reports that training converges approximately $$\mathbf{I} \in \mathbb{R}^{V \times H \times W \times 3}$$8 faster than previous best transformer segmentation methods such as OneFormer3D. This is attributed to the model’s combination of early semantic injection, efficient object-query representations, and spatially informed attention, though the causal wording in the source is interpretive rather than formal (Qu et al., 19 Sep 2025).

A plausible implication is that the efficiency claim depends on the compactness of the decoder-side representation. Whereas maintaining large banks of multi-view image features can impose substantial memory costs, SegDINO3D converts high-confidence 2D detections into a bounded set of object-level queries. The source explicitly frames this as a memory-efficiency mechanism and as a practical way to process scenes with thousands of images (Qu et al., 19 Sep 2025).

6. Empirical results and comparative context

SegDINO3D is reported to achieve state-of-the-art performance on both ScanNetV2 and ScanNet200 3D instance segmentation benchmarks. On the ScanNet200 validation set, it achieves mAP 40.2 versus the previous best, ODIN, at 31.5, corresponding to a gain of +8.7 mAP. On the ScanNet200 hidden test set, it achieves mAP 34.6 versus the previous best 27.8, corresponding to +6.8 mAP. On ScanNetV2, it achieves mAP 64.0 and is described as outperforming all previous methods (Qu et al., 19 Sep 2025).

The reported gains are not uniform only in aggregate accuracy. The source states that improvements are consistent on head, common, and tail classes, especially long-tail classes, and qualitatively that the method yields more precise instance boundaries and fewer false positives (Qu et al., 19 Sep 2025).

For context, DITR reports state-of-the-art results on indoor and outdoor 3D semantic segmentation benchmarks by extracting 2D foundation-model features, projecting them to 3D, and injecting them into a 3D point cloud segmentation model. On ScanNet200 semantic segmentation it improves over PTv3 by +7.1 mIoU, and on nuScenes it improves by +4.3 mIoU (Zeid et al., 24 Mar 2025). OpenMaskDINO3D, in a different setting centered on reasoning segmentation, reports ScanNet results of [email protected] 54.21, [email protected] 39.14, and mIoU 39.81, together with gains over prior approaches such as Reason3D and Mask3D (Zhang, 5 Jun 2025). These comparisons indicate that recent systems exploiting 2D features in 3D tasks diverge along at least three axes: semantic versus instance segmentation, direct feature transfer versus object-query transfer, and pure vision conditioning versus language-conditioned reasoning.

7. Significance, limitations, and related directions

SegDINO3D is significant primarily because it formulates 2D-to-3D transfer at two representational levels within a single DETR-like framework: dense image-level transfer in the encoder and compact object-level transfer in the decoder. The source attributes its performance to comprehensive 2D-to-3D knowledge transfer, efficient and aligned feature fusion, memory and computational efficiency, spatially informed attention, better long-tail recognition, and faster convergence (Qu et al., 19 Sep 2025).

Its design also clarifies a recurrent distinction in modern 3D perception. Some methods, such as DITR, are built to retain strong 3D performance even when images are unavailable at inference through a distillation strategy called D-DITR, in which a 3D backbone is pretrained to predict DINOv2 features assigned to points and then fine-tuned on labeled 3D points only (Zeid et al., 24 Mar 2025). SegDINO3D, by contrast, is described as taking both a point cloud and its associated 2D images as input, so the integration of images is intrinsic to its formulation (Qu et al., 19 Sep 2025). This suggests that its strongest performance regime depends on multi-view image availability.

A further related direction appears in systems such as OpenMaskDINO3D, which introduce a special [SEG] token and learnable object identifier tokens <OBJi> so that an LLM can process point cloud data and text prompts to produce 3D instance segmentation masks under natural-language instructions (Zhang, 5 Jun 2025). SegDINO3D does not incorporate this reasoning-oriented interface. Its contribution instead lies in the architectural integration of pre-trained 2D detection knowledge into 3D instance segmentation through anchor-box-based object queries, box-modulated attention, and distance-aware cross-attention.

Taken together, these neighboring works suggest an emerging synthesis in 3D segmentation research: semantically rich 2D representations are increasingly used not merely as auxiliary inputs, but as structured priors that shape 3D query formation, feature fusion, and mask decoding. Within that synthesis, SegDINO3D is characterized by its explicit use of both image-level and object-level 2D features and by the empirical gains it reports on ScanNetV2 and ScanNet200 (Qu et al., 19 Sep 2025).