Open-Vocabulary 3D Object Detection

• Open-Vocabulary 3D Object Detection is the task of localizing and recognizing objects beyond fixed taxonomies by combining 3D geometry with cross-modal semantic cues.
• It employs methods such as pseudo-label generation, cross-modal alignment, and foundation model transfer to bridge 2D vision-language knowledge with 3D spatial data.
• The approach addresses challenges of scarce 3D annotations by decoupling localization from semantic recognition, enabling adaptive detection in diverse environments.

Open-vocabulary 3D object detection is the task of localizing and recognizing objects in 3D space beyond a fixed training taxonomy, typically by predicting 3D bounding boxes for both seen and unseen categories from point clouds, RGB images, RGB-D streams, or multi-view image collections. Across the recent literature, the field is driven by two persistent constraints: annotated 3D data are scarce, and the semantic richness available in large 2D vision-language training corpora is not natively aligned with 3D geometry. As a result, most methods explicitly separate or rebalance localization and recognition, using 3D structure for box estimation and 2D or text-aligned foundation models for semantics (Lu et al., 2022, Wang et al., 2024, Yao et al., 2024).

1. Problem setting and operational regimes

A canonical formalization appears in open-vocabulary monocular 3D detection: given an image $I \in \mathbb{R}^{H \times W \times 3}$ and a text prompt $T = [c_1, c_2, ..., c_M]$, the detector predicts $\{(c_i, B_i)\}_{i=1}^N$, where $B_i$ denotes a 3D bounding box in metric camera coordinates, with the vocabulary partitioned into $\mathcal{C}_{\text{base}}$ and $\mathcal{C}_{\text{novel}}$ (Yao et al., 2024). This prompt-conditioned formulation is also implicit in point-cloud and RGB-D systems such as OV-3DETIC, CoDA, and OpenNav, where inference is driven by an arbitrary list of categories or a natural-language object request (Lu et al., 2022, Cao et al., 2023, Rahman et al., 2024).

The modality regime, however, varies substantially. OV-Uni3DETR frames the task as a unified detector that can accept point clouds, RGB images, or both, supports test-time modality switching, and is intended for both indoor and outdoor scenes (Wang et al., 2024). Other systems specialize in narrower settings: OpenNav operates on RGB-D images for smart wheelchair navigation (Rahman et al., 2024); BoxFusion assumes streaming posed RGB-D video (Lan et al., 18 Jun 2025); SMOV3D assumes a sparse set of posed RGB images (Moliner et al., 19 Sep 2025); Group3D supports pose-known and pose-free multi-view RGB (Kim et al., 23 Mar 2026); Zoo3D extends to point clouds, posed images, and unposed images (Lemeshko et al., 25 Nov 2025); and OVM3D-Det and OVMono3D-LIFT study monocular RGB-only 3D detection (Huang et al., 2024, Yao et al., 2024).

A central terminological distinction concerns what “open-vocabulary” means operationally. Most current formulations remain prompt-driven: the detector localizes objects from an arbitrary list of categories supplied at training or inference. AV3DOD argues that, despite the name, existing methods still rely on user-specified classes at both stages, and therefore introduces Auto-Vocabulary 3D Object Detection, in which labels are generated automatically without user input (Zhang et al., 18 Dec 2025). This redefinition shifts the task from prompt-conditioned detection toward autonomous semantic discovery.

2. Core architectural paradigms

The earliest point-cloud line uses 2D semantics as a supervisory bridge rather than as an inference-time requirement. OV-3DETIC is built on 3DETR and DETR, broadens vocabulary with ImageNet1K image-level labels, generates pseudo labels for unseen classes by combining 3D localization with 2D classification, and transfers semantics through Debiased Cross-modal Contrastive Learning (DECC). Its inference phase uses only the point-cloud detector, and it reports unseen-class $mAP_{25}$ of $13.03$ on SUN-RGBD and $12.65$ on ScanNet (Lu et al., 2022).

A second paradigm jointly discovers novel boxes and aligns 3D features with image and text spaces. CoDA introduces 3D Novel Object Discovery (3D-NOD) and Discovery-Driven Cross-Modal Alignment (DCMA), with discovered boxes iteratively enlarging the alignment pool (Cao et al., 2023). CoDAv2 extends this design with 3D-NODE, which enriches the novel-object distribution by inserting discovered objects into training scenes, and with Box-DCMA, where 2D box guidance suppresses background noise during alignment (Cao et al., 2024). In parallel, INHA combines Image-guided Novel class discovery (IGND) with hierarchical alignment at instance, category, and scene levels, using Detic and CLIP to improve both novel recall and open-vocabulary recognition (Jiao et al., 2024).

A third family emphasizes direct foundation-model transfer. FM-OV3D uses Grounded-SAM for open-vocabulary localization, GPT-3 for descriptive text prompts, Stable Diffusion for synthetic visual prompts, and CLIP for cross-modal alignment, blending multiple foundation models rather than relying on a single source of prior knowledge (Zhang et al., 2023). HCMA similarly argues that object-only representations lose essential context, and therefore constructs object-, view-, and scene-level data through Hierarchical Data Integration (HDI), aligns them through Interactive Cross-Modal Alignment (ICMA), and refines them with Object-Focusing Context Adjustment (OFCA) (Zhao et al., 10 Mar 2025). OV-SCAN, in the autonomous-driving regime, focuses on semantically consistent alignment by generating precise pseudo 3D annotations with SC-NOD and filtering low-quality or corrupted 3D/2D pairs before Hierarchical Two-Stage Alignment (H2SA) (Chow et al., 9 Mar 2025).

A fourth paradigm pursues modality and scene unification as first-class objectives. OV-Uni3DETR proposes cycle-modality propagation: 2D semantic knowledge guides novel-class discovery in 3D, while 3D geometric knowledge supervises 2D images lacking 3D labels. Its design includes switched-modality training, regularized voxel-space fusion, and a dual-branch output head to accommodate pseudo-label noise (Wang et al., 2024). This formulation directly targets three limitations identified in prior work: restricted vocabulary, modality lock-in, and scene restriction.

3. Supervision reduction and data-centric expansion

One response to 3D data scarcity is to generate new 3D training evidence rather than only align to external 2D models. Object2Scene augments 3D scene datasets by inserting objects from ShapeNet, OmniObject3D, and Objaverse into existing scenes, generating explicit box annotations and referring expressions for the inserted objects. Its associated detector, L3Det, unifies 3D detection and visual grounding, and introduces a cross-domain category-level contrastive loss,

$$\mathcal{L}_{cl}= -\frac{1}{N}\sum_{i=1}^N\log\frac{\sum_{i\neq j, y_i=y_j} \exp(f_i^\top f_j/\tau)}{\sum_{i\neq k} \exp(f_i^\top f_k/\tau)},$$

to mitigate the domain gap between original scene objects and inserted objects from external 3D datasets (Zhu et al., 2023).

A more aggressive strategy is to eliminate 3D supervision entirely during pretraining. ImOV3D explores learning open-vocabulary 3D detection from only 2D images by constructing a pseudo multimodal representation: 2D images are lifted into pseudo point clouds using monocular depth, rendered back into images, and aligned through a detector trained on both geometric and semantic surrogates (Yang et al., 2024). Its pretraining phase reports $T = [c_1, c_2, ..., c_M]$0 $T = [c_1, c_2, ..., c_M]$1 on SUNRGBD and $T = [c_1, c_2, ..., c_M]$2 on ScanNet without any real 3D training data, and its adaptation phase reaches $T = [c_1, c_2, ..., c_M]$3 and $T = [c_1, c_2, ..., c_M]$4 with a small amount of real 3D data (Yang et al., 2024).

RGB-only monocular training introduces a related but distinct regime. OVM3D-Det uses Grounded-SAM plus pseudo-LiDAR to auto-label 3D objects from RGB images, then calibrates those labels with adaptive pseudo-LiDAR erosion and bounding-box refinement guided by GPT-4 size priors (Huang et al., 2024). Its reported novel-category AP is $T = [c_1, c_2, ..., c_M]$5 on KITTI, $T = [c_1, c_2, ..., c_M]$6 on nuScenes, $T = [c_1, c_2, ..., c_M]$7 on SUN RGB-D, and $T = [c_1, c_2, ..., c_M]$8 on ARKitScenes (Huang et al., 2024). In a supervised-but-open-vocabulary monocular setting, OVMono3D-LIFT decouples open-vocabulary 2D recognition from class-agnostic 2D-to-3D lifting and reports novel-category $T = [c_1, c_2, ..., c_M]$9 of $\{(c_i, B_i)\}_{i=1}^N$0 on Omni3D, outperforming both a geometric baseline and 3D heads attached to Grounding DINO (Yao et al., 2024).

OpenM3D pursues annotation-free multi-view indoor detection from images alone. It adapts 2D-induced voxel features from ImGeoNet, learns class-agnostic localization from graph-embedding-based pseudo 3D boxes, and aligns voxel features with diverse CLIP features sampled from associated 2D segments (Hsu et al., 27 Aug 2025). On ScanNet200, it reports class-agnostic $\{(c_i, B_i)\}_{i=1}^N$1 of $\{(c_i, B_i)\}_{i=1}^N$2, open-vocabulary $\{(c_i, B_i)\}_{i=1}^N$3 of $\{(c_i, B_i)\}_{i=1}^N$4, $\{(c_i, B_i)\}_{i=1}^N$5 of $\{(c_i, B_i)\}_{i=1}^N$6, and inference time of $\{(c_i, B_i)\}_{i=1}^N$7 seconds per scene (Hsu et al., 27 Aug 2025). This line suggests that high-quality pseudo boxes and semantic alignment targets can substitute for human 3D annotations when the training loop remains tightly coupled to geometry.

4. Multi-view, online, and robotics-oriented systems

OpenNav exemplifies a robotics-centric RGB-D pipeline designed for assistive navigation. Its six stages are open-vocabulary 2D detection, mask generation, morphological erosion, depth isolation with Z-score filtering, point-cloud reconstruction using camera intrinsics and extrinsics, and axis-aligned 3D box generation (Rahman et al., 2024). On Replica, OpenNav with Grounding DINO and RepViT-SAM reports $\{(c_i, B_i)\}_{i=1}^N$8 mAP, $\{(c_i, B_i)\}_{i=1}^N$9 $B_i$0, and $B_i$1 $B_i$2, improving over Open3DIS from $B_i$3 to $B_i$4 at $B_i$5 and from $B_i$6 to $B_i$7 at $B_i$8, while reducing per-scene inference from $B_i$9 seconds to $\mathcal{C}_{\text{base}}$0–$\mathcal{C}_{\text{base}}$1 seconds (Rahman et al., 2024). The same pipeline is integrated into a smart powered wheelchair using APF and MPC for semi-autonomous motion (Rahman et al., 2024).

BoxFusion addresses a different bottleneck: the computational overhead of dense reconstruction in online RGB-D detection. It uses Cubify Anything for single-view 3D box proposals, CLIP for open-vocabulary semantics, 3D NMS plus 2D correspondence matching for association, and an IoU-guided efficient random optimization technique based on particle filtering for multi-view fusion (Lan et al., 18 Jun 2025). On CA-1M it reports class-agnostic AP of $\mathcal{C}_{\text{base}}$2 at IoU $\mathcal{C}_{\text{base}}$3, $\mathcal{C}_{\text{base}}$4 at $\mathcal{C}_{\text{base}}$5, and $\mathcal{C}_{\text{base}}$6 at $\mathcal{C}_{\text{base}}$7; on ScanNetV2 it reports $\mathcal{C}_{\text{base}}$8, $\mathcal{C}_{\text{base}}$9, and $\mathcal{C}_{\text{novel}}$0 at the same thresholds, while running at $\mathcal{C}_{\text{novel}}$1 FPS with $\mathcal{C}_{\text{novel}}$2 GB GPU memory (Lan et al., 18 Jun 2025). The paper explicitly argues that sparse box-based scene representations are sufficient for many embodied tasks.

Sparse-view RGB has become a separate regime rather than a degradation of dense multi-view input. SMOV3D is training-free and relies on off-the-shelf 2D foundation models for detection, segmentation, monocular depth, and dense CLIP features, then lifts 2D masks to 3D and optimizes proposal scale and shift through photometric and semantic consistency across views (Moliner et al., 19 Sep 2025). On ScanNet10, with $\mathcal{C}_{\text{novel}}$3 random sparse RGB views, it reports $\mathcal{C}_{\text{novel}}$4 $\mathcal{C}_{\text{novel}}$5; on Replica, with only $\mathcal{C}_{\text{novel}}$6 views, it reports $\mathcal{C}_{\text{novel}}$7 versus $\mathcal{C}_{\text{novel}}$8 for OpenIns3D (Moliner et al., 19 Sep 2025). Group3D and Zoo3D then push multi-view RGB further toward semantic and training-free operation. Group3D introduces scene-adaptive vocabularies and semantic compatibility groups to constrain fragment merging, reporting $\mathcal{C}_{\text{novel}}$9 $mAP_{25}$0 on pose-free ScanNet20, $mAP_{25}$1 on ScanNet200, and $mAP_{25}$2 on ARKitScenes (Kim et al., 23 Mar 2026). Zoo3D is described as the first training-free 3D object detection framework; its zero-shot Zoo3D$mAP_{25}$3 and self-supervised Zoo3D$mAP_{25}$4 use graph clustering of 2D masks, best-view selection, and view-consensus mask generation, and they operate on point clouds, posed images, and unposed images (Lemeshko et al., 25 Nov 2025).

5. Evaluation protocols and empirical patterns

The field uses several benchmark families and metric conventions. Indoor work frequently reports mAP, $mAP_{25}$5, and $mAP_{25}$6, where OpenNav states

$mAP_{25}$7

and evaluates on Replica with $mAP_{25}$8 classes across $mAP_{25}$9 scenes and $13.03$0 viewpoints per scene (Rahman et al., 2024). Point-cloud and open-vocabulary methods commonly use $13.03$1, $13.03$2, and $13.03$3 on SUN RGB-D and ScanNet (Zhang et al., 2023). Urban autonomous-driving work also reports NDS, as in OV-SCAN on nuScenes (Chow et al., 9 Mar 2025), while online class-agnostic systems such as BoxFusion report AP at IoU thresholds $13.03$4, $13.03$5, and $13.03$6 together with FPS and GPU memory (Lan et al., 18 Jun 2025).

A notable evaluation controversy concerns incomplete annotation and naming ambiguity. Open-vocabulary monocular 3D detection argues that traditional mAP assumes exhaustive labels for all categories in each image, which is violated in open-vocabulary settings. Its target-aware evaluation protocol therefore prompts the detector only with categories that are actually labeled in a given image, reducing penalization from missing annotations and synonym mismatches (Yao et al., 2024). This is not merely bookkeeping: it changes the interpretation of “false positives” in zero-shot detection.

AV3DOD adds a distinct semantic metric because localization accuracy alone is insufficient when labels are auto-generated. For matched boxes above IoU $13.03$7, it computes cosine similarity between CLIP text embeddings of predicted labels and ground-truth classes, forms AUC curves over thresholds $13.03$8, multiplies them by coverage, and defines

$13.03$9

$12.65$0

On ScanNetV2, AV3DOD reports $12.65$1 mAP and $12.65$2 SS, surpassing CoDA at $12.65$3 mAP and $12.65$4 SS; on SUNRGB-D it reports $12.65$5 mAP and $12.65$6 SS (Zhang et al., 18 Dec 2025).

Across benchmarked systems, several empirical milestones recur. OV-3DETIC established a strong early baseline for unseen classes with $12.65$7 $12.65$8 on SUN-RGBD and $12.65$9 on ScanNet (Lu et al., 2022). FM-OV3D then reported $$\mathcal{L}_{cl}= -\frac{1}{N}\sum_{i=1}^N\log\frac{\sum_{i\neq j, y_i=y_j} \exp(f_i^\top f_j/\tau)}{\sum_{i\neq k} \exp(f_i^\top f_k/\tau)},$$0 $$\mathcal{L}_{cl}= -\frac{1}{N}\sum_{i=1}^N\log\frac{\sum_{i\neq j, y_i=y_j} \exp(f_i^\top f_j/\tau)}{\sum_{i\neq k} \exp(f_i^\top f_k/\tau)},$$1 on SUN RGB-D and $$\mathcal{L}_{cl}= -\frac{1}{N}\sum_{i=1}^N\log\frac{\sum_{i\neq j, y_i=y_j} \exp(f_i^\top f_j/\tau)}{\sum_{i\neq k} \exp(f_i^\top f_k/\tau)},$$2 on ScanNet, using multi-foundation-model knowledge blending (Zhang et al., 2023). CoDA reported novel AP of $$\mathcal{L}_{cl}= -\frac{1}{N}\sum_{i=1}^N\log\frac{\sum_{i\neq j, y_i=y_j} \exp(f_i^\top f_j/\tau)}{\sum_{i\neq k} \exp(f_i^\top f_k/\tau)},$$3 versus $$\mathcal{L}_{cl}= -\frac{1}{N}\sum_{i=1}^N\log\frac{\sum_{i\neq j, y_i=y_j} \exp(f_i^\top f_j/\tau)}{\sum_{i\neq k} \exp(f_i^\top f_k/\tau)},$$4 for the best-performing alternative on SUN-RGBD (Cao et al., 2023), and CoDAv2 later reported $$\mathcal{L}_{cl}= -\frac{1}{N}\sum_{i=1}^N\log\frac{\sum_{i\neq j, y_i=y_j} \exp(f_i^\top f_j/\tau)}{\sum_{i\neq k} \exp(f_i^\top f_k/\tau)},$$5 of $$\mathcal{L}_{cl}= -\frac{1}{N}\sum_{i=1}^N\log\frac{\sum_{i\neq j, y_i=y_j} \exp(f_i^\top f_j/\tau)}{\sum_{i\neq k} \exp(f_i^\top f_k/\tau)},$$6 on SUN-RGBD and $$\mathcal{L}_{cl}= -\frac{1}{N}\sum_{i=1}^N\log\frac{\sum_{i\neq j, y_i=y_j} \exp(f_i^\top f_j/\tau)}{\sum_{i\neq k} \exp(f_i^\top f_k/\tau)},$$7 on ScanNetv2, substantially higher than $$\mathcal{L}_{cl}= -\frac{1}{N}\sum_{i=1}^N\log\frac{\sum_{i\neq j, y_i=y_j} \exp(f_i^\top f_j/\tau)}{\sum_{i\neq k} \exp(f_i^\top f_k/\tau)},$$8-CLIP at $$\mathcal{L}_{cl}= -\frac{1}{N}\sum_{i=1}^N\log\frac{\sum_{i\neq j, y_i=y_j} \exp(f_i^\top f_j/\tau)}{\sum_{i\neq k} \exp(f_i^\top f_k/\tau)},$$9 and $T = [c_1, c_2, ..., c_M]$00 (Cao et al., 2024). OV-Uni3DETR reported open-vocabulary gains of $T = [c_1, c_2, ..., c_M]$01–$T = [c_1, c_2, ..., c_M]$02 AP for novel classes at IoU $T = [c_1, c_2, ..., c_M]$03 and stated that its image-only inference is on par with or surpasses previous point-cloud-based methods (Wang et al., 2024). HCMA reported $T = [c_1, c_2, ..., c_M]$04 of $T = [c_1, c_2, ..., c_M]$05 on ScanNet, $T = [c_1, c_2, ..., c_M]$06 on SUN RGB-D, and $T = [c_1, c_2, ..., c_M]$07 on ScanNet200 (Zhao et al., 10 Mar 2025). Collectively, these results indicate that the strongest gains have come from better pseudo-label quality, better cross-modal alignment, and stronger multi-view semantics rather than from a single dominant detector backbone.

6. Limitations, misconceptions, and emerging directions

Several limitations recur across otherwise different systems. OV-SCAN identifies semantically inconsistent 3D/2D alignment pairs—caused by noisy pseudo annotations, occlusion, and resolution effects—as a major failure mode, and therefore filters low-quality pairs before alignment (Chow et al., 9 Mar 2025). FM-OV3D notes residual modality gaps between 2D projections and true 3D boundaries, prompt and synthetic-image quality sensitivity, and noise from broad negative sample sets (Zhang et al., 2023). SMOV3D identifies monocular depth as the principal bottleneck in sparse RGB-only lifting, with additional sensitivity to occlusion and camera calibration (Moliner et al., 19 Sep 2025). Open-vocabulary monocular 3D detection similarly emphasizes depth estimation as the major bottleneck in class-agnostic lifting (Yao et al., 2024).

Multi-view association introduces another class of errors. Group3D argues that geometry-only fragment merging can produce irreversible over-merging of distinct objects or fragmentation of a single instance when evidence is incomplete across views (Kim et al., 23 Mar 2026). BoxFusion, from the opposite direction, argues that dense reconstruction itself can be the wrong abstraction for real-time open-vocabulary perception, because it imposes substantial compute and memory costs (Lan et al., 18 Jun 2025). These positions are not contradictory: they point to a shared design tension between richer geometric aggregation and controllable, object-centric scene representations.

A persistent misconception is that “open-vocabulary” already implies unrestricted semantic autonomy. AV3DOD explicitly disputes this and replaces prompt-conditioned vocabularies with automatic class-name generation, evaluated by Semantic Score as well as mAP (Zhang et al., 18 Dec 2025). At the same time, training-free systems such as Zoo3D and SMOV3D, reconstruction-free online systems such as BoxFusion, and scene-adaptive semantic systems such as Group3D suggest a broadening of the field beyond prompt-conditioned point-cloud detection (Lemeshko et al., 25 Nov 2025, Moliner et al., 19 Sep 2025, Lan et al., 18 Jun 2025, Kim et al., 23 Mar 2026). A plausible implication is that the next phase of open-vocabulary 3D detection will be organized less by a single sensor modality and more by three axes already visible in current work: supervision budget, semantic autonomy, and deployment latency.