Open-YOLO 3D: Efficient Open-Vocabulary 3D Detection

• Open-YOLO 3D is a family of open-vocabulary 3D detection and segmentation frameworks that integrate 2D detectors with class-agnostic 3D proposals for rapid inference.
• It leverages YOLO-style single-stage regression, multi-view prompt association, and box-guided mask uplift to achieve state-of-the-art trade-offs in throughput and accuracy.
• The approach overcomes challenges in rare and long-tailed category detection by fusing 2D open-vocabulary cues with 3D proposals, avoiding reliance on heavy foundation models.

Open-YOLO 3D comprises a family of architectures and algorithms for efficient 3D object detection and instance segmentation with an open-vocabulary capability, focusing on rapid inference and strong generalization to novel classes. Building on both single-stage YOLO paradigms for 3D bounding box regression and modern open-vocabulary vision-language techniques, Open-YOLO 3D variants target real-time robotics, AR/VR, and scene retrieval tasks where conventional closed-set pipelines falter due to speed or lack of vocabulary extensibility. Approaches converge on a strategy that leverages 2D open-vocabulary detectors, multi-view RGB(D) imagery, and class-agnostic 3D proposals, eschewing heavy foundation models such as SAM and CLIP at inference. Key formulations achieve state-of-the-art trade-offs between throughput and accuracy in large-scale 3D scene benchmarks such as ScanNet200 and Replica, and recent work integrates box-guided techniques to bolster recall for rare or long-tailed classes (Boudjoghra et al., 4 Jun 2024, Nguyen et al., 22 Dec 2025).

1. Motivation and Problem Formulation

Open-vocabulary 3D instance segmentation necessitates segmenting and labeling objects in 3D point clouds according to arbitrary user-supplied text prompts, including previously unseen object categories. Conventional methods rely on 2D foundation models (SAM for mask proposal/refinement, CLIP for text-vision embedding) with computational bottlenecks caused by per-view mask extraction and multi-view feature aggregation. The core motivations for Open-YOLO 3D include:

• Reducing inference latency from several minutes per scene to practical timescales (tens of seconds or less).
• Minimizing redundant computation arising from detailed per-view 2D segmentation, noting that projections of 3D proposals often already encode necessary instance information.
• Achieving prompt–mask association via fast 2D detection, backed by empirical evidence that open-vocabulary 2D detectors such as YOLO-World provide adequate class signal for multi-view association.
• Generalizing beyond YOLO-closed set detection to accommodate novel categories or unknown object discovery in 3D (Cen et al., 2021).
• Overcoming limitations in recognizing rare or long-tail 3D object categories, sometimes missed by standard 3D segmenters, by incorporating 2D-driven box guidance (Nguyen et al., 22 Dec 2025).

2. Core Architectures and Algorithmic Principles

Open-YOLO 3D systems share several architectural pillars:

(a) Class-Agnostic 3D Proposals

A pretrained 3D instance segmentation network (e.g., Mask3D) processes a voxelized or point-based cloud $P \in \mathbb{R}^{4 \times N}$, producing $K_{3D}$ binary masks $M \in \{0,1\}^{K_{3D} \times N}$. Superpoint graph-based presegmentations, as well as hybrid point/RGBD proposals, further refine the mask candidate set in advanced variants (Nguyen et al., 22 Dec 2025).

(b) 2D Open-Vocabulary Detection and Label Map Construction

Each multi-view RGB frame $\mathcal{I}_i$ is processed by a real-time 2D detector (YOLO-World), which outputs bounding boxes and class predictions (both known and novel). Outputs are rasterized into low-granularity (LG) label maps $\mathcal{L}_i \in \mathbb{Z}^{H \times W}$, using a box “weight” to resolve pixel label assignment conflicts.

(c) Multi-View Prompt Association

Each 3D mask is projected into all camera frames using joint intrinsic/extrinsic transformations; per-point (and per-mask) visibilities are computed to filter out occluded or out-of-frame projections. The multi-view prompt distribution $\mathcal{D}_j$ for each 3D proposal aggregates label assignments from LG maps over the top-$k$ most visible viewpoints. The final class label for each 3D mask is assigned by majority voting over $\mathcal{D}_j$.

(d) Hybrid Box-Guided Mask Uplift

To address missed detections for rare categories, the box-guided extension transforms high-confidence 2D boxes to 3D by projecting their supporting pixels (via depth maps, camera intrinsics/extrinsics) into the world frame. Superpoint grouping within these lifted boxes produces additional 3D masks, merged across frames and filtered to avoid duplication with point-based proposals (Nguyen et al., 22 Dec 2025).

(e) Confidence Scoring

The per-proposal confidence $s_m$ is calculated as the product of class assignment frequency (maximum in $\mathcal{D}_j$) and the average IoU between projected 3D masks and matched 2D detector boxes.

3. Training Protocols and Loss Formulations

For the segmentation backbone (e.g., Mask3D), training is conducted in a closed-set regime using standard mask IoU and classification objectives. The 2D open-vocabulary detector (YOLO-World) is typically pretrained via CLIP-based contrastive losses for text-image alignment, enabling zero-shot class recall. The mask classifier and box-guided mask merger operate without further training; their thresholds are set via validation.

Metric learning approaches for open-set 3D detection leverage a soft prototype embedding head, with probability of class $t$ for candidate embedding $z$ given by

$$p_t(z) = \frac{\exp(-\|z - m_t\|^2)}{\sum_{k=1}^C \exp(-\|z - m_k\|^2)}$$

with open-set identification via the sum-of-distance threshold $\mathrm{EDS}(z)$ to discriminate unknown from known classes (Cen et al., 2021).

The loss function for direct 3D box regression in the LiDAR-centric Open-YOLO 3D follows:

$$\begin{aligned} L = & \lambda_{\text{coord}} \sum_{i,j} L_{ij}^{obj}[(x_{ij} - \hat{x}_{ij})^2 + \cdots ] \ & + \lambda_\phi \sum_{i,j} L_{ij}^{obj} (\phi_{ij} - \hat{\phi}_{ij})^2 \ & + \lambda_{\text{conf}} \sum_{i,j} [ L_{ij}^{obj} (C_{ij} - \hat{C}_{ij})^2 + L_{ij}^{noobj} (C_{ij} - \hat{C}_{ij})^2 ] \ & + \lambda_{\text{cls}} \sum_{i,j} L_{ij}^{obj} \sum_{c} (p_{ij}(c) - \hat{p}_{ij}(c))^2 \end{aligned}$$

(Ali et al., 2018).

4. Quantitative Evaluation and Runtime Benchmarks

Benchmarking is reported primarily on ScanNet200 and Replica datasets, with standard metrics such as mAP, AP$_{50}$, AP$_{25}$, and splits among head/common/tail categories. Key quantitative findings:

Method	mAP	AP$_{50}$	AP$_{25}$	Tail mAP	Time / Scene (s)	Extra Foundation Models
Open3DIS	23.7	29.4	32.8	21.8	360.1	SAM + CLIP
Open-YOLO 3D	24.7	31.7	36.2	21.6	21.8	None (YOLO-World only)
Box-Guided OY3D	24.9	32.1	36.8	22.4	55.9	None

On Replica, Open-YOLO 3D achieves mAP = 23.7% at 16.6 s/scene. The box-guided extension increases tail class recall (by +0.8 mAP$_{tail}$) while maintaining inference under one minute per scene (Nguyen et al., 22 Dec 2025).

Ablation studies highlight that replacing CLIP embeddings entirely with label map voting substantially boosts mAP, and that accelerated visibility computation leads to 20× faster inference with no loss in accuracy.

5. Methodological Variants and Open-Set Detection

Open-set detection in Open-YOLO 3D is achieved by modifying the head of the YOLO-style network to predict an embedding vector. Metric-learning classification using fixed class prototypes enables distance-based open-set reasoning. Proposals with low aggregate similarity to all known prototypes are treated as unknown, with downstream unsupervised clustering (e.g., 3D DBSCAN) recovering tight bounding boxes for unknown objects (Cen et al., 2021). This methodology enables both known and out-of-distribution object detection in a single unified head with minimal architectural complexity.

LiDAR-based Open-YOLO 3D (Ali et al., 2018) directly regresses 3D object boxes from BEV maps at real-time speeds (≈40 fps) by refining the YOLOv2 grid/anchor regression paradigm to 3D.

The YOLOStereo3D variant (Liu et al., 2021) leverages a backbone with stereo matching for rapid 3D detection from binocular images, eschewing depth map reconstruction for direct one-stage prediction, retaining real-time capability and simplicity.

6. Limitations and Extensions

Identified limitations include:

• Class-agnostic 3D mask proposals from Mask3D suffer recall losses on thin or small objects, due to voxelization and resolution bottlenecks.
• Open-vocabulary capability hinges on the recall of the 2D open-vocabulary detector; rare-class recognition is improved but may remain challenging for objects unseen in both 2D and 3D data.
• The speed/accuracy frontier remains at risk as real-time segmentation and clustering (especially for high-resolution or crowded point clouds) is still a computational bottleneck for certain use cases.

Proposed extensions involve fusing fast 2D instance segmentation (e.g., FastSAM) for richer proposal generation, adaptively selecting the number of views for each proposal during prompt assignment, and investigating CLIP–label map hybrid voting to further boost recall for rare and ambiguous classes (Boudjoghra et al., 4 Jun 2024, Nguyen et al., 22 Dec 2025).

7. Impact, Comparative Analysis, and Future Directions

Open-YOLO 3D systems represent a critical advance for deployable open-vocabulary 3D understanding at practical speeds. By decoupling instance segmentation from heavyweight foundation inference and transferring zero-shot class generalization from high-throughput 2D detectors to the 3D domain, they unlock scene-level search, robotics manipulation, and AR/VR perception at previously unattainable throughput.

Key technical insights—such as the sufficiency of 2D class signals for 3D prompt assignment, and the efficiency of label map voting over large precomputed vision-language embeddings—inform a growing trend away from rigid closed-set or multi-stage 3D pipelines.

Ongoing research focuses on further reducing compute cost, improving rare-category recall, and generalizing to increasingly unconstrained environments where 3D instance segmentation must continuously adapt to novel text prompts and complex scene compositions (Boudjoghra et al., 4 Jun 2024, Nguyen et al., 22 Dec 2025).