OP3Det: Prompt-Free 3D Objectness Detector

• OP3Det is a prompt-free, class-agnostic 3D objectness detector that focuses on localizing all objects without fixed taxonomies.
• It integrates 2D semantic priors from foundation models with 3D geometric cues using a cross-modal mixture-of-experts architecture for effective feature fusion.
• Evaluations across indoor, outdoor, and cross-dataset scenarios show significant gains in average recall, highlighting robust open-world detection performance.

OP3Det most directly denotes a class-agnostic, prompt-free framework for open-world 3D objectness learning that aims to detect all physically discrete objects in a 3D scene, including categories unseen during training, without relying on text prompts at inference (Liu et al., 20 Oct 2025). In this formulation, the central target is not category recognition but 3D objectness: the likelihood that a spatial region corresponds to a valid object irrespective of semantic label. OP3Det combines 2D semantic priors from foundation models with 3D geometric priors, then fuses point-cloud and image features through a cross-modal mixture-of-experts (MoE) architecture to produce class-agnostic 3D proposals and objectness scores (Liu et al., 20 Oct 2025).

1. Problem formulation and conceptual scope

OP3Det is positioned in the open-world 3D detection regime. Its target is to detect all objects in a 3D scene—both seen and novel—without assuming a fixed taxonomy or requiring hand-crafted text prompts at test time (Liu et al., 20 Oct 2025). This distinguishes it from closed-set 3D detection, in which objects outside the training taxonomy are treated as background, and from open-vocabulary 3D detection, which conditions detection on user-provided text prompts or expanded vocabularies and is therefore exposed to vocabulary incompleteness and semantic overlap (Liu et al., 20 Oct 2025).

The paper formalizes 3D objectness as a binary foreground decision over spatial regions. Let $\phi(\cdot)$ denote a learnable model mapping input features $F_{\text{input}}$ to an objectness confidence score. Then objectness is expressed as

$I \left[ \phi(F_{\text{input}}) > \tau \right],$

where $\tau$ is a confidence threshold and $I[\cdot]$ is the indicator function (Liu et al., 20 Oct 2025). Training inputs consist of point clouds $X_P$, RGB images $X_I$, and annotated 3D bounding boxes with class-agnostic labels $c_i \in \{0,1\}$, so the learning target is object discovery and localization rather than semantic classification (Liu et al., 20 Oct 2025).

This formulation reflects a specific position within the broader 3D detection landscape. The model emphasizes recall and localization of “any” object as a precursor to downstream semantic labeling when needed. A plausible implication is that OP3Det is best understood as an objectness-centric detector rather than a category-centric detector, even though semantic priors enter the pipeline during discovery and training (Liu et al., 20 Oct 2025).

2. Object discovery from 2D priors and 3D geometry

The first major stage in OP3Det is a 3D Object Discovery pipeline used for pre-training data expansion (Liu et al., 20 Oct 2025). This stage exploits 2D foundation models to generate class-agnostic candidate regions and then imposes 3D geometric consistency to convert them into pseudo 3D boxes.

The semantic-prior stack consists of Segment Anything Model (SAM), a class-agnostic 2D detector denoted LDET, and self-supervised attention from DINO or DINOv2 (Liu et al., 20 Oct 2025). SAM is applied to RGB frames with grid point prompts of size $64 \times 64$ to produce candidate masks; LDET refines these masks into whole-object boxes and objectness scores; DINO/DINOv2 attention acts as a saliency prior for per-point objectness estimation (Liu et al., 20 Oct 2025). The paper notes that SAM alone can produce fragmented masks or object-part artifacts, so 3D geometric priors are introduced to suppress fragments and retain whole-object structure (Liu et al., 20 Oct 2025).

The 2D-to-3D lifting stage uses camera geometry through $KR_t$. Image features are projected into voxel space, and for proposal generation the implementation note is to project 3D points to 2D via $F_{\text{input}}$0, identify points whose projections lie inside each refined 2D box, and then cluster them to obtain pseudo 3D boxes $F_{\text{input}}$1 (Liu et al., 20 Oct 2025). The discovery mapping is summarized as

$F_{\text{input}}$2

where $F_{\text{input}}$3 is SAM, $F_{\text{input}}$4 is the class-agnostic detector, and $F_{\text{input}}$5 is the selected point set (Liu et al., 20 Oct 2025).

A central mechanism is multi-scale point sampling guided by 3D proximity. OP3Det builds an object-prior map by combining SAM IoU and DINO attention, then iteratively selects source points with the highest prior while suppressing neighboring points whose nearest 3D points fall within a distance threshold $F_{\text{input}}$6 (Liu et al., 20 Oct 2025). Multiple scales are used, specifically $F_{\text{input}}$7, and selections are merged via NMS (Liu et al., 20 Oct 2025). Masks are then refined and scored by multiplying SAM’s IoU prediction and LDET’s objectness, with a filtering threshold of $F_{\text{input}}$8 for low-quality discoveries (Liu et al., 20 Oct 2025).

The paper’s ablations show why this geometric filtering matters. “SAM only” degrades performance because fragmentation noise outweighs the value of dense mask proposals, while “SAM + multi-scale PS” restores and improves results, indicating that 3D-aware sampling is not an auxiliary refinement but a core component of discovery quality (Liu et al., 20 Oct 2025).

3. Cross-modal mixture-of-experts architecture

After discovery, OP3Det trains a detector that jointly consumes point-cloud and RGB information through a cross-modal MoE. The point cloud backbone is a sparse voxel-based 3D ResNet producing

$F_{\text{input}}$9

while the RGB backbone is ResNet-50 + FPN producing

$I \left[ \phi(F_{\text{input}}) > \tau \right],$0

(Liu et al., 20 Oct 2025). Image features are projected into the 3D voxel space to obtain

$I \left[ \phi(F_{\text{input}}) > \tau \right],$1

and multi-view lifted features are aggregated before fusion (Liu et al., 20 Oct 2025). The combined multimodal representation is

$I \left[ \phi(F_{\text{input}}) > \tau \right],$2

Each modality is encoded by self-attention to produce modality-specific features, and a multi-modal router $I \left[ \phi(F_{\text{input}}) > \tau \right],$3 computes routing probabilities

$I \left[ \phi(F_{\text{input}}) > \tau \right],$4

(Liu et al., 20 Oct 2025). Three experts are then applied: a semantic image expert $I \left[ \phi(F_{\text{input}}) > \tau \right],$5, a geometric point-cloud expert $I \left[ \phi(F_{\text{input}}) > \tau \right],$6, and a fused expert $I \left[ \phi(F_{\text{input}}) > \tau \right],$7, each implemented as 3D convolution stacks with kernel sizes 1–3–1 (Liu et al., 20 Oct 2025). Their outputs are combined as

$I \left[ \phi(F_{\text{input}}) > \tau \right],$8

The resulting fused feature feeds a 3D detection transformer head, described as DETR/Uni3DETR-style, which predicts class-agnostic objectness and 3D boxes (Liu et al., 20 Oct 2025). No auxiliary MoE load-balancing or sparsity regularizers are reported; routing is learned end-to-end through downstream detection losses (Liu et al., 20 Oct 2025).

The empirical significance of this design is explicit in the fusion ablations. Point-cloud-only fusion yields substantially better results than image-only or naïve additive or concatenative fusion, but CM-MoE exceeds all of them, suggesting that dynamic routing is important for preserving modality-specific cues and reducing what the paper terms “semantic noise” interfering with geometric objectness (Liu et al., 20 Oct 2025).

4. Supervision, optimization, and implementation

OP3Det is trained with both annotated class-agnostic 3D boxes and pseudo boxes discovered by the 3D Object Discovery pipeline (Liu et al., 20 Oct 2025). Multi-view image features are projected into voxel space with each view’s $I \left[ \phi(F_{\text{input}}) > \tau \right],$9 and then aggregated, after which the detector operates over the resulting multi-modal representation (Liu et al., 20 Oct 2025).

The objective “primarily follows the loss function in [OV-Uni3DETR], with the classification loss formulated as a class-agnostic binary classification loss” (Liu et al., 20 Oct 2025). The paper does not list explicit coefficients, but presents a typical composite objective as

$\tau$0

where $\tau$1 is binary cross-entropy on objectness and $\tau$2 includes box regression terms consistent with OV-Uni3DETR (Liu et al., 20 Oct 2025).

The implementation uses MMDetection3D and AdamW (Liu et al., 20 Oct 2025). The image backbone is ResNet-50 + FPN and the voxel backbone is a sparse 3D ResNet (Liu et al., 20 Oct 2025). Discovery hyperparameters include the multi-scale thresholds $\tau$3, a point budget $\tau$4 the number of total candidate points, and a discovery score threshold of $\tau$5 (Liu et al., 20 Oct 2025). Data augmentation, learning-rate schedule, batch size, and epoch counts are described as following standard MMDetection3D practice and are not detailed further in the paper (Liu et al., 20 Oct 2025).

At inference, OP3Det directly processes point cloud–image pairs and performs prompt-free detection. No prompts or external discovery stages are required at test time; the prompt-free property refers specifically to inference, even though 2D foundation models are used during the discovery stage of training (Liu et al., 20 Oct 2025). This distinction is important because it prevents confusion between OP3Det and prompt-conditioned open-vocabulary systems.

5. Evaluation settings and quantitative results

OP3Det is evaluated on indoor, outdoor, and cross-dataset settings. For indoor benchmarks, SUN RGB-D and ScanNet V2 are used with base/novel splits following CoDA: the “top 10” classes are treated as base, and the remaining classes are novel, with novel labels removed during training in the class-agnostic setting (Liu et al., 20 Oct 2025). For outdoor evaluation, KITTI is used with training on “car” as the base class and evaluation on car, cyclist, and pedestrian (Liu et al., 20 Oct 2025).

The primary metric is Average Recall at IoU 0.25, denoted AR25, reported for novel, base, and all classes (Liu et al., 20 Oct 2025). AP is also reported, and for KITTI the main outdoor metric is AP3D at moderate difficulty with 40 recall positions (Liu et al., 20 Oct 2025).

Benchmark	Main OP3Det result	Reported comparison
SUN RGB-D	AR_novel 78.8, AR_all 89.7, AR_base 93.1, AP_all 65.4	+16.0 AR_novel over OV-Uni3DETR; +13.5 over FCAF3D
ScanNet	AR_novel 79.9, AR_all 83.2, AR_base 87.3, AP_all 28.6	+12.3 AR_novel over closed-world baselines
KITTI	AP3D 66.75, APBEV 71.37	higher than PV-RCNN, Uni3DETR, OV-Uni3DETR

On SUN RGB-D, OP3Det achieves AR_novel $\tau$6, AR_all $\tau$7, AR_base $\tau$8, and AP_all $\tau$9, corresponding to a reported gain of $I[\cdot]$0 AR_novel over OV-Uni3DETR and $I[\cdot]$1 over FCAF3D (Liu et al., 20 Oct 2025). On ScanNet, it reaches AR_novel $I[\cdot]$2, AR_all $I[\cdot]$3, AR_base $I[\cdot]$4, and AP_all $I[\cdot]$5, with a reported $I[\cdot]$6 AR_novel improvement over closed-world baselines (Liu et al., 20 Oct 2025). On KITTI, OP3Det attains AP3D $I[\cdot]$7 and APBEV $I[\cdot]$8, exceeding the compared values for PV-RCNN, Uni3DETR, and OV-Uni3DETR (Liu et al., 20 Oct 2025).

Cross-dataset generalization is a notable part of the evaluation. In the ScanNet$I[\cdot]$9SUN transfer setting, OP3Det reports AR25 $X_P$0, AR50 $X_P$1, AP25 $X_P$2, and AP50 $X_P$3; in SUN$X_P$4ScanNet, it reports AR25 $X_P$5, AR50 $X_P$6, AP25 $X_P$7, and AP50 $X_P$8 (Liu et al., 20 Oct 2025). The paper characterizes these results as showing strong domain transfer, with cross-dataset AR25 within approximately $X_P$9 of in-dataset results (Liu et al., 20 Oct 2025).

Class-specific detection numbers are also reported in class-agnostic evaluation form. On SUN RGB-D, OP3Det achieves AP_novel $X_I$0, AP_base $X_I$1, and AP_all $X_I$2, exceeding OV-Uni3DETR and GLRD; on ScanNet, it reaches AP_novel $X_I$3, AP_base $X_I$4, and AP_all $X_I$5, again ranking best among the compared open-vocabulary methods (Liu et al., 20 Oct 2025).

6. Ablations, limitations, and terminological context

The principal ablation sequence isolates the contribution of discovery and fusion. A baseline without SAM, point sampling, or MoE yields AR_novel $X_I$6, AR_all $X_I$7, and AR_base $X_I$8 (Liu et al., 20 Oct 2025). Adding SAM alone reduces performance to AR_novel $X_I$9, AR_all $c_i \in \{0,1\}$0, and AR_base $c_i \in \{0,1\}$1, directly evidencing fragmentation noise (Liu et al., 20 Oct 2025). Adding multi-scale point sampling raises results to AR_novel $c_i \in \{0,1\}$2, AR_all $c_i \in \{0,1\}$3, and AR_base $c_i \in \{0,1\}$4, while the full system with CM-MoE reaches AR_novel $c_i \in \{0,1\}$5, AR_all $c_i \in \{0,1\}$6, and AR_base $c_i \in \{0,1\}$7 (Liu et al., 20 Oct 2025).

Fusion ablations reinforce the same conclusion. Point-cloud-only input obtains AR_novel $c_i \in \{0,1\}$8, AR_all $c_i \in \{0,1\}$9, and AR_base $64 \times 64$0; image-only input drops to AR_novel $64 \times 64$1, AR_all $64 \times 64$2, and AR_base $64 \times 64$3; naïve fusion underperforms point-cloud-only; and CM-MoE yields the best results at AR_novel $64 \times 64$4, AR_all $64 \times 64$5, and AR_base $64 \times 64$6 (Liu et al., 20 Oct 2025). This suggests that not all multimodal fusion is beneficial: OP3Det’s gains depend on routed expert specialization rather than simple feature combination.

The reported limitations are primarily geometric and calibration-related. Precise 2D-to-3D projection depends on accurate intrinsics and extrinsics; miscalibration degrades both lifting and routing (Liu et al., 20 Oct 2025). Fragmented, low-texture, non-rigid, or overexposed objects remain difficult despite multi-scale sampling (Liu et al., 20 Oct 2025). Heavy occlusion can still force uni-modal dominance even though CM-MoE mitigates it, and very sparse point clouds reduce the reliability of geometric priors (Liu et al., 20 Oct 2025). The model also introduces additional parameters and FLOPs through multi-modal backbones, voxelization, and MoE routing, while the paper does not report runtime or memory (Liu et al., 20 Oct 2025).

In the broader literature, the string “OP3Det” is not used uniformly. Point-DETR3D addresses one-point-supervised weakly semi-supervised 3D detection and is described as conceptually aligned with an “OP3Det-style” paradigm, but the paper does not use the OP3Det terminology (Gao et al., 2024). Conversely, the OPD literature defines single-view 3D openable-part detection, and the provided literature notes that this task is equivalent to Openable-Part 3D Detection even though the term “OP3Det” is likewise not used in that paper (Jiang et al., 2022). In current arXiv usage, however, OP3Det most specifically refers to the class-agnostic, prompt-free open-world 3D detector introduced for generalized 3D objectness learning (Liu et al., 20 Oct 2025).