WildDet3D: Open-World Monocular 3D Detection

• The paper introduces a promptable monocular 3D detection framework that accepts text, box, and point prompts to recover accurate 3D object extents from a single RGB image.
• It implements a unified geometry-aware architecture with dual-vision encoders that leverage optional depth signals to enhance 3D localization and orientation estimation.
• WildDet3D-Data offers a large-scale, hybrid annotated 3D detection dataset with over one million images and 13,499 categories for evaluating open-world detection performance.

Searching arXiv for the WildDet3D paper and closely related detection benchmarks/methods to ground the article in current literature. Searching for "WildDet3D Scaling Promptable 3D Detection in the Wild". WildDet3D is an open-world monocular 3D detection framework for recovering object extent, location, and orientation from a single RGB image under arbitrary real-world conditions, while supporting multiple prompt modalities and optional geometric cues at inference time (Huang et al., 9 Apr 2026). The system is defined by two coupled contributions: a unified geometry-aware architecture that natively accepts text, point, and box prompts and can incorporate auxiliary depth signals, and WildDet3D-Data, a large-scale open 3D detection dataset built by lifting existing 2D annotations into 3D and retaining only human-verified or VLM-filtered candidates (Huang et al., 9 Apr 2026). In the reported evaluations, WildDet3D achieves $22.6/24.8$ AP$_{3D}$ on WildDet3D-Bench with text and box prompts, $34.2/36.4$ AP$_{3D}$ on Omni3D, $40.3/48.9$ ODS in zero-shot transfer to Argoverse 2 and ScanNet, and an average gain of $+20.7$ AP across settings when depth is provided at inference time (Huang et al., 9 Apr 2026).

1. Problem setting and design thesis

WildDet3D addresses the problem of open-world monocular 3D object detection in the “wild” setting: arbitrary real images, long-tailed object vocabularies, and flexible user interaction (Huang et al., 9 Apr 2026). The paper identifies two bottlenecks in prior monocular 3D detection. First, existing methods typically assume a fixed prompt interface, such as class names for open-vocabulary methods or oracle 2D boxes for promptable methods, and therefore do not function as an interactive 3D perception module that can accept text, clicks, or boxes interchangeably. Second, most models lack a principled mechanism for consuming optional geometric signals such as depth at test time, even though real systems may have sparse LiDAR, stereo, or ToF depth available (Huang et al., 9 Apr 2026).

The model’s central thesis is that RGB plus optional depth is the most practical compromise. RGB provides semantic richness for open-vocabulary recognition, but monocular 3D localization remains affected by scale ambiguity and occlusion; LiDAR provides geometry but is sparse and weak for full 6-DoF reasoning. WildDet3D is therefore designed to operate as a monocular detector when depth is absent and to improve gracefully when depth is available (Huang et al., 9 Apr 2026).

The formal input is an RGB image $\mathbf{I}$, optional intrinsics $\mathbf{K}$, optional depth $\mathbf{D}$, and a prompt $\mathcal{P}$. The output consists of 3D boxes

$_{3D}$0

where $_{3D}$1 is the metric center, $_{3D}$2 are physical dimensions, $_{3D}$3 is rotation, and $_{3D}$4 is confidence (Huang et al., 9 Apr 2026). This formulation places WildDet3D at the intersection of monocular 3D detection, promptable perception, and geometry-conditioned inference.

2. Geometry-aware architecture

The architecture comprises three major components: a dual-vision encoder, a promptable detector, and a deeply supervised 3D detection head (Huang et al., 9 Apr 2026). The encoder explicitly separates semantic image features from geometric depth features. The image branch uses a ViT-H with a SimpleFPN neck, initialized from SAM3, to produce dense multi-scale visual features. In parallel, an RGBD branch uses a DINOv2 ViT-L/14 backbone, initialized from LingBot-Depth, and accepts a 4-channel RGBD input. When depth is unavailable, the depth channel is zero-filled so that the branch still produces a learned geometric prior from RGB alone. This branch outputs depth latents

$_{3D}$5

The two branches are deliberately pretrained for different purposes—segmentation for the image encoder and depth completion for the RGBD encoder—so that strong recognition features are preserved while a geometric backend is added (Huang et al., 9 Apr 2026).

Depth is injected through a ControlNet-style residual fusion module: $_{3D}$6 where $_{3D}$7 is bilinearly upsampled to match the visual resolution, $_{3D}$8 is layer norm, and the $_{3D}$9 convolution is zero-initialized so that fusion begins as the identity map (Huang et al., 9 Apr 2026). This design preserves the pretrained visual distribution while allowing depth to enrich image features rather than overwrite them.

Training of the geometry branch is stochastic: $34.2/36.4$0 monocular with zero depth, $34.2/36.4$1 patch-masked depth, and $34.2/36.4$2 full depth. This teaches the system to degrade gracefully from full geometry to pure monocular inference (Huang et al., 9 Apr 2026). A common misconception is that WildDet3D is fundamentally a depth-dependent detector. The training schedule and zero-filled RGBD branch indicate the opposite: depth is optional, but the architecture is expressly constructed so that optional geometry can be integrated without redesign.

3. Prompt interfaces and 3D lifting

The promptable detector unifies text, point, box, and exemplar prompts within a shared conditioning interface (Huang et al., 9 Apr 2026). Text prompts are tokenized with a CLIP-style BPE tokenizer and encoded by a 24-layer causal text Transformer, then projected into a shared embedding space. Box and point prompts are processed by a geometry encoder that combines a linear projection of coordinates, ROI-aligned or grid-sampled image features, and sinusoidal position encoding; point prompts additionally receive a learned positive or negative label embedding. A small Transformer refines the prompt tokens through cross-attention to image features. Exemplar prompts reuse the box pipeline, are disambiguated by a special “visual” token, and are trained with multi-target matching so that all instances of the same category become positive targets (Huang et al., 9 Apr 2026).

All prompt embeddings are concatenated into a single prompt sequence and used as cross-attention memory in both encoder and decoder stages. An important engineering choice is per-prompt batching: instead of batching whole images, the batch is organized per unique prompt, allowing arbitrary numbers of objects or categories per image without padding or truncation (Huang et al., 9 Apr 2026). This makes prompt diversity a native property of the training loop rather than a special-case extension.

The 3D head is a transformer decoder with deep supervision, so each layer predicts its own 3D outputs and receives loss (Huang et al., 9 Apr 2026). Each layer sequentially incorporates camera geometry and depth cues via two cross-attention branches. Camera rays are encoded using 8th-order real spherical harmonics,

$34.2/36.4$3

and the resulting ray features are cross-attended into the query states. Depth latents $34.2/36.4$4 are then cross-attended into the same queries. The head therefore aggregates semantic image features, ray geometry derived from intrinsics, and depth latents when available (Huang et al., 9 Apr 2026).

The output box parameterization is a 12D encoding,

$34.2/36.4$5

where center offsets are normalized by $34.2/36.4$6, log-depth is $34.2/36.4$7 with $34.2/36.4$8, log-dimensions are scaled similarly, and rotation uses the continuous 6D representation converted by Gram–Schmidt (Huang et al., 9 Apr 2026). The paper also introduces “unambiguous rotation normalization”: width and length are swapped so that $34.2/36.4$9, then yaw is folded into $_{3D}$0. This removes a 4-way ambiguity in oriented-box parameterization and makes regression targets unique (Huang et al., 9 Apr 2026).

4. WildDet3D-Data

WildDet3D-Data is described as the largest open 3D detection dataset to date (Huang et al., 9 Apr 2026). It is built from existing 2D annotation sources—COCO, LVIS, Objects365, and V3Det—which supply broad and long-tailed vocabulary coverage. For each 2D annotation, the pipeline first estimates depth and camera parameters: $_{3D}$1 super-resolution is applied, MoGe-2 estimates metric depth, PerspectiveFields estimates roll and pitch, and WildCamera estimates intrinsics. The depth map is reprojected into a point cloud, after which five complementary systems generate candidate 3D boxes: 3D-MOOD, DetAny3D, SAM-3D, RANSAC-PCA, and LabelAny3D (Huang et al., 9 Apr 2026).

Each candidate is refined by translation optimization and rotation optimization. Translation is optimized with a coarse-to-fine search under

$_{3D}$2

where $_{3D}$3 combines inclusion and tightness terms and only center translation is optimized while dimensions and rotation remain fixed (Huang et al., 9 Apr 2026). Candidate annotations are then unified into a common 10D format and filtered for boundary contact, occlusion, projection-size consistency, depicted-object detection, composite-image detection, and category-specific size or geometry constraints estimated by GPT-4.1-mini. Small objects are revisited and can be upgraded if they satisfy stricter VLM criteria (Huang et al., 9 Apr 2026).

Final candidate selection proceeds through two channels. In a balanced subset, human annotators on Prolific inspect up to five candidates per object using the image overlay and three point-cloud views, choose the best candidate, and assign one of three labels: good_fit, acceptable, or unacceptable. Gold tasks are mixed in for quality control. For the remaining images, a Molmo2-based VLM scorer selects among candidates using six criteria: category correctness, scale, translation, shape, rotation, and vertical tilt. Candidates with total score above 10 are retained (Huang et al., 9 Apr 2026).

The resulting dataset contains $_{3D}$4 images, $_{3D}$5 valid 3D annotations, and $_{3D}$6 categories, which the paper describes as a $_{3D}$7 increase over Omni3D’s 98 categories (Huang et al., 9 Apr 2026). The scenes span 22 scene categories, with roughly $_{3D}$8 indoor, $_{3D}$9 urban, and $40.3/48.9$0 nature scenes. Approximately $40.3/48.9$1K images are human-annotated and about $40.3/48.9$2K are VLM-filtered. The validation and test splits are balanced for category rarity and scene diversity (Huang et al., 9 Apr 2026). A frequent misunderstanding is that the dataset is purely human-labeled; in fact, it is hybrid, with a smaller human-verified subset and a much larger VLM-filtered portion.

5. Optimization, benchmarks, and empirical findings

Training uses the multitask objective

$40.3/48.9$3

The 3D regression term is an $40.3/48.9$4 loss over encoded 3D parameters with component validity weights. The confidence branch predicts a 3D quality score $40.3/48.9$5 with an IoU-aware focal BCE loss whose soft target combines depth quality and 3D IoU: $40.3/48.9$6 where

$40.3/48.9$7

At inference, the final score is

$40.3/48.9$8

The auxiliary losses supervise the depth backend with metric depth $40.3/48.9$9, scale-invariant logarithmic depth, depth validity BCE, affine-invariant point-map losses, edge-aware losses, and camera-ray losses, while the 2D head uses IoU-aware classification, GIoU/$+20.7$0 box regression, category presence loss, and one-to-many matching (Huang et al., 9 Apr 2026).

WildDet3D-Bench is the paper’s open-world benchmark, built from the validation split and containing 700+ categories from COCO, LVIS, and Objects365 (Huang et al., 9 Apr 2026). It uses center-distance AP with frequency splits over rare, common, and frequent categories, with the matching rule

$+20.7$1

for thresholds $+20.7$2. Because not all objects have valid 3D labels, federated evaluation treats predictions overlapping ignored objects as neutral (Huang et al., 9 Apr 2026). On this benchmark, WildDet3D achieves $+20.7$3 AP$+20.7$4 with text prompts and $+20.7$5 AP$+20.7$6 with box prompts; with depth at test time, it reaches $+20.7$7 and $+20.7$8 AP, respectively (Huang et al., 9 Apr 2026).

On Omni3D, WildDet3D reaches $+20.7$9 AP$\mathbf{I}$0 with text prompts and $\mathbf{I}$1 AP$\mathbf{I}$2 with box prompts, and further improves to $\mathbf{I}$3 and $\mathbf{I}$4 AP when depth is added at inference time (Huang et al., 9 Apr 2026). The gains are reported as especially large on depth-rich indoor datasets such as SUNRGBD, Hypersim, and ARKitScenes. Notably, the main stage uses only 12 training epochs, whereas the paper states that competing systems often train for 80–120 epochs (Huang et al., 9 Apr 2026).

For zero-shot transfer, the model is trained on Omni3D and evaluated on Argoverse 2 and ScanNet using

$\mathbf{I}$5

WildDet3D attains $\mathbf{I}$6 ODS on Argoverse 2 and $\mathbf{I}$7 ODS on ScanNet, with raw AP values of $\mathbf{I}$8 and $\mathbf{I}$9, respectively (Huang et al., 9 Apr 2026). Adding depth slightly improves ScanNet from $\mathbf{K}$0 to $\mathbf{K}$1 ODS but has essentially no effect on Argoverse 2, which the paper interprets as evidence that monocular geometry learned from Omni3D already matches outdoor driving scale reasonably well (Huang et al., 9 Apr 2026). On Stereo4D, zero-shot evaluation with real stereo depth raises AP to $\mathbf{K}$2 (Huang et al., 9 Apr 2026).

Ablation studies identify joint 2D+3D prediction as the dominant design choice: removing the 2D head reduces AP from $\mathbf{K}$3 to $\mathbf{K}$4. Removing the 3D confidence head costs $\mathbf{K}$5 AP. One-to-many matching provides the largest training-side gain; geometry losses are also important; deep supervision and ignore-aware suppression contribute smaller but consistent improvements (Huang et al., 9 Apr 2026). These results support the interpretation that WildDet3D is not merely a stronger box regressor, but a coupled grounding-and-lifting system in which 2D localization supplies a critical prior for 3D reasoning.

6. Relation to earlier “in the wild” 3D learning

WildDet3D should be distinguished from earlier work on 3D learning from real videos in the wild, notably “Unsupervised Learning of 3D Object Categories from Videos in the Wild” (Henzler et al., 2021). That work addresses category-level 3D reconstruction and discovery rather than monocular 3D detection: given a few images of a new object instance, the objective is to reconstruct 3D shape, appearance, and novel views without manual 3D annotations (Henzler et al., 2021). Its training signal comes from videos of object instances in the wild, weak automatic preprocessing from SfM via COLMAP, and automatically obtained object masks from Mask R-CNN (Henzler et al., 2021).

The earlier reconstruction system models an implicit neural surface or radiance field conditioned on a latent code and introduces warp-conditioned ray embedding (WCR), which makes the source embedding depend on the queried 3D point by projecting that point into source views and sampling dense CNN/U-Net-style descriptors (Henzler et al., 2021). It also contributes the AMT Objects dataset, a collection of object-centric videos across seven COCO object categories—apple, sandwich, orange, donut, banana, carrot, and hydrant—with $\mathbf{K}$6–$\mathbf{K}$7 videos per class and an $\mathbf{K}$8 train/test split, plus evaluation on Freiburg Cars (Henzler et al., 2021).

The distinction is methodological and epistemic. The earlier work learns category priors for reconstruction from multi-view videos under noisy SfM alignment, whereas WildDet3D predicts metric 3D boxes from a single image with prompt-conditioned inference and optional depth (Henzler et al., 2021, Huang et al., 9 Apr 2026). This suggests that the shared “wild” designation refers to unconstrained real-world data rather than to a common task or shared architecture. In that sense, WildDet3D occupies a different point in the broader evolution of 3D perception: not unsupervised 3D category discovery, but scalable, promptable, open-world 3D detection.