OpenM3D: Image-Based Open-Vocabulary 3D Detection

• OpenM3D is an image-based framework that performs open-vocabulary indoor 3D detection without requiring manual 3D box annotations.
• It leverages 2D-induced voxel features, dense 3D convolutions, and a semantic alignment head to integrate geometric precision with CLIP-derived recognition.
• The system generates high-quality pseudo-boxes through graph-based clustering from multi-view RGB images, enabling fast single-stage detection.

Searching arXiv for the cited OpenM3D paper and closely related works to ground the article in current papers. OpenM3D is an image-based framework for open-vocabulary multi-view indoor 3D object detection that is trained without human 3D box annotations or class labels and performs inference from only multi-view RGB images and camera poses. It is presented as a single-stage detector that adapts the 2D-induced voxel features of ImGeoNet, combines class-agnostic 3D localization with voxel-semantic alignment to CLIP-derived features, and uses automatically generated 3D pseudo boxes rather than manual supervision. Within the literature on open-vocabulary 3D perception, its defining claim is that open-vocabulary 3D detection can be realized in a multi-view image-only inference setting, rather than being tied to point clouds, depth at test time, or a two-stage proposal-plus-classification pipeline (Hsu et al., 27 Aug 2025).

1. Problem formulation and scope

OpenM3D addresses indoor 3D object detection under a weakly supervised regime in which posed RGB-D images are available during training, but human annotations of 3D boxes or classes are not. At test time, the required inputs are reduced to multi-view RGB images and camera poses, with depth not needed at inference. The target output is a set of 3D bounding boxes together with open-vocabulary class assignment obtained by matching detector features to text embeddings of category names (Hsu et al., 27 Aug 2025).

This setting combines several technical demands. The detector must perform 3D localization without human 3D labels, support arbitrary text prompts rather than a fixed taxonomy, and recover 3D structure from images alone. The paper emphasizes that existing open-vocabulary 3D methods are predominantly point-cloud-based, while prior multi-view image-based 3D detectors such as ImVoxelNet, NeRF-Det, and ImGeoNet are largely fixed-vocabulary. OpenM3D is therefore positioned at the intersection of two previously separate lines: image-based multi-view 3D detection and open-vocabulary recognition (Hsu et al., 27 Aug 2025).

The framework is also explicitly distinguished from heavy two-stage pipelines. A strong comparison baseline, denoted S2D, uses OpenM3D’s class-agnostic detector to generate candidate boxes and then classifies them with CLIP. OpenM3D instead learns semantic alignment inside the detector, so that open-vocabulary classification is performed in the detector’s aligned embedding space during a single forward pass (Hsu et al., 27 Aug 2025).

2. Representation, architecture, and detector design

The detector begins with a sequence of RGB images $\{I_t\}$ together with intrinsics $K$ and extrinsics $P_t$. A ResNet-50 pretrained on ImageNet extracts 2D features, which are then lifted into a 3D voxel volume

$$V \in \mathbb{R}^{H_v \times W_v \times D_v \times C}.$$

The supplementary specifies a volume spanning $6.4 \times 6.4 \times 2.56$ meters with voxel size $0.16$ m in all dimensions, which implies an effective resolution of approximately $40 \times 40 \times 16$ voxels (Hsu et al., 27 Aug 2025).

OpenM3D inherits from ImGeoNet a geometry shaping mechanism. An augmented voxel representation $V'$, formed by concatenating the original feature volume with feature variance, is processed to predict a shaping volume

$S = geo(V'),$

and geometry-aware voxel features are then formed by

$V_{geo} = S \odot V.$

The stated interpretation is that $K$0 estimates the likelihood that a voxel lies on object surfaces, thereby improving geometric precision in the 3D representation (Hsu et al., 27 Aug 2025).

These geometry-aware features are passed through dense 3D convolution layers to obtain multiscale feature volumes,

$K$1

A single-stage anchor-free detector head operates on these multiscale features and predicts, per voxel cell, 3D box localization, centerness, and binary foreground probability. The localization output includes center, size, and yaw. No transformer module, sparse convolution scheme, or explicit feature pyramid network name is introduced in the paper; the architecture is described as a 2D CNN front end followed by dense 3D convolution and an anchor-free detection head (Hsu et al., 27 Aug 2025).

To support open-vocabulary semantics, OpenM3D adds a semantic alignment head. Because the detector’s voxel features and CLIP embeddings have different dimensionalities, an MLP layer atop the voxel feature projects detector features into the CLIP embedding space. Open-vocabulary classification is then carried out implicitly via similarity in this aligned space, rather than by a learned fixed-vocabulary classifier (Hsu et al., 27 Aug 2025).

3. Pseudo-box generation without human 3D labels

A central contribution of OpenM3D is its pipeline for generating class-agnostic pseudo 3D boxes from posed RGB-D images. The method proceeds through four stages: obtaining 2D segments, lifting them into partial 3D segments, merging those partial structures across views using graph embedding and clustering, and converting the merged structures into 3D pseudo boxes (Hsu et al., 27 Aug 2025).

For each RGB image, the method uses SAM as a class-agnostic 2D instance segmenter, producing a set of segments

$K$2

The supplement adds several preprocessing rules. To run SAM, the method chooses up to 300 frames per scene using a Laplacian sharpness heuristic. Background-like large structures such as floor, wall, ceiling are filtered using MaskDINO-based processing. Tiny segments are removed using a minimum bounding box size of 30 pixels and a minimum 3D point occupancy ratio of 0.02 in the segment bounding box (Hsu et al., 27 Aug 2025).

Each 2D segment is back-projected to a partial 3D segment using depth and camera calibration:

$K$3

The resulting point set is denoted

$K$4

The supplement further states that voxelization + KNN are used to fuse noisy 3D points to mesh vertices as a coordinate standardization step, reducing depth-induced inconsistency across views (Hsu et al., 27 Aug 2025).

The novelty of the pseudo-labeling stage lies in the global graph construction over all viewpoints. Each node is a partial 3D segment $K$5, and an edge is defined by 3D overlap:

$K$6

with

$K$7

For efficiency, only node pairs within the same voxel are considered. The graph is embedded using DeepWalk and clustered using K-means with $K$8 for all scenes. Nodes assigned to one cluster form a complete 3D structure,

$K$9

The supplement adds that a connected-component algorithm is used to split spatially distant point sets that accidentally share the same cluster, and that majority voting resolves points shared by multiple nodes (Hsu et al., 27 Aug 2025).

An optional Mesh Segmentation Refinement (MSR) stage uses the ground-truth mesh and an off-the-shelf graph cut mesh over-segmentation. Mesh segments are matched to image-derived 3D segments using the same overlap definition, and segments with the same updated index are merged. The paper states that MSR improves pseudo-box quality, with larger gains on ScanNet200 than on ARKitScenes because refinement depends on mesh quality (Hsu et al., 27 Aug 2025).

Finally, each complete 3D segment $P_t$0 is converted into an axis-aligned 3D box whose center is the mean 3D coordinate and whose dimensions come from the min/max coordinates along each axis. Boxes are filtered using a minimum point count of 300 for ScanNet200 and 500 for ARKitScenes, together with a maximum box volume of $P_t$1. These filtered boxes are used as pseudo labels for training the localization branch (Hsu et al., 27 Aug 2025).

4. Training objectives, CLIP alignment, and inference

OpenM3D is trained jointly with a class-agnostic 3D localization loss and a voxel-semantic alignment loss. The localization branch teaches objectness and geometry from pseudo boxes, while the alignment branch injects open-vocabulary semantics through CLIP-derived features. The paper’s framing is explicit that a highly accurate single-stage detector requires both losses to be learned toward high-quality targets (Hsu et al., 27 Aug 2025).

The localization branch predicts, per voxel, box localization, centerness, and foreground probability. The stated loss components are rotated 3D IoU loss for box localization, cross-entropy loss for centerness, and focal loss for foreground/background classification. The paper does not provide balancing coefficients or a more detailed positive-assignment rule in the summarized text (Hsu et al., 27 Aug 2025).

The semantic branch aligns voxel features to CLIP features derived from 2D segments. For each 2D segment $P_t$2, the CLIP image encoder produces an embedding $P_t$3. After back-projection and voxelization, the associated voxel index set is denoted $P_t$4, and the corresponding voxel features are $P_t$5. The alignment loss is given as

$P_t$6

This is a direct segment-to-voxel cosine-distance alignment, rather than a contrastive InfoNCE-style objective (Hsu et al., 27 Aug 2025).

Several practical details govern CLIP usage. The main experiments use ViT-L/14, although ViT-B/16 and ViT-B/32 are also evaluated and perform similarly. Because CLIP is trained on natural image crops, the method does not use only the tight mask crop; it also includes surrounding regions at 110% and 120% of the segment bounding box. The supplement further notes that the method follows OV-3DET’s vocabulary of 364 categories to improve CLIP segment embeddings by comparing a segment embedding to all category text embeddings and using the embedding of the closest category (Hsu et al., 27 Aug 2025).

Inference is materially simpler than training. At test time, OpenM3D requires only multi-view RGB images and camera poses. The forward pass extracts 2D features, lifts them into the voxel volume, applies geometry shaping, processes them with 3D convolutions, and predicts object boxes and objectness. Open-vocabulary classification is then obtained by comparing aligned detector features to precomputed CLIP text embeddings derived from category names wrapped in prompt templates. The paper reports that prompt choice is not a major factor: using CIFAR-100 prompts instead of ImageNet templates changes ScanNet200 performance only slightly, from 4.23 / 15.12 to 4.14 / 14.79 in mAP@25 / mAR@25 (Hsu et al., 27 Aug 2025).

5. Empirical performance and operating characteristics

The paper evaluates OpenM3D on ScanNet200, ARKitScenes, and ScanNetv2. For ScanNet200, the stated protocol uses 20 views per scene for image-based methods, with image resolution standardized to $P_t$7. ScanNet200 is emphasized because it contains 200 categories, with 189 classes present in validation and a head/common/tail split of 66 / 68 / 66. ARKitScenes is used mainly for recall evaluation because its category diversity is limited, and the paper notes that its point clouds and depth are lower quality than those of ScanNet (Hsu et al., 27 Aug 2025).

Pseudo-box quality is one of the strongest quantitative results. On ScanNet200, pseudo-box precision and recall at IoU thresholds 0.25 and 0.50 are reported as follows: OV-3DET achieves P@25 11.62, P@50 4.40, R@25 21.13, R@50 7.99; SAM3D reaches 14.48, 9.05, 57.70, 36.07; OpenM3D w/o MSR reaches 27.09, 11.98, 52.43, 23.18; and OpenM3D reaches 32.07, 18.14, 58.30, 32.99. The paper’s stated takeaway is that OpenM3D’s pseudo boxes are substantially more precise than both baselines, while recall is much better than OV-3DET and competitive with SAM3D (Hsu et al., 27 Aug 2025).

These pseudo boxes improve downstream localization. On ScanNet200 class-agnostic 3D detection, training with OV-3DET boxes yields AP@25 19.53 / AR@25 35.19, training with SAM3D boxes yields 23.77 / 47.82, training with OpenM3D w/o MSR yields 25.95 / 48.14, and training with OpenM3D pseudo boxes yields 26.92 / 51.19. The paper explicitly interprets this as evidence that pseudo-box quality is a primary determinant of detector quality (Hsu et al., 27 Aug 2025).

For open-vocabulary 3D detection on ScanNet200, the main results are modest in absolute terms but favorable relative to baselines. OpenM3D trained with OV-3DET boxes obtains mAP@25 3.13 / mAR@25 10.83; with SAM3D boxes it reaches 3.92 / 13.33; with OpenM3D w/o MSR boxes it reaches 4.04 / 13.77; and with OpenM3D pseudo boxes it reaches 4.23 / 15.12. The strong two-stage detector S2D reaches 4.17 / 10.05 when using OpenM3D candidate boxes, while S2D with depth-estimated boxes reaches 3.80 / 8.60. The paper highlights the fact that OpenM3D surpasses the two-stage baseline that uses its own detector boxes plus a CLIP-based classifier, especially in mAR, as evidence that voxel-semantic alignment is not reducible to better proposal quality alone (Hsu et al., 27 Aug 2025).

On ARKitScenes, the paper emphasizes recall because only 17 annotated classes are available, which makes precision harder to interpret when unlabeled objects count as false positives. The reported mAR@25 values are 42.77 for OpenM3D and 19.58 for S2D. On ScanNetv2, OpenM3D reaches AP@25 = 19.76 using only images at inference, compared with OV-3DET 18.02, CoDA 19.32, and ImOV3D 21.45 (Hsu et al., 27 Aug 2025).

A major practical claim concerns efficiency. OpenM3D runs in 0.3 seconds per scene on a V100 GPU. The supplementary comparison reports OV-3DET 5 s, S2D 2.1 s, S2D with depth estimation 81 s, and OpenM3D 0.3 s. The paper attributes this speed to the single-stage design, the absence of test-time depth estimation, the absence of test-time CLIP image encoding, and the elimination of a second-stage classifier (Hsu et al., 27 Aug 2025).

6. Interpretation, limitations, and neighboring uses of similar names

OpenM3D’s principal significance lies in showing that open-vocabulary 3D object detection can be implemented as a single-stage, image-based detector with image-only inference, while avoiding human 3D annotations during training. This suggests a different design point from open-vocabulary 3D systems built around point clouds, reconstruction-first pipelines, or post-hoc proposal classification. Within the paper’s own evidence, the two decisive ingredients are high-quality pseudo boxes and embedded voxel-semantic alignment (Hsu et al., 27 Aug 2025).

The limitations are equally clear. The paper explicitly notes a large gap between class-agnostic detection and open-vocabulary detection, indicating that CLIP-based semantics remain weak for many semantically similar household objects. Performance also depends strongly on the quality of 2D segmentation and pseudo-box generation. Although depth is not needed at inference, it is still required during training and preprocessing, and the optional Mesh Segmentation Refinement depends on mesh quality, with smaller gains on ARKitScenes than on ScanNet200. The absolute mAP@25 on ScanNet200 remains low despite the relative improvements, and the preprocessing pipeline still includes SAM segmentation, graph embedding, K-means clustering, optional mesh refinement, and CLIP feature extraction (Hsu et al., 27 Aug 2025).

The name “OpenM3D” can also be confused with adjacent but distinct efforts. OpenTrack3D addresses open-vocabulary 3D instance segmentation rather than 3D object detection; it is a training-free, mesh-free pipeline operating on RGB-D streams, uses a visual-spatial tracker, and replaces CLIP-based final recognition with an MLLM for language grounding (Zhou et al., 3 Dec 2025). OpenMaterial is not named OpenM3D at all; it is a dataset and benchmark for 3D reconstruction under complex materials, focusing on metals, glass, and other non-diffuse optical regimes rather than open-vocabulary detection (Dang et al., 2024). In that narrower terminological sense, OpenM3D refers specifically to the annotation-free, single-stage, multi-view indoor open-vocabulary 3D detector introduced in 2025, not to general open 3D reconstruction benchmarks or language-grounded 3D instance segmentation systems.