---
title: 'OpenM3D: Image-Based Open-Vocabulary 3D Detection'
url: https://www.emergentmind.com/topics/openm3d
type: topic
---

# OpenM3D: Image-Based Open-Vocabulary 3D Detection

Searching arXiv for the cited OpenM3D paper and closely related works to ground the article in current papers.
OpenM3D is an image-based framework for **open-vocabulary multi-view indoor 3D object detection** that is trained **without human 3D box annotations or class labels** and performs inference from **only multi-view RGB images and camera poses**. It is presented as a **single-stage detector** that adapts the **2D-induced voxel features** of ImGeoNet, combines **class-agnostic 3D localization** with **voxel-semantic alignment** to CLIP-derived features, and uses automatically generated **3D pseudo boxes** rather than manual supervision. Within the literature on open-vocabulary 3D perception, its defining claim is that open-vocabulary 3D detection can be realized in a **multi-view image-only inference setting**, rather than being tied to point clouds, depth at test time, or a two-stage proposal-plus-classification pipeline [2508.20063].

## 1. Problem formulation and scope

OpenM3D addresses indoor 3D object detection under a weakly supervised regime in which **posed RGB-D images** are available during training, but **human annotations of 3D boxes or classes are not**. At test time, the required inputs are reduced to **multi-view RGB images** and **camera poses**, with **depth not needed at inference**. The target output is a set of **3D bounding boxes** together with **open-vocabulary class assignment** obtained by matching detector features to text embeddings of category names [2508.20063].

This setting combines several technical demands. The detector must perform **3D localization without human 3D labels**, support **arbitrary text prompts rather than a fixed taxonomy**, and recover 3D structure from images alone. The paper emphasizes that existing open-vocabulary 3D methods are predominantly **point-cloud-based**, while prior multi-view image-based 3D detectors such as **ImVoxelNet**, **NeRF-Det**, and **ImGeoNet** are largely **fixed-vocabulary**. OpenM3D is therefore positioned at the intersection of two previously separate lines: image-based multi-view 3D detection and open-vocabulary recognition [2508.20063].

The framework is also explicitly distinguished from heavy two-stage pipelines. A strong comparison baseline, denoted **S2D**, uses OpenM3D’s class-agnostic detector to generate candidate boxes and then classifies them with CLIP. OpenM3D instead learns semantic alignment **inside the detector**, so that open-vocabulary classification is performed in the detector’s aligned embedding space during a single forward pass [2508.20063].

## 2. Representation, architecture, and detector design

The detector begins with a sequence of RGB images \(\{I_t\}\) together with intrinsics \(K\) and extrinsics \(P_t\). A **ResNet-50 pretrained on ImageNet** extracts 2D features, which are then lifted into a 3D voxel volume

\[
V \in \mathbb{R}^{H_v \times W_v \times D_v \times C}.
\]

The supplementary specifies a volume spanning **\(6.4 \times 6.4 \times 2.56\) meters** with **voxel size \(0.16\) m** in all dimensions, which implies an effective resolution of approximately **\(40 \times 40 \times 16\)** voxels [2508.20063].

OpenM3D inherits from ImGeoNet a **geometry shaping** mechanism. An augmented voxel representation \(V'\), formed by concatenating the original feature volume with **feature variance**, is processed to predict a shaping volume

\[
S = geo(V'),
\]

and geometry-aware voxel features are then formed by

\[
V_{geo} = S \odot V.
\]

The stated interpretation is that \(S\) estimates the likelihood that a voxel lies on object surfaces, thereby improving geometric precision in the 3D representation [2508.20063].

These geometry-aware features are passed through **dense 3D convolution layers** to obtain multiscale feature volumes,

\[
\{V_h^{(i)} = \text{Conv3D}^{(i)}(V_{geo}) \mid i \in \{0,1,\dots,L-1\}\}.
\]

A **single-stage anchor-free detector head** operates on these multiscale features and predicts, per voxel cell, **3D box localization**, **centerness**, and **binary foreground probability**. The localization output includes **center**, **size**, and **yaw**. No transformer module, sparse convolution scheme, or explicit feature pyramid network name is introduced in the paper; the architecture is described as a 2D CNN front end followed by dense 3D convolution and an anchor-free detection head [2508.20063].

To support open-vocabulary semantics, OpenM3D adds a **semantic alignment head**. Because the detector’s voxel features and CLIP embeddings have different dimensionalities, an **MLP layer atop the voxel feature** projects detector features into the CLIP embedding space. Open-vocabulary classification is then carried out implicitly via similarity in this aligned space, rather than by a learned fixed-vocabulary classifier [2508.20063].

## 3. Pseudo-box generation without human 3D labels

A central contribution of OpenM3D is its pipeline for generating **class-agnostic pseudo 3D boxes** from posed RGB-D images. The method proceeds through four stages: obtaining **2D segments**, lifting them into **partial 3D segments**, merging those partial structures across views using **graph embedding and clustering**, and converting the merged structures into **3D pseudo boxes** [2508.20063].

For each RGB image, the method uses **SAM** as a class-agnostic 2D instance segmenter, producing a set of segments

\[
S^{2D} = \{n_j^{2D}\}.
\]

The supplement adds several preprocessing rules. To run SAM, the method chooses up to **300 frames per scene** using a **Laplacian sharpness heuristic**. Background-like large structures such as **floor, wall, ceiling** are filtered using **MaskDINO-based processing**. Tiny segments are removed using a **minimum bounding box size of 30 pixels** and a **minimum 3D point occupancy ratio of 0.02 in the segment bounding box** [2508.20063].

Each 2D segment is back-projected to a **partial 3D segment** using depth and camera calibration:

\[
\begin{bmatrix} x \\ y \\ z \end{bmatrix}
=
R^T K^{-1} D(u,v)
\begin{bmatrix} u \\ v \\ 1 \end{bmatrix}
-
R^T t.
\]

The resulting point set is denoted

\[
n_j^{3D} = \{(x,y,z)_q\}.
\]

The supplement further states that **voxelization + KNN** are used to fuse noisy 3D points to mesh vertices as a coordinate standardization step, reducing depth-induced inconsistency across views [2508.20063].

The novelty of the pseudo-labeling stage lies in the **global graph construction** over all viewpoints. Each node is a partial 3D segment \(n_j^{3D}\), and an edge is defined by 3D overlap:

\[
e_{jk} = \mathrm{edge}(n_j^{3D}, n_k^{3D}) =
\begin{cases}
1, & \text{if } O(n_j^{3D}, n_k^{3D}) > \theta, \\
0, & \text{otherwise.}
\end{cases}
\]

with

\[
O(n_j^{3D}, n_k^{3D}) =
\frac{|n_j^{3D} \cap n_k^{3D}|}
{\min(|n_j^{3D}|, |n_k^{3D}|)}.
\]

For efficiency, only node pairs within the **same voxel** are considered. The graph is embedded using **DeepWalk** and clustered using **K-means** with **\(K=100\)** for all scenes. Nodes assigned to one cluster form a complete 3D structure,

\[
\hat{n}_q^{3D} = \{n^{3D} \in \mathcal{C}_q\}.
\]

The supplement adds that a **connected-component algorithm** is used to split spatially distant point sets that accidentally share the same cluster, and that **majority voting** resolves points shared by multiple nodes [2508.20063].

An optional **Mesh Segmentation Refinement (MSR)** stage uses the **ground-truth mesh** and an off-the-shelf **graph cut** mesh over-segmentation. Mesh segments are matched to image-derived 3D segments using the same overlap definition, and segments with the same updated index are merged. The paper states that MSR improves pseudo-box quality, with larger gains on **ScanNet200** than on **ARKitScenes** because refinement depends on mesh quality [2508.20063].

Finally, each complete 3D segment \(\hat{n}_q^{3D}\) is converted into an **axis-aligned 3D box** whose center is the **mean 3D coordinate** and whose dimensions come from the **min/max coordinates** along each axis. Boxes are filtered using a **minimum point count** of **300 for ScanNet200** and **500 for ARKitScenes**, together with a **maximum box volume of \(8.5\,\mathrm{m}^3\)**. These filtered boxes are used as pseudo labels for training the localization branch [2508.20063].

## 4. Training objectives, CLIP alignment, and inference

OpenM3D is trained jointly with a **class-agnostic 3D localization loss** and a **voxel-semantic alignment loss**. The localization branch teaches objectness and geometry from pseudo boxes, while the alignment branch injects open-vocabulary semantics through CLIP-derived features. The paper’s framing is explicit that a highly accurate single-stage detector requires **both losses to be learned toward high-quality targets** [2508.20063].

The localization branch predicts, per voxel, **box localization**, **centerness**, and **foreground probability**. The stated loss components are **rotated 3D IoU loss** for box localization, **cross-entropy loss** for centerness, and **focal loss** for foreground/background classification. The paper does not provide balancing coefficients or a more detailed positive-assignment rule in the summarized text [2508.20063].

The semantic branch aligns voxel features to CLIP features derived from 2D segments. For each 2D segment \(n_j^{2D}\), the CLIP image encoder produces an embedding \(f_j^{2D}\). After back-projection and voxelization, the associated voxel index set is denoted \(V_j^{3D}\), and the corresponding voxel features are \(f_k^{3D}\). The alignment loss is given as

\[
\mathcal{L}_{align}\left(f_j^{2D}, \{f_k^{3D}\}\right)
=
\sum_{k \in V_j^{3D}}
\left(
1 -
\frac{f_j^{2D} \cdot f_k^{3D}}
{\|f_j^{2D}\| \, \|f_k^{3D}\|}
\right).
\]

This is a direct **segment-to-voxel cosine-distance alignment**, rather than a contrastive InfoNCE-style objective [2508.20063].

Several practical details govern CLIP usage. The main experiments use **ViT-L/14**, although **ViT-B/16** and **ViT-B/32** are also evaluated and perform similarly. Because CLIP is trained on natural image crops, the method does not use only the tight mask crop; it also includes surrounding regions at **110%** and **120%** of the segment bounding box. The supplement further notes that the method follows OV-3DET’s vocabulary of **364 categories** to improve CLIP segment embeddings by comparing a segment embedding to all category text embeddings and using the embedding of the **closest category** [2508.20063].

Inference is materially simpler than training. At test time, OpenM3D requires **only multi-view RGB images and camera poses**. The forward pass extracts 2D features, lifts them into the voxel volume, applies geometry shaping, processes them with 3D convolutions, and predicts object boxes and objectness. Open-vocabulary classification is then obtained by comparing aligned detector features to **precomputed CLIP text embeddings** derived from category names wrapped in prompt templates. The paper reports that prompt choice is not a major factor: using **CIFAR-100 prompts** instead of **ImageNet** templates changes ScanNet200 performance only slightly, from **4.23 / 15.12** to **4.14 / 14.79** in **mAP@25 / mAR@25** [2508.20063].

## 5. Empirical performance and operating characteristics

The paper evaluates OpenM3D on **ScanNet200**, **ARKitScenes**, and **ScanNetv2**. For ScanNet200, the stated protocol uses **20 views per scene** for image-based methods, with image resolution standardized to **\(480 \times 640\)**. ScanNet200 is emphasized because it contains **200 categories**, with **189 classes** present in validation and a head/common/tail split of **66 / 68 / 66**. ARKitScenes is used mainly for recall evaluation because its category diversity is limited, and the paper notes that its point clouds and depth are lower quality than those of ScanNet [2508.20063].

Pseudo-box quality is one of the strongest quantitative results. On **ScanNet200**, pseudo-box precision and recall at IoU thresholds **0.25** and **0.50** are reported as follows: **OV-3DET** achieves **P@25 11.62, P@50 4.40, R@25 21.13, R@50 7.99**; **SAM3D** reaches **14.48, 9.05, 57.70, 36.07**; **OpenM3D w/o MSR** reaches **27.09, 11.98, 52.43, 23.18**; and **OpenM3D** reaches **32.07, 18.14, 58.30, 32.99**. The paper’s stated takeaway is that OpenM3D’s pseudo boxes are substantially more precise than both baselines, while recall is much better than OV-3DET and competitive with SAM3D [2508.20063].

These pseudo boxes improve downstream localization. On **ScanNet200** class-agnostic 3D detection, training with **OV-3DET** boxes yields **AP@25 19.53 / AR@25 35.19**, training with **SAM3D** boxes yields **23.77 / 47.82**, training with **OpenM3D w/o MSR** yields **25.95 / 48.14**, and training with **OpenM3D** pseudo boxes yields **26.92 / 51.19**. The paper explicitly interprets this as evidence that pseudo-box quality is a primary determinant of detector quality [2508.20063].

For **open-vocabulary 3D detection on ScanNet200**, the main results are modest in absolute terms but favorable relative to baselines. OpenM3D trained with **OV-3DET** boxes obtains **mAP@25 3.13 / mAR@25 10.83**; with **SAM3D** boxes it reaches **3.92 / 13.33**; with **OpenM3D w/o MSR** boxes it reaches **4.04 / 13.77**; and with **OpenM3D** pseudo boxes it reaches **4.23 / 15.12**. The strong two-stage detector **S2D** reaches **4.17 / 10.05** when using OpenM3D candidate boxes, while **S2D with depth-estimated boxes** reaches **3.80 / 8.60**. The paper highlights the fact that OpenM3D surpasses the two-stage baseline that uses its own detector boxes plus a CLIP-based classifier, especially in **mAR**, as evidence that **voxel-semantic alignment** is not reducible to better proposal quality alone [2508.20063].

On **ARKitScenes**, the paper emphasizes recall because only **17 annotated classes** are available, which makes precision harder to interpret when unlabeled objects count as false positives. The reported **mAR@25** values are **42.77** for OpenM3D and **19.58** for S2D. On **ScanNetv2**, OpenM3D reaches **AP@25 = 19.76** using only images at inference, compared with **OV-3DET 18.02**, **CoDA 19.32**, and **ImOV3D 21.45** [2508.20063].

A major practical claim concerns efficiency. OpenM3D runs in **0.3 seconds per scene** on a **V100 GPU**. The supplementary comparison reports **OV-3DET 5 s**, **S2D 2.1 s**, **S2D with depth estimation 81 s**, and **OpenM3D 0.3 s**. The paper attributes this speed to the **single-stage design**, the absence of **test-time depth estimation**, the absence of **test-time CLIP image encoding**, and the elimination of a second-stage classifier [2508.20063].

## 6. Interpretation, limitations, and neighboring uses of similar names

OpenM3D’s principal significance lies in showing that **open-vocabulary 3D object detection** can be implemented as a **single-stage, image-based detector** with **image-only inference**, while avoiding **human 3D annotations** during training. This suggests a different design point from open-vocabulary 3D systems built around point clouds, reconstruction-first pipelines, or post-hoc proposal classification. Within the paper’s own evidence, the two decisive ingredients are **high-quality pseudo boxes** and **embedded voxel-semantic alignment** [2508.20063].

The limitations are equally clear. The paper explicitly notes a large gap between **class-agnostic detection** and **open-vocabulary detection**, indicating that CLIP-based semantics remain weak for many **semantically similar household objects**. Performance also depends strongly on the quality of **2D segmentation** and **pseudo-box generation**. Although depth is not needed at inference, it is still required during **training and preprocessing**, and the optional **Mesh Segmentation Refinement** depends on mesh quality, with smaller gains on **ARKitScenes** than on **ScanNet200**. The absolute **mAP@25** on ScanNet200 remains low despite the relative improvements, and the preprocessing pipeline still includes **SAM segmentation**, **graph embedding**, **K-means clustering**, optional **mesh refinement**, and **CLIP feature extraction** [2508.20063].

The name “OpenM3D” can also be confused with adjacent but distinct efforts. **OpenTrack3D** addresses **open-vocabulary 3D instance segmentation** rather than 3D object detection; it is a **training-free**, **mesh-free** pipeline operating on **RGB-D streams**, uses a **visual-spatial tracker**, and replaces CLIP-based final recognition with an **MLLM** for language grounding [2512.03532]. **OpenMaterial** is not named OpenM3D at all; it is a **dataset and benchmark for 3D reconstruction under complex materials**, focusing on metals, glass, and other non-diffuse optical regimes rather than open-vocabulary detection [2406.08894]. In that narrower terminological sense, OpenM3D refers specifically to the annotation-free, single-stage, multi-view indoor **open-vocabulary 3D detector** introduced in 2025, not to general open 3D reconstruction benchmarks or language-grounded 3D instance segmentation systems.

Source: https://www.emergentmind.com/topics/openm3d