---
title: Open-Set 3D Object Retrieval
url: https://www.emergentmind.com/topics/open-set-3d-object-retrieval-3dor
type: topic
---

# Open-Set 3D Object Retrieval

Open-set 3D object retrieval (3DOR) is an emerging task aiming to retrieve 3D objects of unseen categories beyond the training set. In current research, the term covers several related settings: retrieval between 3D query and target objects when all testing categories are completely unseen during any training; retrieval of a particular 3D object instance from a 3D scene given a free-form language query; image-to-CAD retrieval from an unlabeled 3D object database; and cross-modal retrieval among multi-view images, voxels, and point clouds [2507.21489] [2505.02325] [2509.24528] [2601.07333] [2407.15376]. Across these settings, “open-vocabulary / open-set” commonly means zero-shot category recognition, no training on target categories or 3D labels, or direct evaluation on categories disjoint from the training set [2509.24528] [2505.02325].

## 1. Problem formulation and task variants

One standard formulation treats 3DOR as retrieval between a **query set** \(\mathcal{Q}\) and a **target (gallery) set** \(\mathcal{T}\), where each element is a 3D object \(\mathcal{O}\), all 3D objects in \(\mathcal{Q}\) and \(\mathcal{T}\) belong to categories that are completely unseen during any training, and no object labels or category names are available during adaptation [2505.02325]. In this setting, the system must retrieve, for each query object, the most similar objects in the target set. This is the dominant formulation in benchmark suites such as OS-ESB-core, OS-NTU-core, OS-MN40-core, and OS-ABO-core [2505.02325] [2507.21489].

A second formulation is **3D object retrieval from language** in reconstructed scenes. CORE-3D defines the input as a 3D scene with object-level 3D clusters and embeddings and a **free-form language query** \(q\), and the output as a **particular 3D object instance** whose IoU with the ground-truth instance is measured [2509.24528]. OVIR-3D similarly formulates open-vocabulary 3D instance retrieval as an information retrieval problem over a 3D scan: given a text query \(Q\), return a ranked set of 3D object instance segments as binary 3D masks over the scene point cloud [2311.02873].

A third formulation is **image-to-3D** or **image-to-CAD** retrieval. OSCAR assumes an input RGB image \(I \in \mathbb{R}^{H \times W \times 3}\), a natural language prompt \(L\), and an unlabeled database of 3D CAD models \(\mathcal{S} = \{s_1, s_2, \dots, s_N\}\), and retrieves the CAD model \(s^* \in \mathcal{S}\) that best matches the object referred to in \(I\) and \(L\), even when the exact CAD instance may not be present [2601.07333]. SAMURAI addresses a related challenge setting in which only a masked RGB image and a natural language description are provided, and the system must rank the top 10 candidate 3D objects [2506.21056].

A fourth formulation is **open-set 3D cross-modal retrieval**. SRCR defines objects with multiple modalities \(\{o_i^r\}_{r=1}^M\), learns a shared embedding function \(f\), and evaluates cross-modal tasks such as Image2Point, Image2Voxel, Point2Image, Point2Voxel, Voxel2Image, and Voxel2Point, with training and testing label spaces explicitly disjoint [2407.15376]. This broadens 3DOR beyond single-modality shape matching to shared latent spaces over heterogeneous 3D representations.

## 2. Representation paradigms

A central paradigm is **multi-view representation**. TeDA follows HGM\(^2\)R’s rendering protocol: place virtual cameras evenly around each 3D object, render \(M=24\) grayscale views at resolution \(256 \times 256\), extract CLIP features for each view, and aggregate them into a single global 3D visual descriptor by mean-pooling [2505.02325]. DAC likewise uses only multi-view images for open-set 3DOR, while DEC starts from the observation that simply mean-pooling over view features from a frozen DINO backbone gives decent performance [2507.21489] [2604.19432]. In a much earlier open-ended recognition line, OrthographicNet builds a global rotation- and scale-invariant representation from three orthographic projections and aggregates per-view CNN features by element-wise pooling [1902.03057].

A second paradigm is **object-level 3D instance representation in scenes**. CORE-3D lifts 2D masks into 3D, merges them with symmetric IoV constraints, splits over-merged masks with DBSCAN in 3D, and obtains a final 3D object set
\[
\mathcal{M}^{3D} = \{ M_1^{3D}, \ldots, M_N^{3D} \},
\]
where each \(M_i^{3D}\) is a cluster of 3D points with a unified CLIP embedding \(\mathbf{e}(M_i^{3D})\) [2509.24528]. OVIR-3D stores 3D instances in a memory bank, updates them by multi-view fusion of 2D open-vocabulary proposals, and represents each instance by a collection of associated 2D region features together with an aggregated 3D feature [2311.02873]. OpenMask3D, ConceptGraphs, and HOV-SG, as evaluated by OpenLex3D, are all object-centric methods in this sense: they maintain instance-level 3D masks or nodes and attach language-aligned embeddings to them [2503.19764].

A third paradigm is **dense or object-centric neural field representation**. OpenObj builds an object stack in which each object \(\mathcal{O}_k\) has an object-level NeRF \(\mathcal{F}_{\theta^k}^{k}\), object-level semantic descriptors \(f^{\mathcal{O}_k}_{\text{clip}}\) and \(f^{\mathcal{O}_k}_{\text{cap}}\), and a dense feature field \(f\) encoding part-level CLIP semantics throughout the object’s 3D volume [2406.08009]. This supports both object-level retrieval and fine-grained part-level retrieval within the same 3D representation.

A fourth paradigm is **residual-center or structure-aware latent representation**. SRCR uses nested autoencoders to derive object semantic centers \(u_i\), residual-center embeddings \(\delta_i^r\), and structure-aware aligned embeddings \(z_i^r\) through a heterogeneous hypergraph and a memory bank [2407.15376]. The intent is to avoid directly mapping data to category centers, which the paper identifies as a source of center deviation under category distribution differences in open-set settings [2407.15376].

## 3. Retrieval pipelines and system families

Training-free scene-level retrieval systems typically couple 2D open-vocabulary perception with 3D geometric fusion. CORE-3D is explicitly designed as a **training-free, open-vocabulary / open-set 3D object retrieval and semantic mapping system**. Its retrieval pipeline consists of query structuring via a lightweight LLM, candidate mining using CLIP-object embeddings, best-view selection and VLM verification, orientation grounding for view-dependent queries, and final reasoning with an LLM over verified candidates, centroids or bounding boxes, and orientation information [2509.24528]. OVIR-3D is also training-free: it uses Detic on every RGB frame, projects 2D regions into 3D, periodically filters and merges 3D instances, and performs language-to-instance retrieval by ranking 3D instances with the maximum cosine similarity between the CLIP text embedding and a set of cluster centers derived from associated 2D region features [2311.02873].

A closely related family frames retrieval as **open-vocabulary 3D instance segmentation**. BoxOVIS combines point-based masks from a pretrained 3D segmenter with RGBD-based masks generated by lifting 2D open-vocabulary boxes from YOLO-World into 3D and assembling superpoints inside them [2512.19088]. Retrieval is operationalized as segment all candidates and assign each a class label from the user-specified prompt set. Open-YOLO 3D is the efficiency-oriented precursor: only Mask3D proposals plus YOLO-World-based box classification, with no SAM and no CLIP [2512.19088]. This line emphasizes 3D mask recall for rare or tail classes rather than CLIP-style instance ranking.

Object-database retrieval systems usually operate on multi-view renderings. TeDA is a **non-parametric, test-time adaptation framework** that projects 3D objects into multi-view images, extracts features using CLIP, fuses them with textual descriptions generated by InternVL, and iteratively updates only the **query 3D embeddings** \(\overline{\mathbf{Q}}\) by minimizing
\[
L(\mathbf{P}', \mathbf{P}(\overline{\mathbf{Q}})) = \sum_{i=1}^{N} D_{\text{KL}}(\mathbf{P}'_i \,\|\, \mathbf{P}_i),
\]
thereby aligning the query distribution to the target distribution at test time [2505.02325]. DAC instead adapts CLIP during training with descriptive class text and then combines view features with MLLM-generated object descriptions at inference through
\[
\mathbf{h} = \tanh\big(\mathbf{g} + \lambda \mathbf{f}_{\text{text}}\big),
\]
using only multi-view images [2507.21489]. DEC shifts the backbone from CLIP to DINO, introduces a Chunking and Adapting Module for dynamic multi-view integration, and regularizes adaptation with Virtual Feature Synthesis based on CLIP’s broad, pre-aligned vision-language space [2604.19432].

Image-to-CAD retrieval systems emphasize cross-domain matching. OSCAR is a training-free method that first renders \(K=8\) synthetic views per CAD model, captions them with LLaVA-v1.5-7B, filters candidate models by CLIP text–image similarity between the query ROI and stored captions, and then refines the match with DINOv2 image–image similarity over rendered views [2601.07333]. SAMURAI uses CLIP-based semantic matching, CLIP embeddings of binary silhouettes, and a weighted majority voting strategy over text-only, shape-only, and text-then-shape rankings to handle masked single-image queries with ambiguous language and noisy masks [2506.21056].

Object-centric neural-field systems can extend retrieval granularity beyond whole objects. OpenObj obtains object-level CLIP and caption features by clustering the features of all masks belonging to the same object and selecting the largest cluster as \(f^{\mathcal{O}_k}_{\text{clip}}\) and \(f^{\mathcal{O}_k}_{\text{cap}}\), while dense feature fields inside each object support queries such as object + part and robotic grasping at specific parts [2406.08009].

## 4. Benchmarks and evaluation protocols

Evaluation protocols differ substantially across task variants. Object-database open-set 3DOR benchmarks typically use unseen categories and ranking metrics such as **mAP**, **NDCG**, and **ANMRR**; scene-language retrieval uses IoU between predicted and ground-truth 3D instances or bounding boxes; image-to-CAD retrieval additionally uses benchmark-specific retrieval scores such as NN, FT, ST, F, and DCG [2505.02325] [2509.24528] [2601.07333] [2407.15376].

| Setting | Representative datasets | Reported metrics |
|---|---|---|
| Object-to-object open-set 3DOR | OS-ESB-core, OS-NTU-core, OS-MN40-core, OS-ABO-core, ZS-Objaverse-Core | mAP, NDCG, ANMRR |
| Scene-level text-to-3D retrieval | Sr3D+, Replica, ScanNet, ScanNet200, OpenLex3D | A@0.1, A@0.25, AP\(_{25}\), AP\(_{50}\), mAP |
| Image-to-CAD or masked-image retrieval | MI3DOR, YCB-V, HouseCat6D, YCB-V+GSO, ROOMELSA | NN, FT, ST, F, DCG, ANMRR, mAP@K, Recall@k, MRR |

For object-level open-set benchmarks, TeDA evaluates on four open-set 3DOR benchmarks from Feng et al. (HGM\(^2\)R): OS-ESB-core, OS-NTU-core, OS-MN40-core, and OS-ABO-core, all using unseen test categories and the metrics mAP, NDCG, and ANMRR [2505.02325]. DAC uses the same four benchmarks and additionally evaluates cross-dataset and single-image setups, plus a zero-shot depth-based retrieval benchmark on Objaverse-LVIS called ZS-Objaverse-Core [2507.21489].

For scene-level open-vocabulary retrieval, CORE-3D uses Sr3D+ with 661 sampled instructions and evaluates Grounding accuracy at IoU thresholds 0.1 and 0.25, reported as **A@0.1** and **A@0.25** [2509.24528]. OpenLex3D introduces a dedicated benchmark over 23 scenes from Replica, ScanNet++, and HM3D, with entirely new label annotations including synonyms, depictions, visually similar labels, and clutter labels, and defines an object retrieval task with synonym-only queries and “depiction + synonym” queries, evaluated by \(AP_{25}\), \(AP_{50}\), and mAP over IoU thresholds \([0.5:0.95:0.05]\) [2503.19764].

For image-to-CAD retrieval, OSCAR evaluates on MI3DOR, YCB-V, HouseCat6D, and YCB-V+GSO. MI3DOR uses NN, FT, ST, F, DCG, and ANMRR; YCB-V, HouseCat6D, and YCB-V+GSO use mAP@k [2601.07333]. SAMURAI uses the ROOMELSA challenge protocol, where each query consists of a masked RGB image plus a natural language description and the output is a ranked top-10 list, evaluated by Recall@1, Recall@5, Recall@10, and Mean Reciprocal Rank [2506.21056].

## 5. Empirical landscape, strengths, and recurring failure modes

Performance on open-set 3DOR is strongly shaped by representation choice and evaluation regime. TeDA reports, with OpenCLIP ViT-L/14, **OS-ESB-core: mAP 65.45 vs. HGM\(^2\)R’s 51.74**, **OS-NTU-core: mAP 67.93 vs. 44.88**, **OS-MN40-core: 73.98 vs. 64.20**, and **OS-ABO-core: 72.12 vs. 63.39**, while remaining training-free at the 3D level [2505.02325]. DAC reports that, with only multi-view images, it **significantly surpasses prior arts by an average of +10.01% mAP on four open-set 3DOR datasets** [2507.21489]. CORE-3D reports on Sr3D+ **A@0.1 = 41.8** and **A@0.25 = 35.6** overall, compared with **BBQ: A@0.1 = 34.2, A@0.25 = 22.7**, and attributes the gain to progressive granularity refinement, context-aware CLIP encoding, symmetric IoV-based 3D fusion, and LLM/VLM reasoning [2509.24528].

Benchmark-dependent results also expose major differences between object-centric and scene-centric formulations. On OpenLex3D, overall mAP remains low: on Replica the best mAP is **OpenMask3D + NMS at 11.47**, on ScanNet++ it is **OpenMask3D + NMS at 4.00**, and on HM3D it is **ConceptGraphs at 5.09** [2503.19764]. The benchmark explicitly shows that many queries have **No Match** at IoU \(\ge 0.25\), indicating that failures are often driven by missing or inaccurate instance segmentation rather than only by feature matching [2503.19764]. OpenObj, which adds caption features and dense part-level feature fields, reports substantially higher Recall@K than ConceptGraphs on four Replica scenes, including **Ontology: R@1 = 0.90, R@2 = 0.95, R@3 = 1.00**, **Relevance: R@1 = 0.75, R@2 = 0.90, R@3 = 1.00**, **Functionality: R@1 = 0.90, R@2 = 0.95, R@3 = 0.95**, and **Part: R@1 = 0.80, R@2 = 0.80, R@3 = 0.80** [2406.08009].

Image-to-CAD retrieval shows a different empirical profile. OSCAR reports on MI3DOR **NN = 0.894**, **FT = 0.708**, **ST = 0.850**, **F = 0.238**, **DCG = 0.844**, **ANMRR = 0.205**, outperforming all baselines, and further reports **90.48\% average precision** during object retrieval on the YCB-V object dataset [2601.07333]. SAMURAI, under the ROOMELSA single-image masked-object challenge, achieves **R@1 = 0.88**, **R@5 = 1.00**, **R@10 = 1.00**, and **MRR = 0.93** [2506.21056].

Failure modes recur across otherwise different pipelines. CORE-3D notes confusions between **very similar categories**, sensitivity to poor masks in extremely cluttered scenes or noisy depth, occlusions or incomplete scans that reduce IoU even if semantic retrieval is correct, and query phrasing or ambiguous relational descriptions [2509.24528]. TeDA notes computational cost of **2000 PGD iterations**, dependence on rendering quality and view configuration, and reliance on CLIP/InternVL biases, especially for industrial components [2505.02325]. OSCAR identifies dependence on caption quality, sensitivity to ROI extraction in cluttered scenes, limitations of its fixed 8-view onboarding scheme, and approximate retrieval for functionally different objects with similar appearance [2601.07333]. BoxOVIS improves tail-class retrieval, but RGBD-based masks built from superpoints can be noisy at higher IoU thresholds, and the current bottleneck is CPU-based Open3D computation of 3D oriented boxes [2512.19088]. DEC explicitly observes that further adaptation of DINO can cause severe overfitting on average view patterns of known classes, motivating dynamic local integration and unseen-class regularization [2604.19432].

## 6. Design patterns, adjacent directions, and open problems

Several design principles recur across the literature. Multi-view rendering remains the dominant bridge from 3D geometry to strong 2D encoders: TeDA, DAC, and DEC all operate on rendered views rather than learned 3D backbones, while OrthographicNet shows an earlier rotation- and scale-invariant variant based on orthographic projections [2505.02325] [2507.21489] [2604.19432] [1902.03057]. A second recurring principle is to avoid direct dependence on closed vocabularies: CORE-3D is training-free with frozen CLIP and no 3D backbone, OVIR-3D performs open-vocabulary 3D instance retrieval without using any 3D data for training, and BoxOVIS transfers open-vocabulary capability from YOLO-World into 3D without SAM or CLIP at inference [2509.24528] [2311.02873] [2512.19088].

The literature also distinguishes between **semantic front-ends** and **geometric front-ends**. OP3Det is not a retrieval method, but it introduces a class-agnostic Open-World Prompt-free 3D Detector that detects any objects within 3D scenes without relying on hand-crafted text prompts; a plausible implication is that such prompt-free, high-recall 3D proposals can serve as front-end units for downstream open-set 3DOR [2510.17686]. 3D-MOOD is likewise an open-set monocular 3D detector rather than a retriever, but its geometry-aware object queries and canonical image space suggest a descriptor source for open-vocabulary, geometry-aware retrieval if additional metric learning is introduced; this suggests a separation between object discovery and semantic retrieval [2507.23567].

Two unresolved tensions are especially visible. The first is **fine-grainedness versus broad semantic alignment**. CLIP-based systems benefit from semantically aligned latent spaces, descriptive prompts, and VLM/LLM reasoning, but multiple papers identify insufficient fine-grainedness or viewpoint dependence as limiting factors [2507.21489] [2509.24528]. DINO-based DEC is motivated exactly by this limitation, while OpenObj addresses it with part-level feature fields inside object-level NeRFs [2604.19432] [2406.08009]. The second is **training-free simplicity versus adaptation**. CORE-3D, OVIR-3D, OSCAR, and SAMURAI emphasize training-free deployment, whereas TeDA uses test-time optimization, DAC uses parameter-efficient CLIP adaptation with AB-LoRA, and DEC adds dynamic view integration plus virtual unseen features [2509.24528] [2311.02873] [2601.07333] [2506.21056] [2505.02325] [2507.21489] [2604.19432]. This suggests that open-set 3DOR remains split between pure orchestration of pretrained models and lightweight adaptation designed to preserve generalization.

OpenLex3D clarifies that existing open-vocabulary 3D methods are far from saturating linguistically rich, scene-level evaluation, especially in cluttered real-world environments [2503.19764]. Future work suggested in the papers includes more efficient optimization schemes for test-time distribution alignment, better prompts and 3D-aware large multimodal models, GPU-based 3D uplift and selective mask refinement, richer part-level or region-level alignment, explicit metric learning on prompt-free 3D proposals, and broader benchmarks that move beyond nouns toward affordances, materials, and other semantic attributes [2505.02325] [2512.19088] [2406.08009] [2510.17686] [2503.19764]. A plausible implication is that progress in open-set 3DOR will continue to come from combining high-quality object proposals, geometry-aware multi-view representations, and language-grounded reasoning rather than from any single backbone family alone.

Source: https://www.emergentmind.com/topics/open-set-3d-object-retrieval-3dor