Open-Set 3D Object Retrieval

Updated 4 July 2026

Open-set 3D object retrieval is a task that retrieves 3D assets from unseen categories using zero-shot learning and multi-modal embeddings.
It employs multi-view representations, cross-modal language inputs, and object instance segmentation to bridge 2D cues with 3D geometry.
Recent techniques combine training-free orchestration with lightweight adaptation, yielding significant gains in standard retrieval metrics.

Open-set 3D object retrieval (3DOR) is an emerging task aiming to retrieve 3D objects of unseen categories beyond the training set. In current research, the term covers several related settings: retrieval between 3D query and target objects when all testing categories are completely unseen during any training; retrieval of a particular 3D object instance from a 3D scene given a free-form language query; image-to-CAD retrieval from an unlabeled 3D object database; and cross-modal retrieval among multi-view images, voxels, and point clouds (Wang et al., 29 Jul 2025, Wang et al., 5 May 2025, Mirzaei et al., 29 Sep 2025, Pulli et al., 12 Jan 2026, Xu et al., 2024). Across these settings, “open-vocabulary / open-set” commonly means zero-shot category recognition, no training on target categories or 3D labels, or direct evaluation on categories disjoint from the training set (Mirzaei et al., 29 Sep 2025, Wang et al., 5 May 2025).

1. Problem formulation and task variants

One standard formulation treats 3DOR as retrieval between a query set $\mathcal{Q}$ and a target (gallery) set $\mathcal{T}$ , where each element is a 3D object $\mathcal{O}$ , all 3D objects in $\mathcal{Q}$ and $\mathcal{T}$ belong to categories that are completely unseen during any training, and no object labels or category names are available during adaptation (Wang et al., 5 May 2025). In this setting, the system must retrieve, for each query object, the most similar objects in the target set. This is the dominant formulation in benchmark suites such as OS-ESB-core, OS-NTU-core, OS-MN40-core, and OS-ABO-core (Wang et al., 5 May 2025, Wang et al., 29 Jul 2025).

A second formulation is 3D object retrieval from language in reconstructed scenes. CORE-3D defines the input as a 3D scene with object-level 3D clusters and embeddings and a free-form language query $q$ , and the output as a particular 3D object instance whose IoU with the ground-truth instance is measured (Mirzaei et al., 29 Sep 2025). OVIR-3D similarly formulates open-vocabulary 3D instance retrieval as an information retrieval problem over a 3D scan: given a text query $Q$ , return a ranked set of 3D object instance segments as binary 3D masks over the scene point cloud (Lu et al., 2023).

A third formulation is image-to-3D or image-to-CAD retrieval. OSCAR assumes an input RGB image $I \in \mathbb{R}^{H \times W \times 3}$ , a natural language prompt $L$ , and an unlabeled database of 3D CAD models $\mathcal{S} = \{s_1, s_2, \dots, s_N\}$ , and retrieves the CAD model $\mathcal{T}$ 0 that best matches the object referred to in $\mathcal{T}$ 1 and $\mathcal{T}$ 2, even when the exact CAD instance may not be present (Pulli et al., 12 Jan 2026). SAMURAI addresses a related challenge setting in which only a masked RGB image and a natural language description are provided, and the system must rank the top 10 candidate 3D objects (Vo et al., 26 Jun 2025).

A fourth formulation is open-set 3D cross-modal retrieval. SRCR defines objects with multiple modalities $\mathcal{T}$ 3, learns a shared embedding function $\mathcal{T}$ 4, and evaluates cross-modal tasks such as Image2Point, Image2Voxel, Point2Image, Point2Voxel, Voxel2Image, and Voxel2Point, with training and testing label spaces explicitly disjoint (Xu et al., 2024). This broadens 3DOR beyond single-modality shape matching to shared latent spaces over heterogeneous 3D representations.

2. Representation paradigms

A central paradigm is multi-view representation. TeDA follows HGM $\mathcal{T}$ 5R’s rendering protocol: place virtual cameras evenly around each 3D object, render $\mathcal{T}$ 6 grayscale views at resolution $\mathcal{T}$ 7, extract CLIP features for each view, and aggregate them into a single global 3D visual descriptor by mean-pooling (Wang et al., 5 May 2025). DAC likewise uses only multi-view images for open-set 3DOR, while DEC starts from the observation that simply mean-pooling over view features from a frozen DINO backbone gives decent performance (Wang et al., 29 Jul 2025, He et al., 21 Apr 2026). In a much earlier open-ended recognition line, OrthographicNet builds a global rotation- and scale-invariant representation from three orthographic projections and aggregates per-view CNN features by element-wise pooling (Kasaei, 2019).

A second paradigm is object-level 3D instance representation in scenes. CORE-3D lifts 2D masks into 3D, merges them with symmetric IoV constraints, splits over-merged masks with DBSCAN in 3D, and obtains a final 3D object set

$\mathcal{T}$ 8

where each $\mathcal{T}$ 9 is a cluster of 3D points with a unified CLIP embedding $\mathcal{O}$ 0 (Mirzaei et al., 29 Sep 2025). OVIR-3D stores 3D instances in a memory bank, updates them by multi-view fusion of 2D open-vocabulary proposals, and represents each instance by a collection of associated 2D region features together with an aggregated 3D feature (Lu et al., 2023). OpenMask3D, ConceptGraphs, and HOV-SG, as evaluated by OpenLex3D, are all object-centric methods in this sense: they maintain instance-level 3D masks or nodes and attach language-aligned embeddings to them (Kassab et al., 25 Mar 2025).

A third paradigm is dense or object-centric neural field representation. OpenObj builds an object stack in which each object $\mathcal{O}$ 1 has an object-level NeRF $\mathcal{O}$ 2, object-level semantic descriptors $\mathcal{O}$ 3 and $\mathcal{O}$ 4, and a dense feature field $\mathcal{O}$ 5 encoding part-level CLIP semantics throughout the object’s 3D volume (Deng et al., 2024). This supports both object-level retrieval and fine-grained part-level retrieval within the same 3D representation.

A fourth paradigm is residual-center or structure-aware latent representation. SRCR uses nested autoencoders to derive object semantic centers $\mathcal{O}$ 6, residual-center embeddings $\mathcal{O}$ 7, and structure-aware aligned embeddings $\mathcal{O}$ 8 through a heterogeneous hypergraph and a memory bank (Xu et al., 2024). The intent is to avoid directly mapping data to category centers, which the paper identifies as a source of center deviation under category distribution differences in open-set settings (Xu et al., 2024).

3. Retrieval pipelines and system families

Training-free scene-level retrieval systems typically couple 2D open-vocabulary perception with 3D geometric fusion. CORE-3D is explicitly designed as a training-free, open-vocabulary / open-set 3D object retrieval and semantic mapping system. Its retrieval pipeline consists of query structuring via a lightweight LLM, candidate mining using CLIP-object embeddings, best-view selection and VLM verification, orientation grounding for view-dependent queries, and final reasoning with an LLM over verified candidates, centroids or bounding boxes, and orientation information (Mirzaei et al., 29 Sep 2025). OVIR-3D is also training-free: it uses Detic on every RGB frame, projects 2D regions into 3D, periodically filters and merges 3D instances, and performs language-to-instance retrieval by ranking 3D instances with the maximum cosine similarity between the CLIP text embedding and a set of cluster centers derived from associated 2D region features (Lu et al., 2023).

A closely related family frames retrieval as open-vocabulary 3D instance segmentation. BoxOVIS combines point-based masks from a pretrained 3D segmenter with RGBD-based masks generated by lifting 2D open-vocabulary boxes from YOLO-World into 3D and assembling superpoints inside them (Nguyen et al., 22 Dec 2025). Retrieval is operationalized as segment all candidates and assign each a class label from the user-specified prompt set. Open-YOLO 3D is the efficiency-oriented precursor: only Mask3D proposals plus YOLO-World-based box classification, with no SAM and no CLIP (Nguyen et al., 22 Dec 2025). This line emphasizes 3D mask recall for rare or tail classes rather than CLIP-style instance ranking.

Object-database retrieval systems usually operate on multi-view renderings. TeDA is a non-parametric, test-time adaptation framework that projects 3D objects into multi-view images, extracts features using CLIP, fuses them with textual descriptions generated by InternVL, and iteratively updates only the query 3D embeddings $\mathcal{O}$ 9 by minimizing

$\mathcal{Q}$ 0

thereby aligning the query distribution to the target distribution at test time (Wang et al., 5 May 2025). DAC instead adapts CLIP during training with descriptive class text and then combines view features with MLLM-generated object descriptions at inference through

$\mathcal{Q}$ 1

using only multi-view images (Wang et al., 29 Jul 2025). DEC shifts the backbone from CLIP to DINO, introduces a Chunking and Adapting Module for dynamic multi-view integration, and regularizes adaptation with Virtual Feature Synthesis based on CLIP’s broad, pre-aligned vision-language space (He et al., 21 Apr 2026).

Image-to-CAD retrieval systems emphasize cross-domain matching. OSCAR is a training-free method that first renders $\mathcal{Q}$ 2 synthetic views per CAD model, captions them with LLaVA-v1.5-7B, filters candidate models by CLIP text–image similarity between the query ROI and stored captions, and then refines the match with DINOv2 image–image similarity over rendered views (Pulli et al., 12 Jan 2026). SAMURAI uses CLIP-based semantic matching, CLIP embeddings of binary silhouettes, and a weighted majority voting strategy over text-only, shape-only, and text-then-shape rankings to handle masked single-image queries with ambiguous language and noisy masks (Vo et al., 26 Jun 2025).

Object-centric neural-field systems can extend retrieval granularity beyond whole objects. OpenObj obtains object-level CLIP and caption features by clustering the features of all masks belonging to the same object and selecting the largest cluster as $\mathcal{Q}$ 3 and $\mathcal{Q}$ 4, while dense feature fields inside each object support queries such as object + part and robotic grasping at specific parts (Deng et al., 2024).

4. Benchmarks and evaluation protocols

Evaluation protocols differ substantially across task variants. Object-database open-set 3DOR benchmarks typically use unseen categories and ranking metrics such as mAP, NDCG, and ANMRR; scene-language retrieval uses IoU between predicted and ground-truth 3D instances or bounding boxes; image-to-CAD retrieval additionally uses benchmark-specific retrieval scores such as NN, FT, ST, F, and DCG (Wang et al., 5 May 2025, Mirzaei et al., 29 Sep 2025, Pulli et al., 12 Jan 2026, Xu et al., 2024).

Setting	Representative datasets	Reported metrics
Object-to-object open-set 3DOR	OS-ESB-core, OS-NTU-core, OS-MN40-core, OS-ABO-core, ZS-Objaverse-Core	mAP, NDCG, ANMRR
Scene-level text-to-3D retrieval	Sr3D+, Replica, ScanNet, ScanNet200, OpenLex3D	[email protected], [email protected], AP $\mathcal{Q}$ 5, AP $\mathcal{Q}$ 6, mAP
Image-to-CAD or masked-image retrieval	MI3DOR, YCB-V, HouseCat6D, YCB-V+GSO, ROOMELSA	NN, FT, ST, F, DCG, ANMRR, mAP@K, Recall@k, MRR

For object-level open-set benchmarks, TeDA evaluates on four open-set 3DOR benchmarks from Feng et al. (HGM $\mathcal{Q}$ 7R): OS-ESB-core, OS-NTU-core, OS-MN40-core, and OS-ABO-core, all using unseen test categories and the metrics mAP, NDCG, and ANMRR (Wang et al., 5 May 2025). DAC uses the same four benchmarks and additionally evaluates cross-dataset and single-image setups, plus a zero-shot depth-based retrieval benchmark on Objaverse-LVIS called ZS-Objaverse-Core (Wang et al., 29 Jul 2025).

For scene-level open-vocabulary retrieval, CORE-3D uses Sr3D+ with 661 sampled instructions and evaluates Grounding accuracy at IoU thresholds 0.1 and 0.25, reported as [email protected] and [email protected] (Mirzaei et al., 29 Sep 2025). OpenLex3D introduces a dedicated benchmark over 23 scenes from Replica, ScanNet++, and HM3D, with entirely new label annotations including synonyms, depictions, visually similar labels, and clutter labels, and defines an object retrieval task with synonym-only queries and “depiction + synonym” queries, evaluated by $\mathcal{Q}$ 8, $\mathcal{Q}$ 9, and mAP over IoU thresholds $\mathcal{T}$ 0 (Kassab et al., 25 Mar 2025).

For image-to-CAD retrieval, OSCAR evaluates on MI3DOR, YCB-V, HouseCat6D, and YCB-V+GSO. MI3DOR uses NN, FT, ST, F, DCG, and ANMRR; YCB-V, HouseCat6D, and YCB-V+GSO use mAP@k (Pulli et al., 12 Jan 2026). SAMURAI uses the ROOMELSA challenge protocol, where each query consists of a masked RGB image plus a natural language description and the output is a ranked top-10 list, evaluated by Recall@1, Recall@5, Recall@10, and Mean Reciprocal Rank (Vo et al., 26 Jun 2025).

5. Empirical landscape, strengths, and recurring failure modes

Performance on open-set 3DOR is strongly shaped by representation choice and evaluation regime. TeDA reports, with OpenCLIP ViT-L/14, OS-ESB-core: mAP 65.45 vs. HGM $\mathcal{T}$ 1R’s 51.74, OS-NTU-core: mAP 67.93 vs. 44.88, OS-MN40-core: 73.98 vs. 64.20, and OS-ABO-core: 72.12 vs. 63.39, while remaining training-free at the 3D level (Wang et al., 5 May 2025). DAC reports that, with only multi-view images, it significantly surpasses prior arts by an average of +10.01% mAP on four open-set 3DOR datasets (Wang et al., 29 Jul 2025). CORE-3D reports on Sr3D+ [email protected] = 41.8 and [email protected] = 35.6 overall, compared with BBQ: [email protected] = 34.2, [email protected] = 22.7, and attributes the gain to progressive granularity refinement, context-aware CLIP encoding, symmetric IoV-based 3D fusion, and LLM/VLM reasoning (Mirzaei et al., 29 Sep 2025).

Benchmark-dependent results also expose major differences between object-centric and scene-centric formulations. On OpenLex3D, overall mAP remains low: on Replica the best mAP is OpenMask3D + NMS at 11.47, on ScanNet++ it is OpenMask3D + NMS at 4.00, and on HM3D it is ConceptGraphs at 5.09 (Kassab et al., 25 Mar 2025). The benchmark explicitly shows that many queries have No Match at IoU $\mathcal{T}$ 2, indicating that failures are often driven by missing or inaccurate instance segmentation rather than only by feature matching (Kassab et al., 25 Mar 2025). OpenObj, which adds caption features and dense part-level feature fields, reports substantially higher Recall@K than ConceptGraphs on four Replica scenes, including Ontology: R@1 = 0.90, R@2 = 0.95, R@3 = 1.00, Relevance: R@1 = 0.75, R@2 = 0.90, R@3 = 1.00, Functionality: R@1 = 0.90, R@2 = 0.95, R@3 = 0.95, and Part: R@1 = 0.80, R@2 = 0.80, R@3 = 0.80 (Deng et al., 2024).

Image-to-CAD retrieval shows a different empirical profile. OSCAR reports on MI3DOR NN = 0.894, FT = 0.708, ST = 0.850, F = 0.238, DCG = 0.844, ANMRR = 0.205, outperforming all baselines, and further reports 90.48\% average precision during object retrieval on the YCB-V object dataset (Pulli et al., 12 Jan 2026). SAMURAI, under the ROOMELSA single-image masked-object challenge, achieves R@1 = 0.88, R@5 = 1.00, R@10 = 1.00, and MRR = 0.93 (Vo et al., 26 Jun 2025).

Failure modes recur across otherwise different pipelines. CORE-3D notes confusions between very similar categories, sensitivity to poor masks in extremely cluttered scenes or noisy depth, occlusions or incomplete scans that reduce IoU even if semantic retrieval is correct, and query phrasing or ambiguous relational descriptions (Mirzaei et al., 29 Sep 2025). TeDA notes computational cost of 2000 PGD iterations, dependence on rendering quality and view configuration, and reliance on CLIP/InternVL biases, especially for industrial components (Wang et al., 5 May 2025). OSCAR identifies dependence on caption quality, sensitivity to ROI extraction in cluttered scenes, limitations of its fixed 8-view onboarding scheme, and approximate retrieval for functionally different objects with similar appearance (Pulli et al., 12 Jan 2026). BoxOVIS improves tail-class retrieval, but RGBD-based masks built from superpoints can be noisy at higher IoU thresholds, and the current bottleneck is CPU-based Open3D computation of 3D oriented boxes (Nguyen et al., 22 Dec 2025). DEC explicitly observes that further adaptation of DINO can cause severe overfitting on average view patterns of known classes, motivating dynamic local integration and unseen-class regularization (He et al., 21 Apr 2026).

6. Design patterns, adjacent directions, and open problems

Several design principles recur across the literature. Multi-view rendering remains the dominant bridge from 3D geometry to strong 2D encoders: TeDA, DAC, and DEC all operate on rendered views rather than learned 3D backbones, while OrthographicNet shows an earlier rotation- and scale-invariant variant based on orthographic projections (Wang et al., 5 May 2025, Wang et al., 29 Jul 2025, He et al., 21 Apr 2026, Kasaei, 2019). A second recurring principle is to avoid direct dependence on closed vocabularies: CORE-3D is training-free with frozen CLIP and no 3D backbone, OVIR-3D performs open-vocabulary 3D instance retrieval without using any 3D data for training, and BoxOVIS transfers open-vocabulary capability from YOLO-World into 3D without SAM or CLIP at inference (Mirzaei et al., 29 Sep 2025, Lu et al., 2023, Nguyen et al., 22 Dec 2025).

The literature also distinguishes between semantic front-ends and geometric front-ends. OP3Det is not a retrieval method, but it introduces a class-agnostic Open-World Prompt-free 3D Detector that detects any objects within 3D scenes without relying on hand-crafted text prompts; a plausible implication is that such prompt-free, high-recall 3D proposals can serve as front-end units for downstream open-set 3DOR (Liu et al., 20 Oct 2025). 3D-MOOD is likewise an open-set monocular 3D detector rather than a retriever, but its geometry-aware object queries and canonical image space suggest a descriptor source for open-vocabulary, geometry-aware retrieval if additional metric learning is introduced; this suggests a separation between object discovery and semantic retrieval (Yang et al., 31 Jul 2025).

Two unresolved tensions are especially visible. The first is fine-grainedness versus broad semantic alignment. CLIP-based systems benefit from semantically aligned latent spaces, descriptive prompts, and VLM/LLM reasoning, but multiple papers identify insufficient fine-grainedness or viewpoint dependence as limiting factors (Wang et al., 29 Jul 2025, Mirzaei et al., 29 Sep 2025). DINO-based DEC is motivated exactly by this limitation, while OpenObj addresses it with part-level feature fields inside object-level NeRFs (He et al., 21 Apr 2026, Deng et al., 2024). The second is training-free simplicity versus adaptation. CORE-3D, OVIR-3D, OSCAR, and SAMURAI emphasize training-free deployment, whereas TeDA uses test-time optimization, DAC uses parameter-efficient CLIP adaptation with AB-LoRA, and DEC adds dynamic view integration plus virtual unseen features (Mirzaei et al., 29 Sep 2025, Lu et al., 2023, Pulli et al., 12 Jan 2026, Vo et al., 26 Jun 2025, Wang et al., 5 May 2025, Wang et al., 29 Jul 2025, He et al., 21 Apr 2026). This suggests that open-set 3DOR remains split between pure orchestration of pretrained models and lightweight adaptation designed to preserve generalization.

OpenLex3D clarifies that existing open-vocabulary 3D methods are far from saturating linguistically rich, scene-level evaluation, especially in cluttered real-world environments (Kassab et al., 25 Mar 2025). Future work suggested in the papers includes more efficient optimization schemes for test-time distribution alignment, better prompts and 3D-aware large multimodal models, GPU-based 3D uplift and selective mask refinement, richer part-level or region-level alignment, explicit metric learning on prompt-free 3D proposals, and broader benchmarks that move beyond nouns toward affordances, materials, and other semantic attributes (Wang et al., 5 May 2025, Nguyen et al., 22 Dec 2025, Deng et al., 2024, Liu et al., 20 Oct 2025, Kassab et al., 25 Mar 2025). A plausible implication is that progress in open-set 3DOR will continue to come from combining high-quality object proposals, geometry-aware multi-view representations, and language-grounded reasoning rather than from any single backbone family alone.