---
title: Zero-shot Open-Vocabulary 3D Grounding
url: https://www.emergentmind.com/topics/zero-shot-open-vocabulary-3d-visual-grounding
type: topic
---

# Zero-shot Open-Vocabulary 3D Grounding

Searching arXiv for recent papers on zero-shot open-vocabulary 3D visual grounding and related methods.
Zero-shot open-vocabulary 3D visual grounding is the task of localizing an object in a 3D scene from a natural-language description without task-specific labeled training data and without restricting the referent to a closed category vocabulary. In the standard formulation, a system receives a 3D scene and a query and must return the target object’s 3D bounding box or object ID. The problem is difficult because descriptions are often compositional, spatially relational, attribute-heavy, and sometimes view-dependent, while the perceptual front-end may provide only imperfect 3D instances, 2D views, or dense feature fields. Recent work has replaced supervised 3D language matching with symbolic reasoning, tool-using language models, 2D vision-language model reasoning, training-free scene parsing, and hybrid 2D–3D consistency mechanisms, producing a distinct methodological lineage within 3D vision-language research [2411.14594], [2412.04383], [2604.26261].

## 1. Problem definition and task scope

In standard 3D visual grounding, given a point cloud of a scene and a language query such as “the chair under the desk,” the system must identify the target object and return its 3D bounding box [2411.14594]. More generally, the task is often formalized as receiving a 3D scene \(\mathcal{S}\) and a natural-language query \(\mathcal{Q}\), and predicting the target object’s 3D bounding box, as in
\[
\text{bbox} = \mathbf{3DVG}(\mathcal{S}, \mathcal{Q})
\]
or, in a point-cloud-centric formulation,
\[
bbox = 3DVG(S, Q)
\]
[2412.04383], [2605.25901].

The “zero-shot” qualifier denotes the absence of task-specific 3D grounding training at inference time, while “open-vocabulary” denotes the ability to handle categories or descriptions not fixed in a predefined label set [2309.12311], [2605.25901]. Several papers further stress that queries may involve spatial relations such as “left of,” “next to,” or “between,” viewpoint-dependent phrases such as “on the right when facing the TV,” and appearance cues such as color, material, texture, or state [2412.04383], [2504.18684], [2605.25901].

A recurring distinction in the literature concerns what is actually “open.” Some systems remain dependent on a pretrained 3D detector or instance segmenter that supplies a fixed candidate set, after which a language or vision-language model performs zero-shot selection among those candidates [2412.04383], [2605.25901]. Other work explicitly criticizes this dependency and seeks “open-world” behavior by removing reliance on dataset-trained 3D perception modules or by dynamically expanding the object memory during inference [2603.08131], [2512.23020]. This suggests that zero-shot open-vocabulary 3D grounding spans a spectrum from zero-shot candidate selection over fixed proposals to training-free or dynamically extensible scene parsing.

## 2. Why the task is difficult

The literature identifies two persistent bottlenecks. The first is language diversity and compositionality. Queries may combine attributes, multiple landmarks, ordinal expressions, negation, or commonsense implications, as in “the second blue chair under the white table,” “the chair between the dining table and window,” or demand-driven instructions whose intended target is only indirectly specified [2411.14594], [2309.12311], [2508.01723]. Earlier open-vocabulary 2D or 3D retrieval systems are described as behaving like “bags of words,” attending to all mentioned nouns while ignoring syntactic structure, landmark roles, and relation words [2309.12311].

The second bottleneck is scene representation. One family of methods serializes 3D object metadata into text and asks an LLM to reason over object centers, sizes, and labels [2311.15383], [2411.14594]. Another renders images and asks a 2D VLM to reason visually [2410.13860], [2412.04383]. A third constructs dense or graph-structured 3D representations aligned to language [2401.09413], [2402.12259]. Each representation incurs characteristic failure modes. Text serialization may lose appearance information and can become token-heavy [2604.26261]. Image-only reasoning may suffer from view selection, occlusion, projection error, and redundancy across many frames [2410.13860], [2604.26261]. Proposal-based pipelines can fail if the target is absent, miscategorized, or geometrically imprecise in the initial 3D proposal set [2411.14594], [2604.26261].

View dependence is a particularly difficult subproblem. The meaning of left, right, front, and behind depends on a reference frame, and several papers emphasize that these relations cannot be solved robustly from raw global coordinates alone [2311.15383], [2411.14594], [2504.18684]. Methods respond by defining egocentric projections, query-aligned viewpoints, or viewpoint-aware geometric tools [2311.15383], [2412.04383], [2504.18684].

A further complication is evaluation. ScanRefer uses **Acc@0.25** and **Acc@0.5**, where the predicted 3D box is correct if its IoU with ground truth exceeds the threshold, whereas Nr3D evaluates the percentage of queries whose target is correctly selected from known object boxes or top-1 object selection accuracy [2411.14594], [2309.12311], [2410.13860]. This difference means some methods are stronger at semantic selection than at precise box recovery, while others improve chiefly by repairing geometry or proposal quality [2410.13860], [2604.26261].

## 3. Main methodological paradigms

The field has developed several methodological paradigms rather than a single dominant design.

Early programmatic approaches cast grounding as executable reasoning over 3D proposals. “Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding” [2311.15383] uses an LLM to generate a visual program composed of view-independent, view-dependent, and functional modules, then translates the program into Python code for execution. Its key module, the Language-Object Correlation module, combines 3D proposal filtering with 2D appearance-based open-vocabulary disambiguation. CSVG moves this line further by reformulating grounding as a symbolic Constraint Satisfaction Problem \((\mathcal{X}, \mathcal{D}, \mathcal{C})\), where objects become variables, candidate instances become domains, and spatial or ordinal relations become constraints [2411.14594]. The principal claim is that the solver enforces all constraints simultaneously, enabling global reasoning over target and anchor objects rather than local relation-by-relation checks.

A second paradigm treats the LLM as an agent that decomposes the query and orchestrates external tools. LLM-Grounder parses the query into target, attributes, landmarks, and relations; invokes a Target Finder and Landmark Finder based on OpenScene or LERF; then reasons over candidate boxes, distances, volumes, and commonsense cues to make the final decision [2309.12311]. SORT3D likewise uses an LLM as a controller rather than a direct geometric reasoner, combining 2D object captions, LLM-based filtering, and a heuristics-driven spatial reasoning toolbox [2504.18684]. AgentGrounder extends the tool-driven paradigm with an offline Object Lookup Table and an online agent that selectively retrieves relevant objects, performs deterministic geometric scoring, and triggers image rendering only when appearance or viewpoint-sensitive evidence is required [2605.25901].

A third paradigm centers 2D VLM reasoning. VLM-Grounder uses a VLM directly on 2D image sequences of a scanned scene, with query analysis, dynamic stitching of image sequences, a grounding-and-feedback loop, and multi-view ensemble projection to recover the 3D box [2410.13860]. SeeGround similarly converts the 3D scene into a hybrid of query-aligned rendered images and spatially enriched text descriptions, then uses a Perspective Adaptation Module and Fusion Alignment Module to let a strong 2D VLM reason over the aligned multimodal input [2412.04383]. These methods are explicitly motivated by the mismatch between raw 3D inputs and the input distribution of large-scale pretrained 2D VLMs.

A fourth paradigm attempts to improve or replace the 3D perceptual substrate itself. POP-3D predicts an open-vocabulary 3D semantic occupancy map from input 2D images, producing dense voxel-level grounded language embeddings that support free-form language grounding and retrieval [2401.09413]. Open3DSG aligns a 3D scene graph backbone with 2D foundation-model embeddings so that arbitrary object classes can be queried and inter-object relationships can be predicted in an open-set manner by a grounded LLM [2402.12259]. OV-SAM3D is training-free open-vocabulary 3D instance understanding based on superpoints, SAM-derived masks, RAM tags, ChatGPT filtering, and CLIP matching, though it is characterized as instance-level open-vocabulary labeling rather than a full query-conditioned grounding pipeline [2405.15580].

A fifth paradigm focuses on training-free scene parsing and open-world extension. UniGround proposes a two-stage pipeline comprising Global Candidate Filtering and Local Precision Grounding, explicitly criticizing prior “zero-shot” methods for their dependence on pretrained 3D detectors or segmenters [2603.08131]. OpenGround extends the scene memory itself through an Active Cognition-based Reasoning module that iteratively grounds context objects, discovers missing objects from selected views, and inserts them into a dynamically updated OLT [2512.23020]. MCM-VG organizes the pipeline around multiple consistent 2D–3D mappings—semantic alignment, instance rectification, and viewpoint distillation—before posing the final target disambiguation as a multiple-choice reasoning problem for a VLM [2604.26261].

## 4. Representation, reasoning, and 2D–3D alignment

A core technical question is how language is grounded into 3D structure. Symbolic methods define explicit variables, domains, and constraints. In CSVG, the system first performs instance segmentation to obtain instances \(\mathcal{I} = \{i_1, i_2, \dots, i_L\}\), where each instance has a point cloud \(pc(i_k)\) and a label \(label(i_k)\). An LLM then generates Python code defining CSP variables and constraints, with domains such as
\[
\mathcal{D}_j = \{ i_k \in \mathcal{I} \mid label(i_k) = label(x_j) \}
\]
or a label-set variant to handle synonymy and noisy segmentation labels [2411.14594]. Predefined constraint functions implement relations such as \(above\), \(right\), \(between\), \(near\), \(beside\), \(inside\), and ordinal or min/max constraints. The solver uses backtracking, additional handling for min/max and negation, and a final heuristic based on the minimum average pairwise distance among solution objects [2411.14594].

Programmatic methods use a related but less formal execution model. The visual programming framework defines operators such as LOC, CLOSEST, FARTHEST, LEFT, RIGHT, FRONT, BEHIND, MIN, MAX, and SIZE [2311.15383]. View-dependent relations are implemented by placing a virtual camera at the scene center, orienting it toward an anchor, and projecting 3D points into an egocentric 2D frame through a look-at transform and camera projection. This yields an interpretable pipeline in which language is first decomposed into subproblems, then grounded through deterministic modules rather than direct free-form LLM answers [2311.15383].

Tool-driven agents instead separate symbolic planning from numerical computation. SORT3D represents each candidate object as \(\{id, name, caption, c_x, c_y, c_z, size\}\) and asks the LLM to invoke heuristic operators for search, ranking, set intersection, set union, and viewpoint-aware geometric relation checking [2504.18684]. AgentGrounder formalizes a similar workflow with an Object Lookup Table
\[
\mathcal{O}=\{(id_i, \ell_i, c_i, d_i)\}_{i=1}^{N},
\]
query decomposition
\[
(\mathcal{L}, \mathcal{R}) = PlanExtract(Q),
\]
candidate retrieval
\[
\mathcal{C} = \{o_i \in \mathcal{O} \mid \ell_i \in \mathcal{L}\},
\]
and geometric scoring
\[
s(o_i \mid Q) = f_{geo}(o_i, \mathcal{R}, \mathcal{O}), \qquad \hat{o} = \arg\max_{o_i \in \mathcal{C}} s(o_i \mid Q)
\]
[2605.25901]. The distinctive feature is on-demand rendering:
\[
I = Render(S, \mathcal{I}_{cand}), \qquad \hat{o} = Resolve(\hat{o}, I, Q),
\]
used only when geometry is insufficient.

VLM-centric systems address the same grounding problem through alignment between rendered or observed 2D evidence and 3D object metadata. SeeGround begins with an open-vocabulary 3D object detector whose output is stored in an OLT, then constructs a hybrid representation \((\mathcal{I}, \mathcal{T}) = \mathcal{F}(\mathcal{S}, \mathcal{Q}, \mathcal{OLT})\) containing a rendered image and a spatially enriched text description [2412.04383]. The Perspective Adaptation Module selects a query-aligned viewpoint based on inferred anchors and candidate targets, and the Fusion Alignment Module uses depth-aware visual prompting to mark visible projected objects without occluding texture or color cues. VLM-Grounder instead selects and stitches RGB views, lets the VLM identify the relevant image and instance, then projects the 2D mask into 3D and refines the box by PATS-based multi-view matching, mask erosion, connected-component filtering, Chamfer-distance-based matched-view rejection, and union of point clouds [2410.13860].

A distinct form of 2D–3D alignment appears in methods that produce 3D semantic feature fields. POP-3D’s 3D-language head outputs a feature vector for every voxel,
\[
O_\text{ft} = h(V),
\]
trained to live in the same semantic space as MaskCLIP+ features sampled through LiDAR-image correspondence [2401.09413]. At inference, free-form text is encoded and compared with each occupied voxel embedding to produce a 3D similarity heatmap or thresholded localization. Open3DSG uses a co-embedding strategy in which 3D node and edge features are distilled toward OpenSeg and InstructBLIP features with the cosine-based loss
\[
L = 1-\cos(o_{2D},o_{3D}) + 1-\cos(r_{2D},r_{3D}),
\]
enabling open-vocabulary object querying and open-set relationship generation over a 3D scene graph [2402.12259].

## 5. Query types, datasets, metrics, and empirical trends

The dominant evaluation benchmarks are ScanRefer and Nr3D, both built on ScanNet scenes [2411.14594]. ScanRefer reports **Acc@0.25** and **Acc@0.5**, whereas Nr3D reports target-selection accuracy and often includes **Easy/Hard** and **View-Dependent/View-Independent** splits [2411.14594], [2412.04383], [2504.18684]. Some later work also evaluates on Sr3D, VLA-3D, EmbodiedScan, Matterport3D, or newly introduced benchmarks such as OpenTarget, depending on whether the emphasis is standard 3DVG, view-dependent reasoning, navigation-oriented instruction grounding, or open-world target discovery [2504.18684], [2603.08131], [2508.01723], [2512.23020].

The table summarizes representative results explicitly reported in the cited papers.

| Method | Benchmark | Reported result |
|---|---|---|
| CSVG [2411.14594] | ScanRefer | **49.6 Acc@0.25**, **39.8 Acc@0.5** with Mask3D segmentation |
| CSVG [2411.14594] | Nr3D | **59.2 overall** |
| VLM-Grounder [2410.13860] | ScanRefer | **51.6% Acc@0.25**, **32.8% Acc@0.5** |
| VLM-Grounder [2410.13860] | Nr3D | **48.0% overall** |
| SeeGround [2412.04383] | ScanRefer | **44.1 / 39.4** overall Acc@0.25 / Acc@0.5 |
| SeeGround [2412.04383] | Nr3D | **46.1 overall** |
| SORT3D [2504.18684] | Nr3D | **60.5 ± 1.0 overall** with GPT-4o |
| SORT3D [2504.18684] | VLA-3D | **71.8 ± 1.8 overall** with GPT-4o |
| AgentGrounder [2605.25901] | ScanRefer | **41.9 Acc@0.5**, **47.2 Acc@0.25** overall |
| AgentGrounder [2605.25901] | Nr3D | **52.4 overall** |
| UniGround [2603.08131] | ScanRefer | **46.1% Acc@0.25**, **34.1% Acc@0.5** |
| OpenGround [2512.23020] | ScanRefer | **61.8% Acc@0.25**, **53.1% Acc@0.50** |
| MCM-VG [2604.26261] | ScanRefer | **62.0% Acc@0.25**, **53.6% Acc@0.5** |

Several empirical patterns recur across papers. First, methods that improve candidate quality or geometric precision tend to gain especially on **Acc@0.5**, which is more sensitive to box accuracy than coarse semantic selection [2411.14594], [2604.26261]. CSVG’s ground-truth segmentation result of **66.3 Acc@0.25** and **61.6 Acc@0.5** on ScanRefer is explicitly presented as evidence that segmentation remains a major bottleneck [2411.14594]. VLM-Grounder’s additional 2D-mask evaluation is said to surpass BUTD-DETR from the 2D perspective, suggesting that its residual error is dominated by projection noise rather than query understanding [2410.13860].

Second, view-dependent reasoning remains a discriminative challenge. SORT3D emphasizes gains on view-dependent subsets by using feasible viewpoints rather than naive scene axes for left/right relations [2504.18684]. SeeGround attributes part of its gains to query-aligned viewpoint selection and depth-aware visual prompting [2412.04383]. MCM-VG explicitly treats viewpoint redundancy as a problem and introduces clustering of camera directions plus RGB–BEV prompt pairs to reduce reasoning noise [2604.26261].

Third, open-vocabulary attribute information materially improves performance. SORT3D reports that captions improve its Nr3D view-dependent performance from **50.7** to **56.6**, with a gain of **11.6%** reported in the ablation table [2504.18684]. SeeGround is motivated by queries depending on color, texture, state, and orientation, while AgentGrounder uses on-demand rendering specifically for color, material, and viewpoint-sensitive cues [2412.04383], [2605.25901].

## 6. Extensions, limitations, and unresolved issues

A notable feature of the area is the rapid extension of grounding systems beyond simple target selection. CSVG explicitly demonstrates handling of negation-based and counting-based queries with only minor extra coding efforts, using `DEFINE_NEGATIVE_VARIABLE(labels=[...])` and ordered variables such as \(CHAIR_0, CHAIR_1, CHAIR_2\) together with `CONSTRAINT_LEFT` or `CONSTRAINT_MORE` [2411.14594]. Open3DSG generalizes from single-object grounding to queryable open-vocabulary 3D scene graphs with open-set relationship generation [2402.12259]. Reasoning3D pushes toward part-level, reasoning-based 3D segmentation from implicit prompts such as “the part of the chair where you would sit,” using multi-view 2D reasoning segmentation projected back onto meshes [2405.19326]. OpenMap targets instruction-to-instance retrieval in embodied navigation, combining structural-semantic consensus for 3D instance aggregation with LLM-assisted free-form instruction parsing and contextual reranking [2508.01723]. These developments suggest that zero-shot open-vocabulary 3D visual grounding is increasingly understood as a general interface between free-form language and 3D scene representations rather than a narrow benchmark task.

At the same time, the literature is explicit about limitations. Many methods remain bottlenecked by 3D instance segmentation or initial proposal quality; if the perceptual front-end misses the target, no amount of reasoning can fully recover [2411.14594], [2605.25901], [2604.26261]. Appearance is incompletely modeled in systems that reason chiefly over geometry or labels [2411.14594]. View selection and multi-view fusion remain brittle, particularly under clutter, occlusion, or poor reconstruction quality [2405.19326], [2410.13860], [2412.04383]. VLM- or LLM-based pipelines often have substantial latency due to iterative prompting, rendering, or API dependence [2309.12311], [2410.13860], [2504.18684].

A deeper controversy concerns what counts as true zero-shot or open-world grounding. UniGround argues that many methods labeled zero-shot still rely on supervised 3D detection or segmentation and therefore remain bounded by the ontology and generalization limits of the 3D front-end [2603.08131]. OpenGround makes a related argument against pre-defined OLTs, proposing dynamic OLT augmentation through Active Cognition Enhancement so that referents absent from the initial object inventory can still be discovered [2512.23020]. A plausible implication is that future progress will be measured not only by higher ScanRefer or Nr3D accuracy, but also by reduced dependence on closed-set perceptual assumptions.

Another unresolved issue is the balance between symbolic and visual reasoning. CSVG and related programmatic methods show that explicit spatial structure, global constraint coupling, and deterministic scoring can outperform purely local or free-form LLM reasoning on standard benchmarks [2411.14594]. Conversely, SeeGround, VLM-Grounder, and MCM-VG argue that rich 2D visual evidence is indispensable for color, texture, state, and fine-grained attribute disambiguation [2410.13860], [2412.04383], [2604.26261]. This suggests that the most successful future systems may continue to hybridize explicit geometry, selective visual inspection, and structured tool use rather than relying on a single monolithic model.

## 7. Broader significance and research direction

Taken together, the recent literature defines zero-shot open-vocabulary 3D visual grounding as a problem of aligning free-form language with structured 3D evidence under severe supervision constraints. The field has moved from direct LLM reasoning over serialized object descriptions [2309.12311], through visual programming and symbolic constraint solving [2311.15383], [2411.14594], to 2D-VLM-centered hybrid representations [2410.13860], [2412.04383], and more recent training-free or open-world architectures that repair, replace, or dynamically expand the scene representation itself [2603.08131], [2512.23020], [2604.26261].

Several common design principles now recur across strong systems. One is selective reasoning: retrieve only relevant objects, views, or features rather than exposing the model to the full scene [2504.18684], [2605.25901]. Another is explicit 2D–3D consistency, whether through projection and depth-aware prompting, dense feature distillation, or back-projected segmentation rectification [2401.09413], [2412.04383], [2604.26261]. A third is decomposition of grounding into semantically distinct subproblems—candidate filtering, relation reasoning, visual verification, and box recovery—handled by specialized modules instead of a single opaque predictor [2411.14594], [2504.18684], [2605.25901].

The broader implication is that zero-shot open-vocabulary 3D grounding is evolving from a benchmark-specific retrieval task into a general substrate for embodied AI, robotics, navigation, scene understanding, and reasoning-intensive 3D interaction. The strongest current evidence supports hybrid systems that combine structured geometry, selective language-model orchestration, and adaptive visual inspection, while remaining attentive to the unresolved problems of proposal quality, viewpoint dependence, context efficiency, and genuinely open-world scene parsing [2411.14594], [2504.18684], [2605.25901], [2604.26261].

Source: https://www.emergentmind.com/topics/zero-shot-open-vocabulary-3d-visual-grounding