Zero-shot Open-Vocabulary 3D Grounding

• The paper presents a framework that uses zero-shot strategies to localize objects in 3D scenes based on open-ended language descriptions.
• It compares symbolic reasoning, tool-using LLMs, and hybrid 2D–3D approaches to address spatial, visual, and semantic challenges.
• Key implications include advancements in embodied AI, robotics, and navigation through improved scene parsing and dynamic candidate selection.

Searching arXiv for papers on zero-shot open-vocabulary 3D visual grounding and related methods. Zero-shot open-vocabulary 3D visual grounding is the task of localizing an object in a 3D scene from a natural-language description without task-specific labeled training data and without restricting the referent to a closed category vocabulary. In the standard formulation, a system receives a 3D scene and a query and must return the target object’s 3D bounding box or object ID. The problem is difficult because descriptions are often compositional, spatially relational, attribute-heavy, and sometimes view-dependent, while the perceptual front-end may provide only imperfect 3D instances, 2D views, or dense feature fields. Recent work has replaced supervised 3D language matching with symbolic reasoning, tool-using LLMs, 2D vision-LLM reasoning, training-free scene parsing, and hybrid 2D–3D consistency mechanisms, producing a distinct methodological lineage within 3D vision-language research (Yuan et al., 2024, Li et al., 2024, Yin et al., 29 Apr 2026).

1. Problem definition and task scope

In standard 3D visual grounding, given a point cloud of a scene and a language query such as “the chair under the desk,” the system must identify the target object and return its 3D bounding box (Yuan et al., 2024). More generally, the task is often formalized as receiving a 3D scene $\mathcal{S}$ and a natural-language query $\mathcal{Q}$, and predicting the target object’s 3D bounding box, as in

$$\text{bbox} = \mathbf{3DVG}(\mathcal{S}, \mathcal{Q})$$

or, in a point-cloud-centric formulation,

$bbox = 3DVG(S, Q)$

(Li et al., 2024, Huynh et al., 25 May 2026).

The “zero-shot” qualifier denotes the absence of task-specific 3D grounding training at inference time, while “open-vocabulary” denotes the ability to handle categories or descriptions not fixed in a predefined label set (Yang et al., 2023, Huynh et al., 25 May 2026). Several papers further stress that queries may involve spatial relations such as “left of,” “next to,” or “between,” viewpoint-dependent phrases such as “on the right when facing the TV,” and appearance cues such as color, material, texture, or state (Li et al., 2024, Zantout et al., 25 Apr 2025, Huynh et al., 25 May 2026).

A recurring distinction in the literature concerns what is actually “open.” Some systems remain dependent on a pretrained 3D detector or instance segmenter that supplies a fixed candidate set, after which a language or vision-LLM performs zero-shot selection among those candidates (Li et al., 2024, Huynh et al., 25 May 2026). Other work explicitly criticizes this dependency and seeks “open-world” behavior by removing reliance on dataset-trained 3D perception modules or by dynamically expanding the object memory during inference (Zhang et al., 9 Mar 2026, Huang et al., 28 Dec 2025). This suggests that zero-shot open-vocabulary 3D grounding spans a spectrum from zero-shot candidate selection over fixed proposals to training-free or dynamically extensible scene parsing.

2. Why the task is difficult

The literature identifies two persistent bottlenecks. The first is language diversity and compositionality. Queries may combine attributes, multiple landmarks, ordinal expressions, negation, or commonsense implications, as in “the second blue chair under the white table,” “the chair between the dining table and window,” or demand-driven instructions whose intended target is only indirectly specified (Yuan et al., 2024, Yang et al., 2023, Li et al., 3 Aug 2025). Earlier open-vocabulary 2D or 3D retrieval systems are described as behaving like “bags of words,” attending to all mentioned nouns while ignoring syntactic structure, landmark roles, and relation words (Yang et al., 2023).

The second bottleneck is scene representation. One family of methods serializes 3D object metadata into text and asks an LLM to reason over object centers, sizes, and labels (Yuan et al., 2023, Yuan et al., 2024). Another renders images and asks a 2D VLM to reason visually (Xu et al., 2024, Li et al., 2024). A third constructs dense or graph-structured 3D representations aligned to language (Vobecky et al., 2024, Koch et al., 2024). Each representation incurs characteristic failure modes. Text serialization may lose appearance information and can become token-heavy (Yin et al., 29 Apr 2026). Image-only reasoning may suffer from view selection, occlusion, projection error, and redundancy across many frames (Xu et al., 2024, Yin et al., 29 Apr 2026). Proposal-based pipelines can fail if the target is absent, miscategorized, or geometrically imprecise in the initial 3D proposal set (Yuan et al., 2024, Yin et al., 29 Apr 2026).

View dependence is a particularly difficult subproblem. The meaning of left, right, front, and behind depends on a reference frame, and several papers emphasize that these relations cannot be solved robustly from raw global coordinates alone (Yuan et al., 2023, Yuan et al., 2024, Zantout et al., 25 Apr 2025). Methods respond by defining egocentric projections, query-aligned viewpoints, or viewpoint-aware geometric tools (Yuan et al., 2023, Li et al., 2024, Zantout et al., 25 Apr 2025).

A further complication is evaluation. ScanRefer uses [email protected] and [email protected], where the predicted 3D box is correct if its IoU with ground truth exceeds the threshold, whereas Nr3D evaluates the percentage of queries whose target is correctly selected from known object boxes or top-1 object selection accuracy (Yuan et al., 2024, Yang et al., 2023, Xu et al., 2024). This difference means some methods are stronger at semantic selection than at precise box recovery, while others improve chiefly by repairing geometry or proposal quality (Xu et al., 2024, Yin et al., 29 Apr 2026).

3. Main methodological paradigms

The field has developed several methodological paradigms rather than a single dominant design.

Early programmatic approaches cast grounding as executable reasoning over 3D proposals. “Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding” (Yuan et al., 2023) uses an LLM to generate a visual program composed of view-independent, view-dependent, and functional modules, then translates the program into Python code for execution. Its key module, the Language-Object Correlation module, combines 3D proposal filtering with 2D appearance-based open-vocabulary disambiguation. CSVG moves this line further by reformulating grounding as a symbolic Constraint Satisfaction Problem $(\mathcal{X}, \mathcal{D}, \mathcal{C})$, where objects become variables, candidate instances become domains, and spatial or ordinal relations become constraints (Yuan et al., 2024). The principal claim is that the solver enforces all constraints simultaneously, enabling global reasoning over target and anchor objects rather than local relation-by-relation checks.

A second paradigm treats the LLM as an agent that decomposes the query and orchestrates external tools. LLM-Grounder parses the query into target, attributes, landmarks, and relations; invokes a Target Finder and Landmark Finder based on OpenScene or LERF; then reasons over candidate boxes, distances, volumes, and commonsense cues to make the final decision (Yang et al., 2023). SORT3D likewise uses an LLM as a controller rather than a direct geometric reasoner, combining 2D object captions, LLM-based filtering, and a heuristics-driven spatial reasoning toolbox (Zantout et al., 25 Apr 2025). AgentGrounder extends the tool-driven paradigm with an offline Object Lookup Table and an online agent that selectively retrieves relevant objects, performs deterministic geometric scoring, and triggers image rendering only when appearance or viewpoint-sensitive evidence is required (Huynh et al., 25 May 2026).

A third paradigm centers 2D VLM reasoning. VLM-Grounder uses a VLM directly on 2D image sequences of a scanned scene, with query analysis, dynamic stitching of image sequences, a grounding-and-feedback loop, and multi-view ensemble projection to recover the 3D box (Xu et al., 2024). SeeGround similarly converts the 3D scene into a hybrid of query-aligned rendered images and spatially enriched text descriptions, then uses a Perspective Adaptation Module and Fusion Alignment Module to let a strong 2D VLM reason over the aligned multimodal input (Li et al., 2024). These methods are explicitly motivated by the mismatch between raw 3D inputs and the input distribution of large-scale pretrained 2D VLMs.

A fourth paradigm attempts to improve or replace the 3D perceptual substrate itself. POP-3D predicts an open-vocabulary 3D semantic occupancy map from input 2D images, producing dense voxel-level grounded language embeddings that support free-form language grounding and retrieval (Vobecky et al., 2024). Open3DSG aligns a 3D scene graph backbone with 2D foundation-model embeddings so that arbitrary object classes can be queried and inter-object relationships can be predicted in an open-set manner by a grounded LLM (Koch et al., 2024). OV-SAM3D is training-free open-vocabulary 3D instance understanding based on superpoints, SAM-derived masks, RAM tags, ChatGPT filtering, and CLIP matching, though it is characterized as instance-level open-vocabulary labeling rather than a full query-conditioned grounding pipeline (Tai et al., 2024).

A fifth paradigm focuses on training-free scene parsing and open-world extension. UniGround proposes a two-stage pipeline comprising Global Candidate Filtering and Local Precision Grounding, explicitly criticizing prior “zero-shot” methods for their dependence on pretrained 3D detectors or segmenters (Zhang et al., 9 Mar 2026). OpenGround extends the scene memory itself through an Active Cognition-based Reasoning module that iteratively grounds context objects, discovers missing objects from selected views, and inserts them into a dynamically updated OLT (Huang et al., 28 Dec 2025). MCM-VG organizes the pipeline around multiple consistent 2D–3D mappings—semantic alignment, instance rectification, and viewpoint distillation—before posing the final target disambiguation as a multiple-choice reasoning problem for a VLM (Yin et al., 29 Apr 2026).

4. Representation, reasoning, and 2D–3D alignment

A core technical question is how language is grounded into 3D structure. Symbolic methods define explicit variables, domains, and constraints. In CSVG, the system first performs instance segmentation to obtain instances $\mathcal{I} = \{i_1, i_2, \dots, i_L\}$, where each instance has a point cloud $pc(i_k)$ and a label $label(i_k)$. An LLM then generates Python code defining CSP variables and constraints, with domains such as

$$\mathcal{D}_j = \{ i_k \in \mathcal{I} \mid label(i_k) = label(x_j) \}$$

or a label-set variant to handle synonymy and noisy segmentation labels (Yuan et al., 2024). Predefined constraint functions implement relations such as $above$, $\mathcal{Q}$0, $\mathcal{Q}$1, $\mathcal{Q}$2, $\mathcal{Q}$3, $\mathcal{Q}$4, and ordinal or min/max constraints. The solver uses backtracking, additional handling for min/max and negation, and a final heuristic based on the minimum average pairwise distance among solution objects (Yuan et al., 2024).

Programmatic methods use a related but less formal execution model. The visual programming framework defines operators such as LOC, CLOSEST, FARTHEST, LEFT, RIGHT, FRONT, BEHIND, MIN, MAX, and SIZE (Yuan et al., 2023). View-dependent relations are implemented by placing a virtual camera at the scene center, orienting it toward an anchor, and projecting 3D points into an egocentric 2D frame through a look-at transform and camera projection. This yields an interpretable pipeline in which language is first decomposed into subproblems, then grounded through deterministic modules rather than direct free-form LLM answers (Yuan et al., 2023).

Tool-driven agents instead separate symbolic planning from numerical computation. SORT3D represents each candidate object as $\mathcal{Q}$5 and asks the LLM to invoke heuristic operators for search, ranking, set intersection, set union, and viewpoint-aware geometric relation checking (Zantout et al., 25 Apr 2025). AgentGrounder formalizes a similar workflow with an Object Lookup Table

$\mathcal{Q}$6

query decomposition

$\mathcal{Q}$7

candidate retrieval

$\mathcal{Q}$8

and geometric scoring

$\mathcal{Q}$9

(Huynh et al., 25 May 2026). The distinctive feature is on-demand rendering: $$\text{bbox} = \mathbf{3DVG}(\mathcal{S}, \mathcal{Q})$$0 used only when geometry is insufficient.

VLM-centric systems address the same grounding problem through alignment between rendered or observed 2D evidence and 3D object metadata. SeeGround begins with an open-vocabulary 3D object detector whose output is stored in an OLT, then constructs a hybrid representation $$\text{bbox} = \mathbf{3DVG}(\mathcal{S}, \mathcal{Q})$$1 containing a rendered image and a spatially enriched text description (Li et al., 2024). The Perspective Adaptation Module selects a query-aligned viewpoint based on inferred anchors and candidate targets, and the Fusion Alignment Module uses depth-aware visual prompting to mark visible projected objects without occluding texture or color cues. VLM-Grounder instead selects and stitches RGB views, lets the VLM identify the relevant image and instance, then projects the 2D mask into 3D and refines the box by PATS-based multi-view matching, mask erosion, connected-component filtering, Chamfer-distance-based matched-view rejection, and union of point clouds (Xu et al., 2024).

A distinct form of 2D–3D alignment appears in methods that produce 3D semantic feature fields. POP-3D’s 3D-language head outputs a feature vector for every voxel,

$$\text{bbox} = \mathbf{3DVG}(\mathcal{S}, \mathcal{Q})$$2

trained to live in the same semantic space as MaskCLIP+ features sampled through LiDAR-image correspondence (Vobecky et al., 2024). At inference, free-form text is encoded and compared with each occupied voxel embedding to produce a 3D similarity heatmap or thresholded localization. Open3DSG uses a co-embedding strategy in which 3D node and edge features are distilled toward OpenSeg and InstructBLIP features with the cosine-based loss

$$\text{bbox} = \mathbf{3DVG}(\mathcal{S}, \mathcal{Q})$$3

enabling open-vocabulary object querying and open-set relationship generation over a 3D scene graph (Koch et al., 2024).

5. Query types, datasets, metrics, and empirical trends

The dominant evaluation benchmarks are ScanRefer and Nr3D, both built on ScanNet scenes (Yuan et al., 2024). ScanRefer reports [email protected] and [email protected], whereas Nr3D reports target-selection accuracy and often includes Easy/Hard and View-Dependent/View-Independent splits (Yuan et al., 2024, Li et al., 2024, Zantout et al., 25 Apr 2025). Some later work also evaluates on Sr3D, VLA-3D, EmbodiedScan, Matterport3D, or newly introduced benchmarks such as OpenTarget, depending on whether the emphasis is standard 3DVG, view-dependent reasoning, navigation-oriented instruction grounding, or open-world target discovery (Zantout et al., 25 Apr 2025, Zhang et al., 9 Mar 2026, Li et al., 3 Aug 2025, Huang et al., 28 Dec 2025).

The table summarizes representative results explicitly reported in the cited papers.

Method	Benchmark	Reported result
CSVG (Yuan et al., 2024)	ScanRefer	49.6 [email protected], 39.8 [email protected] with Mask3D segmentation
CSVG (Yuan et al., 2024)	Nr3D	59.2 overall
VLM-Grounder (Xu et al., 2024)	ScanRefer	51.6% [email protected], 32.8% [email protected]
VLM-Grounder (Xu et al., 2024)	Nr3D	48.0% overall
SeeGround (Li et al., 2024)	ScanRefer	44.1 / 39.4 overall [email protected] / [email protected]
SeeGround (Li et al., 2024)	Nr3D	46.1 overall
SORT3D (Zantout et al., 25 Apr 2025)	Nr3D	60.5 ± 1.0 overall with GPT-4o
SORT3D (Zantout et al., 25 Apr 2025)	VLA-3D	71.8 ± 1.8 overall with GPT-4o
AgentGrounder (Huynh et al., 25 May 2026)	ScanRefer	41.9 [email protected], 47.2 [email protected] overall
AgentGrounder (Huynh et al., 25 May 2026)	Nr3D	52.4 overall
UniGround (Zhang et al., 9 Mar 2026)	ScanRefer	46.1% [email protected], 34.1% [email protected]
OpenGround (Huang et al., 28 Dec 2025)	ScanRefer	61.8% [email protected], 53.1% [email protected]
MCM-VG (Yin et al., 29 Apr 2026)	ScanRefer	62.0% [email protected], 53.6% [email protected]

Several empirical patterns recur across papers. First, methods that improve candidate quality or geometric precision tend to gain especially on [email protected], which is more sensitive to box accuracy than coarse semantic selection (Yuan et al., 2024, Yin et al., 29 Apr 2026). CSVG’s ground-truth segmentation result of 66.3 [email protected] and 61.6 [email protected] on ScanRefer is explicitly presented as evidence that segmentation remains a major bottleneck (Yuan et al., 2024). VLM-Grounder’s additional 2D-mask evaluation is said to surpass BUTD-DETR from the 2D perspective, suggesting that its residual error is dominated by projection noise rather than query understanding (Xu et al., 2024).

Second, view-dependent reasoning remains a discriminative challenge. SORT3D emphasizes gains on view-dependent subsets by using feasible viewpoints rather than naive scene axes for left/right relations (Zantout et al., 25 Apr 2025). SeeGround attributes part of its gains to query-aligned viewpoint selection and depth-aware visual prompting (Li et al., 2024). MCM-VG explicitly treats viewpoint redundancy as a problem and introduces clustering of camera directions plus RGB–BEV prompt pairs to reduce reasoning noise (Yin et al., 29 Apr 2026).

Third, open-vocabulary attribute information materially improves performance. SORT3D reports that captions improve its Nr3D view-dependent performance from 50.7 to 56.6, with a gain of 11.6% reported in the ablation table (Zantout et al., 25 Apr 2025). SeeGround is motivated by queries depending on color, texture, state, and orientation, while AgentGrounder uses on-demand rendering specifically for color, material, and viewpoint-sensitive cues (Li et al., 2024, Huynh et al., 25 May 2026).

6. Extensions, limitations, and unresolved issues

A notable feature of the area is the rapid extension of grounding systems beyond simple target selection. CSVG explicitly demonstrates handling of negation-based and counting-based queries with only minor extra coding efforts, using DEFINE_NEGATIVE_VARIABLE(labels=[...]) and ordered variables such as $$\text{bbox} = \mathbf{3DVG}(\mathcal{S}, \mathcal{Q})$$4 together with CONSTRAINT_LEFT or CONSTRAINT_MORE (Yuan et al., 2024). Open3DSG generalizes from single-object grounding to queryable open-vocabulary 3D scene graphs with open-set relationship generation (Koch et al., 2024). Reasoning3D pushes toward part-level, reasoning-based 3D segmentation from implicit prompts such as “the part of the chair where you would sit,” using multi-view 2D reasoning segmentation projected back onto meshes (Chen et al., 2024). OpenMap targets instruction-to-instance retrieval in embodied navigation, combining structural-semantic consensus for 3D instance aggregation with LLM-assisted free-form instruction parsing and contextual reranking (Li et al., 3 Aug 2025). These developments suggest that zero-shot open-vocabulary 3D visual grounding is increasingly understood as a general interface between free-form language and 3D scene representations rather than a narrow benchmark task.

At the same time, the literature is explicit about limitations. Many methods remain bottlenecked by 3D instance segmentation or initial proposal quality; if the perceptual front-end misses the target, no amount of reasoning can fully recover (Yuan et al., 2024, Huynh et al., 25 May 2026, Yin et al., 29 Apr 2026). Appearance is incompletely modeled in systems that reason chiefly over geometry or labels (Yuan et al., 2024). View selection and multi-view fusion remain brittle, particularly under clutter, occlusion, or poor reconstruction quality (Chen et al., 2024, Xu et al., 2024, Li et al., 2024). VLM- or LLM-based pipelines often have substantial latency due to iterative prompting, rendering, or API dependence (Yang et al., 2023, Xu et al., 2024, Zantout et al., 25 Apr 2025).

A deeper controversy concerns what counts as true zero-shot or open-world grounding. UniGround argues that many methods labeled zero-shot still rely on supervised 3D detection or segmentation and therefore remain bounded by the ontology and generalization limits of the 3D front-end (Zhang et al., 9 Mar 2026). OpenGround makes a related argument against pre-defined OLTs, proposing dynamic OLT augmentation through Active Cognition Enhancement so that referents absent from the initial object inventory can still be discovered (Huang et al., 28 Dec 2025). A plausible implication is that future progress will be measured not only by higher ScanRefer or Nr3D accuracy, but also by reduced dependence on closed-set perceptual assumptions.

Another unresolved issue is the balance between symbolic and visual reasoning. CSVG and related programmatic methods show that explicit spatial structure, global constraint coupling, and deterministic scoring can outperform purely local or free-form LLM reasoning on standard benchmarks (Yuan et al., 2024). Conversely, SeeGround, VLM-Grounder, and MCM-VG argue that rich 2D visual evidence is indispensable for color, texture, state, and fine-grained attribute disambiguation (Xu et al., 2024, Li et al., 2024, Yin et al., 29 Apr 2026). This suggests that the most successful future systems may continue to hybridize explicit geometry, selective visual inspection, and structured tool use rather than relying on a single monolithic model.

7. Broader significance and research direction

Taken together, the recent literature defines zero-shot open-vocabulary 3D visual grounding as a problem of aligning free-form language with structured 3D evidence under severe supervision constraints. The field has moved from direct LLM reasoning over serialized object descriptions (Yang et al., 2023), through visual programming and symbolic constraint solving (Yuan et al., 2023, Yuan et al., 2024), to 2D-VLM-centered hybrid representations (Xu et al., 2024, Li et al., 2024), and more recent training-free or open-world architectures that repair, replace, or dynamically expand the scene representation itself (Zhang et al., 9 Mar 2026, Huang et al., 28 Dec 2025, Yin et al., 29 Apr 2026).

Several common design principles now recur across strong systems. One is selective reasoning: retrieve only relevant objects, views, or features rather than exposing the model to the full scene (Zantout et al., 25 Apr 2025, Huynh et al., 25 May 2026). Another is explicit 2D–3D consistency, whether through projection and depth-aware prompting, dense feature distillation, or back-projected segmentation rectification (Vobecky et al., 2024, Li et al., 2024, Yin et al., 29 Apr 2026). A third is decomposition of grounding into semantically distinct subproblems—candidate filtering, relation reasoning, visual verification, and box recovery—handled by specialized modules instead of a single opaque predictor (Yuan et al., 2024, Zantout et al., 25 Apr 2025, Huynh et al., 25 May 2026).

The broader implication is that zero-shot open-vocabulary 3D grounding is evolving from a benchmark-specific retrieval task into a general substrate for embodied AI, robotics, navigation, scene understanding, and reasoning-intensive 3D interaction. The strongest current evidence supports hybrid systems that combine structured geometry, selective language-model orchestration, and adaptive visual inspection, while remaining attentive to the unresolved problems of proposal quality, viewpoint dependence, context efficiency, and genuinely open-world scene parsing (Yuan et al., 2024, Zantout et al., 25 Apr 2025, Huynh et al., 25 May 2026, Yin et al., 29 Apr 2026).