Open-Vocabulary 3D Visual Grounding

• Open-vocabulary 3D visual grounding is a task that locates arbitrary natural language-described entities in 3D scenes without relying on fixed class labels.
• It integrates neural representations, vision-language models, and structured scene graphs to map free-form text queries to precise spatial regions.
• Advances in this area enable robust applications in embodied AI, robotics, augmented reality, and autonomous navigation while addressing challenges like occlusion and dynamic environments.

Open-vocabulary 3D visual grounding refers to the task of localizing arbitrary target entities—objects, regions, parts, or affordances—within a 3D environment using unconstrained natural language queries, beyond any fixed set of class or relation labels. This capability is foundational in domains such as embodied AI, robotics, augmented reality, and autonomous navigation, where agents must perceive, interpret, and manipulate their surroundings in response to instructions that can reference unseen object categories, complex descriptors, spatial contexts, and compositional attributes. The field has advanced rapidly, enabled by neural representations, vision-LLM (VLM) architectures, LLMs, structured scene graphs, and self-supervised learning. Core innovations span mapping open-vocabulary semantics to 3D geometry, contextual reasoning over relations, training-free or zero-shot pipelines, and robotics integration.

1. Core Principles and Task Formulation

Open-vocabulary 3D visual grounding generalizes the closed-set 3DVG problem by localizing entities described by arbitrary free-form text within a 3D scene, eschewing reliance on fixed taxonomies of object classes or spatial predicates. The system must map a query such as "fetch the controller from the washing area" to a precise 3D region, possibly composed of multiple modalities and contextual relations, and must do so for both known and previously unseen categories or affordances.

Formally, the input is a 3D scene representation (point cloud, mesh, voxel grid, neural field, or scene graph) and a language query $Q$; the output is a 3D bounding volume, mask, or spatial annotation $M^*$ such that

$$M^* = \arg\max_{M \subset S} \mathrm{Score}(Q, M; S)$$

where the scoring function measures how well $M$ satisfies the textual description given the scene $S$ (Liu et al., 9 Jul 2025, Shao et al., 2024). The open-vocabulary property mandates compositional generalization, allowing the model to ground queries for any entity or attribute encountered at test time (Yang et al., 2023, Koch et al., 2024).

The task encompasses subproblems including object/entity instance grounding, referential region selection, context-aware relation matching, affordance localization, and instruction-to-action mapping. Real-world deployments require robustness to dynamic, unstructured environments, handling of partial point clouds, fragmented 3D entities, and semantic ambiguities in query language (Qiu et al., 2024).

2. Scene Representations and Semantics

Representing 3D environments to support open-vocabulary grounding requires rich alignment of geometric structure with semantic content:

• Volumetric 3D Semantic Maps: Workspace is discretized into a voxel grid $V=\{v_i\}$, each with occupancy $o(v_i)$ and semantic-score vector $s(v_i)$ across a semantic space of classes or phrases. Online updates fuse 2D detections (via pre-trained VLMs) into this grid using depth reprojection and instance-based score accumulation (Qiu et al., 2024).
• Scene Graphs and Hierarchies: Scene graphs $G=(O, R)$ represent objects as nodes (with features, poses, and optional textual descriptors) and relations as edges, supporting both spatial (e.g., "on", "left of") and abstract (e.g., "preferred by Mary") predicates (Chang et al., 2023, Yu et al., 8 Nov 2025, Linok et al., 16 Jul 2025). Hierarchical scene graphs explicitly encode multi-level spatial organization, e.g., building $\to$ floor $M^*$0 room $M^*$1 object layers in OVIGo-3DHSG (Linok et al., 16 Jul 2025).
• Feature Fields and Splatting: Neural and Gaussian fields parameterize the 3D space such that language-aligned features and instance discriminators can be rendered and queried at arbitrary physical scales (Liu et al., 30 Mar 2025, Liu et al., 9 Jul 2025).
• Dynamic and Retrieval-Indexed Chunks: For scalability and multimodal retrieval, features and relations are chunked and stored in vector databases for fast cosine-similarity search against text queries (Yu et al., 8 Nov 2025).
• Fusion of 3D with 2D Semantics: Semantic feature extraction leverages per-instance 2D crops, projected and pooled across views to yield robust descriptors that are co-embedded in CLIP or compatible open-vocabulary feature spaces (Koch et al., 2024, Li et al., 2024).

These representations underpin the model's ability to support querying by unseen words, multi-attribute phrases, and compositional relations.

3. Zero-Shot, Training-Free, and Retrieval-Augmented Pipelines

Recent approaches aim to maximize generalization and practicality by eschewing explicit supervision on 3D language annotations or 3DVG datasets:

• Training-Free Scene Parsing and Candidate Filtering: UniGround (Zhang et al., 9 Mar 2026) segments point clouds into superpoints and instance candidates using VCCS and graph-based merging, without any learned 3DVG detector, and ranks candidates using frozen multimodal encoders and structured VLM reasoning.
• 3D Scene Graph Construction and Retrieval: Open3DSG co-embeds 3D backbone features and 2D vision-LLM features, enabling zero-shot node and relation querying using CLIP and grounded LLM modules (Koch et al., 2024). Retrieval-augmented reasoning further encodes scene-characterizing chunks into a vector database, supporting efficient query-to-instance mapping (Yu et al., 8 Nov 2025).
• LLM-Guided Reasoning and Visual Programs: Dialog-based and visual programming frameworks prompt LLMs to decompose complex queries into structured plans or subproblems, matching referents via CLIP similarity and deterministic spatial/logical modules (e.g., view-dependent projections, closest/farthest search, relation enforcement) (Yuan et al., 2023, Yang et al., 2023).
• Language-to-3D Grounding and Search: Pipelines combine LLM-driven instruction parsing, region abstraction, region prioritization, and composite scoring (combining detection scores and CLIP fusions) to guide embodied agents and robots (Qiu et al., 2024, Li et al., 3 Aug 2025).
• Fusion of Geometric and Semantic Consensus: For fragile real-world observations, joint consensus constraints enforce instance aggregation only when sufficient geometric and vision-language similarity supports it, boosting the robustness of mapping and downstream grounding (Li et al., 3 Aug 2025).

These zero-shot and modular strategies enable open-vocabulary 3DVG systems to process any referential language and generalize to new entities, relations, and layouts without retraining.

4. Spatial, Relational, and Context-Aware Reasoning

Handling arbitrary queries demands explicit modeling of spatial relations and context:

• Subgraph Matching and Relation Scoring: OVSG frames grounding as subgraph matching between a language-induced query graph and the scene graph, with node and edge distances formulated in language-aligned feature spaces and spatial relations parameterized via learned predictors (Chang et al., 2023).
• Structured Prompting and Reasoning Chains: Structured VLM and LLM prompting parses spatial relations, generates global/candidate-centric visual prompts, and reasons over both appearance and scene layout, often using chain-of-thought or programmatic templates (Zhang et al., 9 Mar 2026, Yuan et al., 2023).
• Affordance and Intention-Guided Integration: GREAT decouples geometric cues (object part, structure) from intention analogies (potential interactions), fusing both via multi-head chain-of-thought prompting and collaborative cross-attention, enabling open-vocabulary affordance grounding on 3D shapes (Shao et al., 2024).
• Handling Implicit and Multi-Hop Reasoning: Hierarchical splatting fields (Liu et al., 30 Mar 2025) and neural property fields (Liu et al., 9 Jul 2025) combine LLM-driven decomposition, multi-instance clustering, and feature field queries to enable reasoning from implicit language and infer occluded/partially visible referents.
• LLM-Assisted Region and Instance Selection: In OpenMap, LLMs parse high-level intent, refine candidate pools with local context (neighbor objects), and use spatial context to disambiguate instances among similar classes (Li et al., 3 Aug 2025).
• Hierarchical Graph Traversal: OVIGo-3DHSG leverages LLM-driven graph programs to traverse floor $M^*$2 room $M^*$3 object hierarchies and apply semantic, spatial, and context filters in concert (Linok et al., 16 Jul 2025).

This class of reasoning frameworks is essential for handling queries with compositional structure, multiple referents, context-dependent relations, and indirect cues.

5. Robotics, Embodiment, and Application Domains

Open-vocabulary 3D visual grounding is increasingly demonstrated in robotics, navigation, embodied agent, and AR contexts:

• Mobile Manipulation: OVMM (Qiu et al., 2024) integrates 3D semantic mapping, open-vocabulary detection, and LLM-driven region reasoning to realize object fetching, navigation, and adaptive replanning with a mobile 10-DoF manipulator. Empirical metrics show 80.95% navigation success, 73.33% overall task success, and large improvements in SFT (Success on First Trial) and SPL (Success weighted by Path Length) over randomized baselines.
• Affordance-Conditioned Grasp Synthesis: HiFi-CS fuses CLIP visual-language features and hierarchical FiLM decoding to ground complex open-vocabulary referring expressions to 2D/3D object segments, guiding grasp pose estimation and enabling 90.33% real-world grounding accuracy for robotic grasping on diverse scenes (Bhat et al., 2024).
• Instruction Following and Navigation: OpenMap achieves zero-shot grounding of arbitrary navigation commands, with robust retrieval rates (49.6% SR) in Matterport3D, enabled by architectural modules for instruction parsing, structural-semantic consensus, and LLM-assisted candidate selection (Li et al., 3 Aug 2025).
• Autonomous Driving and Outdoor Perception: Open3DWorld fuses CLIP-encoded language and BEV LiDAR features, enabling the alignment and localization of arbitrary text-denoted entities (including long-tail rare nouns) directly in large-scale 3D vehicle-centric scenes (Cheng et al., 2024, Vobecky et al., 2024). POP-3D demonstrates competitive language-driven 3D segmentation accuracy without manual 3D labels.
• Augmented Reality and Real-Time Perception: UniGround and SeeGround provide training-free pipelines capable of localizing free-form expressions in live captured or reconstructed scans, supporting AR and embodied agent operation in previously unseen scenes (Zhang et al., 9 Mar 2026, Li et al., 2024). ReasonGrounder shows amodal object localization in occluded views and implicit queries essential for vision-language navigation (Liu et al., 30 Mar 2025).

The modularity and open-world extensibility of these pipelines align with application requirements for scalable, robust, and explainable multimodal grounding.

6. Evaluation, Limitations, and Future Directions

Open-vocabulary 3D visual grounding is evaluated on dedicated benchmarks (ScanRefer, Nr3D, 3DSSG, EmbodiedScan, DOVE-G, PIADv2, Matterport3D, ReasoningGD) with metrics such as accuracy at IoU thresholds, mean IoU, recall/precision for retrieval, and instruction completion rates. Notable experimental results include:

• UniGround: 46.1%/34.1% [email protected]/0.5 on ScanRefer and 28.7% [email protected] on EmbodiedScan (Zhang et al., 9 Mar 2026).
• OVSG: 58.9% (Top-1, ScanNet, IoU_BB$M^*$40.5) and 78.6% (ICL-NUIM, IoU₃D) (Chang et al., 2023).
• HiFi-CS: Exceeds 50% IoU and 90% segmentation accuracy in real-world open-vocabulary grasping (Bhat et al., 2024).
• OpenMap: 14.3 AP / 26.0 AP$M^*$5 in ScanNet200 zero-shot; 49.6% SR in Matterport3D (Li et al., 3 Aug 2025).
• ReasonGrounder: 86.7% localization accuracy and 55.2% implicit IoU (LERF) (Liu et al., 30 Mar 2025).
• ViGiL3D: Leading methods (e.g., PQ3D) achieve only 26.2% accuracy on the linguistically challenging prompts, revealing major generalization gaps (Wang et al., 2 Jan 2025).

Key limitations and future challenges include:

• Low grounding accuracy in open-world, out-of-distribution, or linguistically rich settings, especially in ViGiL3D (Wang et al., 2 Jan 2025).
• Limited spatial reasoning beyond simple relations; specialized modules for counting, negation, ordinal, and multi-object queries remain weak.
• Computational bottlenecks from heavy reliance on large LLMs/VLMs, restricting deployment in low-latency, on-device, or edge settings (Zhang et al., 9 Mar 2026).
• Absence of end-to-end jointly optimized models; most pipelines are modular and heavily depend on frozen backbones and prompt engineering.
• Physical interaction constraints: generalization to dynamic, cluttered, or outdoor 3D environments is nascent (Cheng et al., 2024).
• Negation, coreference, and text-reading remain unsolved, reflected in poor ViGiL3D performance on these phenomena (Wang et al., 2 Jan 2025).

Proposed directions to address these gaps include curriculum training on balanced linguistic phenomena, end-to-end neural field integration, scalable and energy-efficient inference, scene-graph pruning for large-scale settings, and the expansion of diagnostic datasets like ViGiL3D with richer annotations. Collaborative and analogical reasoning, explicit spatial reasoning programs, and tighter fusion of visual and textual affordance cues are promising for advancing open-vocabulary grounding to human-level proficiency.

References: (Qiu et al., 2024, Zhang et al., 9 Mar 2026, Chang et al., 2023, Shao et al., 2024, Yu et al., 8 Nov 2025, Koch et al., 2024, Li et al., 3 Aug 2025, Yang et al., 2023, Yuan et al., 2023, Cheng et al., 2024, Liu et al., 30 Mar 2025, Liu et al., 9 Jul 2025, Linok et al., 16 Jul 2025, Li et al., 2024, Prabhudesai et al., 2019, Bhat et al., 2024, Vobecky et al., 2024, Wang et al., 2 Jan 2025)