Object-Directed Grounding

• Object-directed grounding is the process of mapping language to specific object regions in 2D, 3D, and video scenes, enabling clear scene interpretation and interaction.
• Techniques such as scene graph matching, two-stage neural filtering, and cross-modal transformers disambiguate visual and linguistic cues through spatial and temporal reasoning.
• Advancements boost human-robot interaction and vision-language tasks with improved accuracy, though challenges remain in occlusion, detection errors, and ambiguous expressions.

Object-directed grounding is the process of mapping language referring to specific entities in a sensory scene—whether visual, 3D, or multi-modal—onto explicit selections or regions corresponding to real-world objects. This task underpins scene understanding, human-robot interaction, manipulation, multi-modal activity recognition, and vision-language reasoning. The core challenge is to disambiguate object referents from unconstrained, potentially ambiguous, natural language, leveraging scene context, object relations, and semantic, spatial, and/or temporal attributes.

1. Formal Problem Definitions and Scene Representations

Object-directed grounding is rigorously formulated across various perceptual settings (2D images, RGB-D, 3D point clouds, video) and modalities (static, temporal, relational):

• 2D Scene Graph-Based Grounding: Given an image $I$, a referring expression $C$, and automatically extracted scene graphs $(G^I, G^L)$ from the image and language, the grounding objective is to identify a single object node $v^* \in V^I$, ideally maximizing the posterior $P(v \mid C, G^I, G^L)$. Matching proceeds through incremental alignment between linguistic and visual relations, followed by contextual disambiguation (Yi et al., 2022).
• Two-Stage Neural Grounding: With unconstrained image-language inputs, the first stage semantically filters region proposals to match object descriptions, while the second uses pairwise models to resolve spatial and relational ambiguity among candidates. This maximizes either $P(o|r,I)$ or, equivalently, $P(r|o,I)$ over the set of proposals $O$ after a decomposition into semantic and spatial components (Shridhar et al., 2017).
• Video/Object-Tracking Grounding: In the video case, grounding extends to tracklet outputs $\hat T_o = \{ \hat B^o_t \}_{t=1}^T$ maximizing a fused semantic-distance score or multi-modal similarity, often using contextual or interaction cues, e.g., human-object proximity, framewise motion, or temporal self-attention (Liu et al., 2024, Yu et al., 25 Dec 2025).
• 3D and RGB-D Grounding: The task requires mapping language to 3D bounding boxes or per-point activations (for affordances) in scene coordinates $(x, y, z, w, l, h)$ or as heatmaps over segmented regions or point clouds. Spatial and relational information is injected at the voxel, point, or proposal level (Liu et al., 2021, Zhang et al., 5 Jun 2025, Tziafas et al., 2024, Yu et al., 9 Sep 2025, Zhu et al., 7 Apr 2025, Wang et al., 3 Aug 2025).
• Egocentric and Task-Oriented Video: Here, grounding is tightly coupled to the functional roles of objects as dictated by tasks, often involving multi-object, explicit and implicit, temporally localized references, and is evaluated over both single and multiple entity intervals in time and 2D bounding box space (Xu et al., 3 Dec 2025, Feng et al., 7 May 2025).

2. Architectures and Reasoning Strategies

Scene Graphs and Edge-Matching

Methods such as IGSG perform incremental edge pruning between language and image scene graphs, matching first by object, then by subject, predicate, and finally by attributes, with backtracking and natural-language question generation when ambiguity (i.e., candidate set $C$0) remains (Yi et al., 2022). Object relations are explicitly modeled as triplets and used to resolve many-to-one language-intent mappings.

Neural Captioning and Semantic Filtering

In the unconstrained regime, open-vocabulary captioning models are inverted to compute region likelihoods, with semantic filtering via LSTM LLMs trained on large datasets (e.g., Visual Genome), followed by spatial disambiguation using context-pairwise models (max rule, noisy-OR) (Shridhar et al., 2017).

Attention-based Fusion and Cross-Modal Transformers

Modern approaches use bi-directional attention or cross-attention between vision, language, and (optionally) depth features:

• Bi-Aligner (OGRG): Fuses visual, language, and early-fused depth information over four stages, applying cross-attention both from vision-to-language and language-to-vision, with a gated fusion step for grounding masks and grasp predictions. Depth cues are integrated at the feature level for resolving “left/right/above/below” queries and enhancing geometric discrimination among duplicates (Yu et al., 9 Sep 2025).
• Vision-Language-Action Pipelines: Object-centric perception modules segment and select candidate regions conditioned on the instruction tokenization, using instance segmentation followed by LLM prompts to select relevant regions. Geometry-aware grounding incorporates masked RGB and dense depth, providing robust input to downstream manipulation policies (Vo et al., 27 Dec 2025).
• Slot-Based Grounding: NSI uses slot attention to bind compositional program primitives (XML-like visXML element encodings) into image slots, with contrastive objectives anchoring semantic abstractions to emergent spatial representations (Dedhia et al., 2024).

Temporal and Hierarchical Reasoning

• Temporal Multimodal 3D (TrackTeller): Integrates LiDAR–image fusion, language-conditioned decoding, and temporal memory to support grounding that depends on recent object motion or interaction (behavior-dependent referencing), achieving large reductions in false alarm rates in dynamic scenes (Yu et al., 25 Dec 2025).
• Hierarchical and Multi-Stage Processing: Hierarchical contrastive transformers (H-COST) progressively refine multi-object localization, using shrinking distance thresholds in each stage, with contrastive alignment to auxiliary networks built on perfect semantic features (Du et al., 14 Apr 2025).
• Task-Oriented Grounding: Benchmarks such as ToG-Bench require joint temporal and spatial localization of all objects relevant to a task, including implicitly referenced ones, with evaluation assessing per-task and per-object precision and recall (Xu et al., 3 Dec 2025).

3. Disambiguation, Interaction, and Active Clarification

Explicit handling of linguistic and visual ambiguity is tackled via:

• Incremental Dialog (IGSG, DoRO): At any ambiguity point, the system computes a minimal, maximally discriminative query (e.g., “Do you mean the [object] [relation] [neighbor]?”) based on the least frequent relation among candidate object sets, and prunes or selects according to user response. This interactive protocol robustly resolves ambiguous, vague, or erroneous referring expressions and substantially improves accuracy over holistic or LSTM-based baselines (Yi et al., 2022, Pramanick et al., 2022).
• Graph-Based Discrimination: In exploration scenarios, DoRO formalizes both attribute and relational ambiguity, constructing rooted “object graphs” from language and grounded scene observations, detecting missing, ambiguous, or mismatched referents, and generating disambiguating queries using filled templates (Pramanick et al., 2022).
• Contrastive Sampling in Video: Semantic-role or object-level contrastive samples force grounding networks to distinguish among similar scenes/roles, with entity roles aligned across sampled temporal or spatial concatenations, penalizing solutions that ignore inter-object or inter-scene relations (Sadhu et al., 2020).

4. Synthesis, Data, and Generalization

Synthetic Data Pipelines for Grounding

The SOS pipeline synthesizes large-scale object-centric datasets by compositing high-quality segments into images with structured priors (matching real-world object-count, size, and overlap distributions), and generates dense referring expressions using LLMs conditioned on object attributes, spatial configuration, and ground-truth masks. This enables controllable evaluation in intra-class and ambiguous entity settings, yielding larger per-example grounding improvements than scaling real image data (Huang et al., 10 Oct 2025).

Multi-View and 3D Feature Distillation

Multi-view distillation strategies eliminate uninformative views via semantic informativeness scores and object-level fusion of 2D CLIP features, yielding view-independent 3D encoders whose per-point embeddings align with natural language queries for open-vocabulary segmentation and grasping. Object priors ensure crisp spatial boundary transfer from multi-view to single-view encoding, outperforming naïve pixel- or view-pooling (Tziafas et al., 2024).

Affordance Grounding

Frameworks such as DAG and LMAffordance3D leverage frozen diffusion model U-Nets or VLMs to encode affordance priors either as dense heatmaps or via cross-modal attention blocks, enabling open-vocabulary, sample-efficient mapping from interaction-language to 3D object affordance masks. These maintain performance on both seen and unseen affordances, and support rapid adaptation to novel objects (Zhu et al., 7 Apr 2025, Wang et al., 3 Aug 2025).

5. Evaluation, Experimental Findings, and Limitations

Empirical studies consistently show:

• Graph-based, dialog, and scene-relation methods (e.g., IGSG) achieve higher accuracy in ambiguous or multi-instance scenarios. IGSG, for example, improves vague-command success from 42.3% to 88.5% and reduces many classes of false reference via interaction (Yi et al., 2022).
• Multi-modal fusion and early/spatial depth integration (OGRG, OBEYED-VLA) substantially improve grounding and manipulation performance in cluttered scenes, with OGRG mIoU reaching 95.6% and real-robot grasp accuracy jumping from 62.5% to 70.8% using full fusion (Yu et al., 9 Sep 2025, Vo et al., 27 Dec 2025).
• Synthetic data augmentation (SOS) yields outsized benefits for both object discovery and fine-grained intra-class discrimination, e.g., +8.4 $C$1 on gRefCOCO (Huang et al., 10 Oct 2025).
• Temporal-object interaction and attention-tracking (TrackTeller, VOGNet) are critical for locating behavior-dependent or relationally referenced objects in dynamic scenes, with TrackTeller reducing false alarms by 3–4× over prior art (Yu et al., 25 Dec 2025, Sadhu et al., 2020).
• Remaining limitations include fragility to detection errors, incomplete scene-graph construction, limited robustness to occlusion and small objects, and dependence on comprehensive attribute/affordance vocabularies. Systematic failures occur with ambiguous instructions that transcend training ontologies, global positional terms, or rare/novel categories (Yi et al., 2022, Zhu et al., 7 Apr 2025, Huang et al., 10 Oct 2025, Tziafas et al., 2024).

6. Future Challenges and Research Directions

Prominent open avenues include:

• End-to-end, context-aware transformer grounding: Bridging detection and grounding into fully integrated, trainable graph networks or multi-scale attention models, with joint optimization of recognition, spatial, and temporal localization (Zhang et al., 5 Jun 2025, Xu et al., 3 Dec 2025).
• Interactive and task-driven benchmarking: Extension to intent-inferred, multi-object, or one-to-many grounding, requiring models to handle implicit references, commonsense affordances, and long-range dependencies, as formalized in ToG-Bench (Xu et al., 3 Dec 2025).
• Enhancing 3D and affordance reasoning: Advancing data-fusion for partial or occluded geometry, handling articulated or deformable objects, and moving toward real-time closed-loop deployment in manipulation and embodied settings (Zhu et al., 7 Apr 2025, Wang et al., 3 Aug 2025, Vo et al., 27 Dec 2025, Liu et al., 2021).
• Compositional and interpretable slot/object representations: Further abstraction using schema-driven or slot-based factorization, supporting efficient generalization to complex scenes with variable object counts and attributes (Dedhia et al., 2024).
• Augmenting and specializing synthetic data: Continued development of controllable, attribute-rich synthetic data regimes to address targeted weaknesses (rare categories, intra-class failures, etc.) and rapidly bootstrap new application domains (Huang et al., 10 Oct 2025).

Object-directed grounding thus forms the substrate of a broad swath of foundational, applied, and emergent research at the intersection of perception, language, spatial reasoning, and context-aware interaction, with ongoing work leveraging advances in large-scale models, dataset synthesis, and multimodal fusion architectures to approach the challenges of robust, generalizable, and semantic-rich object referencing.