---
title: Embodied Reference Understanding
url: https://www.emergentmind.com/topics/embodied-reference-understanding
type: topic
---

# Embodied Reference Understanding

Searching arXiv for recent papers on embodied reference understanding and related benchmarks.
Embodied Reference Understanding (ERU) is the task of identifying a referent object from multimodal human communicative behavior in a shared environment, typically by localizing a bounding box or selecting an object using language together with embodied nonverbal cues such as pointing, gaze, posture, body orientation, and perspective-dependent spatial expressions. In the literature summarized here, ERU is distinguished from conventional referring expression comprehension by the requirement that the receiver interpret reference under embodiment, often from a viewpoint different from the sender’s, and by the central role of perspective-taking, gesture interpretation, and spatial grounding in physically situated human-robot interaction [2109.03413][2512.06558].

## 1. Conceptual scope and task definition

ERU has been formulated most directly as localizing a referent object using both language and embodied gestural signals, especially pointing, in an image or video frame [2210.05668]. In the YouRefIt formulation, one agent uses both language and gesture to refer to an object to another agent in a shared physical environment, and the task requires understanding multimodal cues with perspective-taking to identify which object is being referred to [2109.03413]. In later work, this definition is broadened to embodied referring expression comprehension (E-RFE), where robots must understand how humans refer to objects not only through words but also through gestures, gaze, posture, and perspective-dependent language [2512.06558].

The literature consistently treats ERU as harder than standard referring expression comprehension. Language alone is often ambiguous, and gesture alone is also incomplete; the two modalities are complementary, with language helping specify object type or semantic attributes and gesture helping select the intended instance among candidates [2210.05668]. This multimodal dependence becomes more acute when instructions are viewpoint-sensitive, such as “the one on your left,” or when nonverbal behavior provides the only reliable disambiguating signal [2512.06558]. A closely related formulation in multi-agent embodied environments defines reference as a communicative game between a speaker and a listener with different visual perspectives, where communicative success depends on whether the listener selects the intended target from language generated under viewpoint mismatch [2410.03959].

Several adjacent research threads sharpen the scope of ERU. Embodied spatial understanding benchmarks treat egocentric interpretation of relations such as left, right, above, below, close, and far as a prerequisite for reliable reference grounding and referential comprehension in embodied AI systems [2406.05756]. Position papers in embodied cognition further suggest that reference resolution should be understood as part of a stateful mental-model construction process grounded in simulation and embodied schemata rather than as isolated entity linking [2005.00311]. This suggests that ERU lies at the intersection of multimodal grounding, spatial reasoning, perspective-taking, and physically situated communication.

## 2. Datasets and benchmarks

The empirical study of ERU has been organized around a sequence of datasets that progressively expand realism, modality coverage, dimensionality, and interaction context.

YouRefIt introduced embodied reference understanding with language and gesture as a crowd-sourced dataset of videos in indoor physical scenes. It contains **4,195 unique reference clips**, collected from **432 indoor scenes**, with **395 object categories**, **497,348 annotated frames**, **8.83 hours** of video retrieved in post-processing, **3.35 hours** of actual reference actions, **average clip duration: 2.81 seconds**, and **average sentence length: 3.73 words** [2109.03413]. The dataset supports two benchmarks: image ERU, using canonical frames plus sentences, and video ERU, which additionally requires canonical frame detection [2109.03413].

ScanERU extends ERU into 3D visual grounding. It is described as the first dataset to cover semi-synthetic scene integration with textual, real-world visual, and synthetic gestural information [2303.13186]. Built on **706 unique indoor scenes**, it contains **9,929 referred objects** and **46,173 descriptions**, with **41,034** descriptions mentioning object attributes; frequent linguistic cues include **spatial terms: 98.7%**, **color terms: 74.7%**, **shape terms: 64.9%**, and **size terms: 14.2%** [2303.13186].

Refer360 was introduced to address limitations identified in earlier datasets, specifically perspective bias, single-view capture, weak coverage of nonverbal behavior, and overreliance on indoor scenes [2512.06558]. It records embodied verbal and nonverbal interactions from egocentric and exocentric viewpoints and includes depth, infrared, audio, and 3D skeletal data, with gaze captured through a Pupil Invisible eye tracker and exocentric RGB/depth/IR/audio/skeleton streams captured via an Azure Kinect DK mounted on an Ohmni telepresence robot [2512.06558]. The reported scale is **66 participants**, **392 sessions**, **13,990 interactions**, about **3.2 million synchronized frames**, **28,736 canonical frames**, and **17.62 hours** of recordings, spanning lab and outside-lab environments including homes and outdoor locations; **96.97%** of participants reported using a combination of language and gesture [2512.06558].

Several benchmarks target specific subproblems central to ERU. EmbSpatial-Bench evaluates embodied spatial understanding of LVLMs from the agent’s egocentric perspective, using **3,640 QA pairs**, **2,181 images**, **294 object categories**, and **277 scenes**, derived from Matterport3D, AI2-THOR, and ScanNet [2406.05756]. A multi-perspective referential communication benchmark introduces **27,504 generated scenes**, split into **24,644 train**, **1,485 validation**, and **1,375 test** scenes, together with a human-labeled subset of **1,485 scenes** and **2,970 human-written referring expressions**, each paired with three listener judgments [2410.03959]. SIGAR, built on YouRefIt for Multi-ref EC, contributes **4,195 images**, **20,193 state-intention expression texts**, an **average expression length: 17.1**, and **395 categories**, explicitly combining indirect state expressions, human intention expressions, and embodied gesture expressions [2503.19240]. ImputeRefer, generated by the Imputer framework for 3D-ERU, is characterized relative to ScanRefer-based text by shorter regenerated expressions, with **average words per instruction in ScanRefer-based text: 19.18** and **average words in ImputeRefer regenerated text: 10.52**, and by a larger average human-to-target distance of **~2.31 m** versus **~1.24 m** in the ScanERU subset [2504.09623].

The following table summarizes major ERU-relevant datasets and benchmarks.

| Dataset / benchmark | Core setting | Reported scale |
|---|---|---|
| YouRefIt [2109.03413] | Indoor video ERU with language and gesture | 4,195 clips; 432 scenes; 395 categories |
| ScanERU [2303.13186] | Semi-synthetic 3D ERU with text and synthetic gesture | 706 scenes; 9,929 objects; 46,173 descriptions |
| Refer360 [2512.06558] | Multi-view, multimodal indoor/outdoor E-RFE | 66 participants; 392 sessions; 13,990 interactions |
| EmbSpatial-Bench [2406.05756] | Egocentric embodied spatial QA | 3,640 QA pairs; 2,181 images |
| SIGAR [2503.19240] | Multi-attribute state-intention-gesture grounding | 4,195 images; 20,193 texts |
| Multi-perspective reference game [2410.03959] | Speaker-listener communication under view mismatch | 27,504 generated scenes; 2,970 human expressions |

Taken together, these resources indicate a shift from static indoor image grounding toward multimodal, multi-view, 3D, temporally extended, and communicative settings. A plausible implication is that benchmark design has become a principal instrument for isolating distinct failure modes in ERU: gesture misinterpretation, egocentric spatial confusion, perspective mismatch, and insufficient persistence over time.

## 3. Geometric, spatial, and perspective-taking foundations

A recurrent finding is that embodied reference cannot be reduced to generic multimodal fusion; it depends on specific geometric and perspectival structure. Early work on Touch-Line Transformer argues that a common assumption in gesture interpretation—the elbow-wrist line—is a misconception, and that the more informative cue is the virtual touch line (VTL), defined as the line from the eye to the fingertip [2210.05668]. The geometric consistency objective explicitly measures cosine similarity between the eye-to-fingertip direction and the eye-to-object-center direction, encouraging predicted referents to lie on or near the touch line [2210.05668]. Computationally, VTL was shown to outperform the elbow-wrist line on YouRefIt [2210.05668].

Subsequent work refines the geometric model rather than discarding it. AD-DINO introduces the Attention-Dynamic Touch Line (ADTL), in which the attention source is not fixed but switches dynamically between the eye for distant interactions and the MCP joint for close interactions, based on upper-limb configuration and a threshold \(t = 0.95\) over cosine similarity [2411.08451]. The central claim is that the meaning of a pointing gesture changes with pointer–referent distance, so a single fixed line is inadequate [2411.08451]. CAPE similarly rejects a single-line assumption, training separate models using head-to-fingertip and wrist-to-fingertip Gaussian ray heatmaps and combining them with a CLIP-aware ensemble [2507.21888]. This line of work suggests that embodied gesture understanding is sensitive not merely to pose detection but to the choice of geometric proxy.

Perspective-taking has emerged as a second foundational theme. REP frames ERU as requiring the receiver to access spatial and visual information relative to the sender, then proposes a sender-centric embodied 3D coordinate system, a body language vector, spatial attention, gesture attention, and verbal fusion [2309.01073]. The key operation is a view rotation that first places the sender at the origin of a 3D coordinate map and then conditions reasoning on sender orientation and gesture [2309.01073]. The emphasis is not only that objects must be localized, but that the receiver must infer how objects are oriented around and seen from the sender.

Egocentric spatial reasoning is also isolated by EmbSpatial-Bench, which insists that spatial relationships “should be described from the egocentric perspective” because agents take themselves as the center of coordinates in embodied tasks [2406.05756]. The benchmark covers exactly six egocentric relations—**above**, **below**, **left**, **right**, **close**, and **far**—and extracts them directly from well-annotated 3D datasets rather than object detectors [2406.05756]. In a text-only abstraction of the same problem, GSU shows that models often struggle when left/right/forward/backward must be updated relative to a moving embodied agent rather than a fixed Cartesian frame [2603.17333].

Multi-agent communication work makes the same issue explicit in interactive form. In photorealistic 3D indoor environments, successful reference generation and comprehension depend on distinct observations \(o_s\) and \(o_l\) for speaker and listener, and communicative success degrades sharply as view overlap decreases and relative orientation becomes more adversarial [2410.03959]. This suggests that spatial language in ERU is inherently relational: not only between agent and object, but between the frames of reference of multiple agents.

## 4. Model families and algorithmic strategies

Method development in ERU has proceeded along several partially overlapping lines: gesture-line modeling, multimodal fusion, 3D grounding, residual adaptation of pretrained vision-language systems, and persistent world modeling.

The Touch-Line Transformer established a joint prediction paradigm in which a transformer predicts both the referent bounding box and a touch-line vector, using a multimodal encoder over ResNet visual tokens and BERT/RoBERTa textual tokens, followed by decoder queries for objects and gestural keypoints [2210.05668]. AD-DINO builds on a Grounding-DINO-style architecture with Swin Transformer, BERT, bidirectional cross-modality fusion, a query selection module, and a decoder that jointly predicts the target object box and the attention source of the gesture, with fingertip detections provided by MediaPipe Handmarker [2411.08451]. CAPE retains a transformer encoder-decoder backbone but adds a dedicated heatmap encoder, Gaussian ray heatmaps, an auxiliary object center prediction head, and a size-aware hybrid ensemble using CLIP similarity and model confidence [2507.21888].

A second family emphasizes explicit perspective modeling. REP adds monocular depth estimation, sender segmentation, a sender-centric 3D coordinate transformation, and staged reasoning through spatial attention, gesture attention, and FiLM-based verbal fusion [2309.01073]. The method is one-stage in output but structurally decomposed into receiver–sender and sender–object relation modeling [2309.01073]. A plausible interpretation is that REP treats perspective-taking as an architectural prior rather than as latent information to be discovered by generic fusion.

A third family generalizes ERU into 3D grounding. ScanERU uses PointNet++ for scene proposals and gesture encoding, GloVe plus GRU for language, and transformer-based multimodal fusion over proposal and gesture features [2303.13186]. Ges3ViG uses PointGroup for 3D instance segmentation, CLIP image and text features, joint feature extraction for human localization, early fusion of gesture and language, and late fusion through a pointing alignment score computed from shoulder, fingertip, and proposal center vectors [2504.09623]. ShapeLLM approaches embodied grounding from the opposite direction: it grounds language in 3D point clouds using a ReCon++ encoder with multi-view image distillation, instruction tuning, and a 3D multimodal LLM architecture with local, global, and absolute position features [2402.17766].

A fourth line adapts large pretrained multimodal models rather than replacing them. MuRes is presented as a multimodal guided residual module inserted into frozen pretrained backbones such as CLIP, DualEncoder, ViLT, or BLIP-2 [2512.06558]. Instead of passing the full residual pathway, MuRes uses projected representations as queries and original features as keys and values in a cross-attention block,
$$
\{V^g, L^g\} = \text{Cross-Attention}(q=\{V^p,L^p\},\ k=\{V,L\},\ v=\{V,L\}),
$$
then forms fused representations by residual addition,
$$
V^f, L^f = V^p + V^g,\ L^p + L^g.
$$
Its role is described as an information bottleneck that extracts salient modality-specific signals and reinforces them into pretrained representations [2512.06558].

Finally, persistent memory architectures connect ERU to long-horizon dynamic scene understanding. Embodied VideoAgent constructs a persistent object memory from egocentric video together with depth and pose sensing, tracks object states such as \(\{\text{open}, \text{close}, \text{in hand}, \text{normal}\}\), and updates object entries through re-identification and VLM-based action-target association [2501.00358]. SNOW organizes RGB images and point clouds into STEP tokens and a globally aligned 4D Scene Graph (4DSG), making object identity, geometry, and temporal history directly queryable by VLMs [2512.16461]. These systems do not target canonical ERU benchmarks directly, but they contribute a persistent referential substrate for expressions such as “the car that moved most” or “the object in front of the fridge” [2512.16461].

## 5. Empirical findings

Across datasets and formulations, the dominant empirical result is that current multimodal models underperform humans and frequently fail on the embodied aspects of reference.

On YouRefIt image ERU, the original full multimodal benchmark model achieved **54.7** at IoU **0.25**, **40.5** at IoU **0.5**, and **14.0** at IoU **0.75**, while human performance was **94.2 ± 0.2**, **85.8 ± 1.4**, and **53.3 ± 4.9** respectively [2109.03413]. Gesture-only models were informative but insufficient, and masking out humans via inpainting degraded performance, supporting the claim that gestures are as critical as language cues [2109.03413].

Touch-Line Transformer improved YouRefIt substantially, reaching **71.1%** at IoU **0.25**, **63.5%** at IoU **0.50**, and **39.0%** at IoU **0.75**, corresponding to gains of **+16.4**, **+23.0**, and **+25.0** over the prior YouRefIt full model; at IoU **0.75**, this closed **63.6%** of the model-human gap [2210.05668]. AD-DINO later reported **76.3%** at IoU **0.25**, **72.4%** at IoU **0.5**, and **55.4%** at IoU **0.75**, with the paper noting that this surpasses human performance at the strict IoU **0.75** criterion for the first time on this benchmark [2411.08451]. CAPE achieved **75.0** mAP@0.25, **65.4** mAP@0.50, and **35.7** mAP@0.75, improving especially at IoU **0.25** and **0.50**, while DA-ERU reported **78.7**, **67.6**, and **38.1** mAP on YouRefIt and **62.7**, **50.1**, and **30.1** on the unseen ISL pointing dataset [2507.21888][2510.08278].

Perspective-aware modeling also produced measurable gains. REP surpassed the previous state of the art on YouRefIt by **+5.22%** absolute accuracy in terms of Prec@0.5, with its 3D view rotation module alone improving the baseline by **2.8% / 3.4% / 5.1%** on Prec@0.25 / 0.50 / 0.75 [2309.01073]. In 3D, Ges3ViG achieved **84.60 / 71.03** on the unique split, **67.57 / 55.77** on the multiple split, and **70.85 / 58.71** overall at IoU@0.25 / IoU@0.5 on ImputeRefer, exceeding the gesture-based ScanERU baseline by **29.87** points overall at IoU@0.5 and surpassing the best language-only 3D grounding baseline M3DRefCLIP by **8.93** points overall [2504.09623]. ScanERU’s own ablation already showed the same qualitative pattern: **Gesture-only: 17.46 / 13.44**, **Language-only: 49.18 / 35.62**, **Full model: 54.45 / 40.94** on validation Acc@0.25 / Acc@0.5 [2303.13186].

Benchmark studies expose broader weaknesses. EmbSpatial-Bench found zero-shot human performance at **90.33%**, while the best reported generation-based score was **Qwen-VL-Max: 49.11%**, the best open-source likelihood score was **MiniGPT-v2: 43.85%**, and **GPT-4V: 36.07%** under generation [2406.05756]. Fine-tuning MiniGPT-v2 on EmbSpatial-SFT improved overall accuracy from **23.93%** to **32.97%** under generation and from **43.85%** to **78.10%** under likelihood, but generation-based performance remained far below human level [2406.05756]. In multi-perspective referential communication, **human-human pairs** achieved **87.6%** communicative success on average, while the best human-speaker/automated-listener setting reached **69.2%** and GPT-4o as automated speaker with human listeners reached **64.9%** [2410.03959]. Multi-ref EC on SIGAR similarly found that multi-attribute references outperform single-attribute ones, with **Qwen-VL** reaching **54.4**, **51.9**, and **32.3** at IoU 0.25 / 0.5 / 0.75, and with **intention + gesture** emerging as the strongest dual-attribute combination at **53.2 / 47.3 / 28.1** [2503.19240].

The cumulative pattern is not that ERU is uniformly unsolved, but that performance is highly sensitive to which component is stressed: strict box precision, long-distance pointing, egocentric depth reasoning, multi-agent perspective mismatch, or dynamic scene persistence.

## 6. Recurrent limitations, controversies, and research directions

Several limitations recur across the literature. First, many methods focus primarily on pointing gestures and only partially model other nonverbal cues. Touch-Line Transformer explicitly notes that it does not fully model richer gaze direction, finger orientation, or lower-limb orientation [2210.05668]. AD-DINO likewise states that the current framework does not fully integrate gaze or richer hand gestures [2411.08451]. Refer360 was created precisely because existing datasets showed weak coverage of nonverbal behavior and indoor-centric bias [2512.06558].

Second, reference-frame ambiguity remains a major source of error. EmbSpatial-Bench reports especially weak performance on depth relations **close** and **far**, attributing this to lack of depth-estimation training and more complex reasoning demands [2406.05756]. GSU shows that many models correctly compute a new heading and then fail to use it, reverting to a fixed Cartesian interpretation in longer egocentric sequences [2603.17333]. Multi-perspective communication experiments similarly find that automated models underuse listener- and speaker-perspective references and instead over-rely on object-relative descriptions [2410.03959].

Third, realism and scalability in data remain contested. ScanERU and ImputeRefer both use synthetic or inserted human gestures in otherwise real 3D scenes [2303.13186][2504.09623]. SIGAR depends on Claude-3.5-Sonnet-generated state and intention annotations that are manually filtered [2503.19240]. This does not invalidate the datasets, but it does indicate that large-scale embodied annotation is still expensive and methodologically heterogeneous. Refer360’s multi-view, indoor-and-outdoor recording setup is one response to this limitation, and it shifts the field toward more natural embodied interaction data [2512.06558].

Fourth, many successful methods remain static or single-turn. The multi-perspective reference game is explicitly single-shot, without clarification dialogue [2410.03959]. SIGAR does not model temporal interaction or sequential feedback [2503.19240]. By contrast, SNOW and Embodied VideoAgent suggest that persistent, world-aligned memory may be necessary for resolving reference in dynamic environments over time [2512.16461][2501.00358]. This suggests a convergence between ERU and broader embodied world modeling.

Finally, there is an unresolved modeling question concerning how much structure should be built in. Gesture-line methods, body-centric coordinate systems, residual bottlenecks, and 4D scene graphs all encode strong priors. Position papers grounded in embodied cognition and image schemas argue that such structure may be necessary for interpretability, efficiency, and human-agent alignment [2005.00311][2503.24110]. A plausible implication is that future ERU systems may combine foundation-model backbones with explicit representations for perspective, geometry, temporal persistence, and embodied conceptual structure rather than relying on scale alone.

In that sense, embodied reference understanding has evolved from a narrowly defined multimodal grounding task into a broader research program on how agents interpret situated human intent. The most concrete trajectory in the current literature is clear: richer datasets that capture natural embodied communication, stronger mechanisms for egocentric and multi-agent perspective-taking, better integration of gesture and language, and increasingly persistent object-centered world models for reference over time [2512.06558][2512.16461].

Source: https://www.emergentmind.com/topics/embodied-reference-understanding