Embodied Reference Understanding
- Embodied Reference Understanding is a multimodal task that integrates language, gesture, and spatial cues for accurate referent identification in shared spaces.
- It distinguishes itself by requiring perspective-taking and geometric modeling to resolve ambiguous references from different viewpoints.
- Benchmarks like YouRefIt, ScanERU, and Refer360 highlight challenges in gesture interpretation and egocentric spatial reasoning in dynamic environments.
Searching arXiv for papers on embodied reference understanding and related benchmarks. Embodied Reference Understanding (ERU) is the task of identifying a referent object from multimodal human communicative behavior in a shared environment, typically by localizing a bounding box or selecting an object using language together with embodied nonverbal cues such as pointing, gaze, posture, body orientation, and perspective-dependent spatial expressions. In the literature summarized here, ERU is distinguished from conventional referring expression comprehension by the requirement that the receiver interpret reference under embodiment, often from a viewpoint different from the sender’s, and by the central role of perspective-taking, gesture interpretation, and spatial grounding in physically situated human-robot interaction (Chen et al., 2021, Islam et al., 6 Dec 2025).
1. Conceptual scope and task definition
ERU has been formulated most directly as localizing a referent object using both language and embodied gestural signals, especially pointing, in an image or video frame (Li et al., 2022). In the YouRefIt formulation, one agent uses both language and gesture to refer to an object to another agent in a shared physical environment, and the task requires understanding multimodal cues with perspective-taking to identify which object is being referred to (Chen et al., 2021). In later work, this definition is broadened to embodied referring expression comprehension (E-RFE), where robots must understand how humans refer to objects not only through words but also through gestures, gaze, posture, and perspective-dependent language (Islam et al., 6 Dec 2025).
The literature consistently treats ERU as harder than standard referring expression comprehension. Language alone is often ambiguous, and gesture alone is also incomplete; the two modalities are complementary, with language helping specify object type or semantic attributes and gesture helping select the intended instance among candidates (Li et al., 2022). This multimodal dependence becomes more acute when instructions are viewpoint-sensitive, such as “the one on your left,” or when nonverbal behavior provides the only reliable disambiguating signal (Islam et al., 6 Dec 2025). A closely related formulation in multi-agent embodied environments defines reference as a communicative game between a speaker and a listener with different visual perspectives, where communicative success depends on whether the listener selects the intended target from language generated under viewpoint mismatch (Tang et al., 2024).
Several adjacent research threads sharpen the scope of ERU. Embodied spatial understanding benchmarks treat egocentric interpretation of relations such as left, right, above, below, close, and far as a prerequisite for reliable reference grounding and referential comprehension in embodied AI systems (Du et al., 2024). Position papers in embodied cognition further suggest that reference resolution should be understood as part of a stateful mental-model construction process grounded in simulation and embodied schemata rather than as isolated entity linking (Tamari et al., 2020). This suggests that ERU lies at the intersection of multimodal grounding, spatial reasoning, perspective-taking, and physically situated communication.
2. Datasets and benchmarks
The empirical study of ERU has been organized around a sequence of datasets that progressively expand realism, modality coverage, dimensionality, and interaction context.
YouRefIt introduced embodied reference understanding with language and gesture as a crowd-sourced dataset of videos in indoor physical scenes. It contains 4,195 unique reference clips, collected from 432 indoor scenes, with 395 object categories, 497,348 annotated frames, 8.83 hours of video retrieved in post-processing, 3.35 hours of actual reference actions, average clip duration: 2.81 seconds, and average sentence length: 3.73 words (Chen et al., 2021). The dataset supports two benchmarks: image ERU, using canonical frames plus sentences, and video ERU, which additionally requires canonical frame detection (Chen et al., 2021).
ScanERU extends ERU into 3D visual grounding. It is described as the first dataset to cover semi-synthetic scene integration with textual, real-world visual, and synthetic gestural information (Lu et al., 2023). Built on 706 unique indoor scenes, it contains 9,929 referred objects and 46,173 descriptions, with 41,034 descriptions mentioning object attributes; frequent linguistic cues include spatial terms: 98.7%, color terms: 74.7%, shape terms: 64.9%, and size terms: 14.2% (Lu et al., 2023).
Refer360 was introduced to address limitations identified in earlier datasets, specifically perspective bias, single-view capture, weak coverage of nonverbal behavior, and overreliance on indoor scenes (Islam et al., 6 Dec 2025). It records embodied verbal and nonverbal interactions from egocentric and exocentric viewpoints and includes depth, infrared, audio, and 3D skeletal data, with gaze captured through a Pupil Invisible eye tracker and exocentric RGB/depth/IR/audio/skeleton streams captured via an Azure Kinect DK mounted on an Ohmni telepresence robot (Islam et al., 6 Dec 2025). The reported scale is 66 participants, 392 sessions, 13,990 interactions, about 3.2 million synchronized frames, 28,736 canonical frames, and 17.62 hours of recordings, spanning lab and outside-lab environments including homes and outdoor locations; 96.97% of participants reported using a combination of language and gesture (Islam et al., 6 Dec 2025).
Several benchmarks target specific subproblems central to ERU. EmbSpatial-Bench evaluates embodied spatial understanding of LVLMs from the agent’s egocentric perspective, using 3,640 QA pairs, 2,181 images, 294 object categories, and 277 scenes, derived from Matterport3D, AI2-THOR, and ScanNet (Du et al., 2024). A multi-perspective referential communication benchmark introduces 27,504 generated scenes, split into 24,644 train, 1,485 validation, and 1,375 test scenes, together with a human-labeled subset of 1,485 scenes and 2,970 human-written referring expressions, each paired with three listener judgments (Tang et al., 2024). SIGAR, built on YouRefIt for Multi-ref EC, contributes 4,195 images, 20,193 state-intention expression texts, an average expression length: 17.1, and 395 categories, explicitly combining indirect state expressions, human intention expressions, and embodied gesture expressions (Guo et al., 25 Mar 2025). ImputeRefer, generated by the Imputer framework for 3D-ERU, is characterized relative to ScanRefer-based text by shorter regenerated expressions, with average words per instruction in ScanRefer-based text: 19.18 and average words in ImputeRefer regenerated text: 10.52, and by a larger average human-to-target distance of ~2.31 m versus ~1.24 m in the ScanERU subset (Mane et al., 13 Apr 2025).
The following table summarizes major ERU-relevant datasets and benchmarks.
| Dataset / benchmark | Core setting | Reported scale |
|---|---|---|
| YouRefIt (Chen et al., 2021) | Indoor video ERU with language and gesture | 4,195 clips; 432 scenes; 395 categories |
| ScanERU (Lu et al., 2023) | Semi-synthetic 3D ERU with text and synthetic gesture | 706 scenes; 9,929 objects; 46,173 descriptions |
| Refer360 (Islam et al., 6 Dec 2025) | Multi-view, multimodal indoor/outdoor E-RFE | 66 participants; 392 sessions; 13,990 interactions |
| EmbSpatial-Bench (Du et al., 2024) | Egocentric embodied spatial QA | 3,640 QA pairs; 2,181 images |
| SIGAR (Guo et al., 25 Mar 2025) | Multi-attribute state-intention-gesture grounding | 4,195 images; 20,193 texts |
| Multi-perspective reference game (Tang et al., 2024) | Speaker-listener communication under view mismatch | 27,504 generated scenes; 2,970 human expressions |
Taken together, these resources indicate a shift from static indoor image grounding toward multimodal, multi-view, 3D, temporally extended, and communicative settings. A plausible implication is that benchmark design has become a principal instrument for isolating distinct failure modes in ERU: gesture misinterpretation, egocentric spatial confusion, perspective mismatch, and insufficient persistence over time.
3. Geometric, spatial, and perspective-taking foundations
A recurrent finding is that embodied reference cannot be reduced to generic multimodal fusion; it depends on specific geometric and perspectival structure. Early work on Touch-Line Transformer argues that a common assumption in gesture interpretation—the elbow-wrist line—is a misconception, and that the more informative cue is the virtual touch line (VTL), defined as the line from the eye to the fingertip (Li et al., 2022). The geometric consistency objective explicitly measures cosine similarity between the eye-to-fingertip direction and the eye-to-object-center direction, encouraging predicted referents to lie on or near the touch line (Li et al., 2022). Computationally, VTL was shown to outperform the elbow-wrist line on YouRefIt (Li et al., 2022).
Subsequent work refines the geometric model rather than discarding it. AD-DINO introduces the Attention-Dynamic Touch Line (ADTL), in which the attention source is not fixed but switches dynamically between the eye for distant interactions and the MCP joint for close interactions, based on upper-limb configuration and a threshold over cosine similarity (Guo et al., 2024). The central claim is that the meaning of a pointing gesture changes with pointer–referent distance, so a single fixed line is inadequate (Guo et al., 2024). CAPE similarly rejects a single-line assumption, training separate models using head-to-fingertip and wrist-to-fingertip Gaussian ray heatmaps and combining them with a CLIP-aware ensemble (Eyiokur et al., 29 Jul 2025). This line of work suggests that embodied gesture understanding is sensitive not merely to pose detection but to the choice of geometric proxy.
Perspective-taking has emerged as a second foundational theme. REP frames ERU as requiring the receiver to access spatial and visual information relative to the sender, then proposes a sender-centric embodied 3D coordinate system, a body language vector, spatial attention, gesture attention, and verbal fusion (Shi et al., 2023). The key operation is a view rotation that first places the sender at the origin of a 3D coordinate map and then conditions reasoning on sender orientation and gesture (Shi et al., 2023). The emphasis is not only that objects must be localized, but that the receiver must infer how objects are oriented around and seen from the sender.
Egocentric spatial reasoning is also isolated by EmbSpatial-Bench, which insists that spatial relationships “should be described from the egocentric perspective” because agents take themselves as the center of coordinates in embodied tasks (Du et al., 2024). The benchmark covers exactly six egocentric relations—above, below, left, right, close, and far—and extracts them directly from well-annotated 3D datasets rather than object detectors (Du et al., 2024). In a text-only abstraction of the same problem, GSU shows that models often struggle when left/right/forward/backward must be updated relative to a moving embodied agent rather than a fixed Cartesian frame (Sidhu et al., 18 Mar 2026).
Multi-agent communication work makes the same issue explicit in interactive form. In photorealistic 3D indoor environments, successful reference generation and comprehension depend on distinct observations and for speaker and listener, and communicative success degrades sharply as view overlap decreases and relative orientation becomes more adversarial (Tang et al., 2024). This suggests that spatial language in ERU is inherently relational: not only between agent and object, but between the frames of reference of multiple agents.
4. Model families and algorithmic strategies
Method development in ERU has proceeded along several partially overlapping lines: gesture-line modeling, multimodal fusion, 3D grounding, residual adaptation of pretrained vision-language systems, and persistent world modeling.
The Touch-Line Transformer established a joint prediction paradigm in which a transformer predicts both the referent bounding box and a touch-line vector, using a multimodal encoder over ResNet visual tokens and BERT/RoBERTa textual tokens, followed by decoder queries for objects and gestural keypoints (Li et al., 2022). AD-DINO builds on a Grounding-DINO-style architecture with Swin Transformer, BERT, bidirectional cross-modality fusion, a query selection module, and a decoder that jointly predicts the target object box and the attention source of the gesture, with fingertip detections provided by MediaPipe Handmarker (Guo et al., 2024). CAPE retains a transformer encoder-decoder backbone but adds a dedicated heatmap encoder, Gaussian ray heatmaps, an auxiliary object center prediction head, and a size-aware hybrid ensemble using CLIP similarity and model confidence (Eyiokur et al., 29 Jul 2025).
A second family emphasizes explicit perspective modeling. REP adds monocular depth estimation, sender segmentation, a sender-centric 3D coordinate transformation, and staged reasoning through spatial attention, gesture attention, and FiLM-based verbal fusion (Shi et al., 2023). The method is one-stage in output but structurally decomposed into receiver–sender and sender–object relation modeling (Shi et al., 2023). A plausible interpretation is that REP treats perspective-taking as an architectural prior rather than as latent information to be discovered by generic fusion.
A third family generalizes ERU into 3D grounding. ScanERU uses PointNet++ for scene proposals and gesture encoding, GloVe plus GRU for language, and transformer-based multimodal fusion over proposal and gesture features (Lu et al., 2023). Ges3ViG uses PointGroup for 3D instance segmentation, CLIP image and text features, joint feature extraction for human localization, early fusion of gesture and language, and late fusion through a pointing alignment score computed from shoulder, fingertip, and proposal center vectors (Mane et al., 13 Apr 2025). ShapeLLM approaches embodied grounding from the opposite direction: it grounds language in 3D point clouds using a ReCon++ encoder with multi-view image distillation, instruction tuning, and a 3D multimodal LLM architecture with local, global, and absolute position features (Qi et al., 2024).
A fourth line adapts large pretrained multimodal models rather than replacing them. MuRes is presented as a multimodal guided residual module inserted into frozen pretrained backbones such as CLIP, DualEncoder, ViLT, or BLIP-2 (Islam et al., 6 Dec 2025). Instead of passing the full residual pathway, MuRes uses projected representations as queries and original features as keys and values in a cross-attention block,
then forms fused representations by residual addition,
Its role is described as an information bottleneck that extracts salient modality-specific signals and reinforces them into pretrained representations (Islam et al., 6 Dec 2025).
Finally, persistent memory architectures connect ERU to long-horizon dynamic scene understanding. Embodied VideoAgent constructs a persistent object memory from egocentric video together with depth and pose sensing, tracks object states such as , and updates object entries through re-identification and VLM-based action-target association (Fan et al., 2024). SNOW organizes RGB images and point clouds into STEP tokens and a globally aligned 4D Scene Graph (4DSG), making object identity, geometry, and temporal history directly queryable by VLMs (Sohn et al., 18 Dec 2025). These systems do not target canonical ERU benchmarks directly, but they contribute a persistent referential substrate for expressions such as “the car that moved most” or “the object in front of the fridge” (Sohn et al., 18 Dec 2025).
5. Empirical findings
Across datasets and formulations, the dominant empirical result is that current multimodal models underperform humans and frequently fail on the embodied aspects of reference.
On YouRefIt image ERU, the original full multimodal benchmark model achieved 54.7 at IoU 0.25, 40.5 at IoU 0.5, and 14.0 at IoU 0.75, while human performance was 94.2 ± 0.2, 85.8 ± 1.4, and 53.3 ± 4.9 respectively (Chen et al., 2021). Gesture-only models were informative but insufficient, and masking out humans via inpainting degraded performance, supporting the claim that gestures are as critical as language cues (Chen et al., 2021).
Touch-Line Transformer improved YouRefIt substantially, reaching 71.1% at IoU 0.25, 63.5% at IoU 0.50, and 39.0% at IoU 0.75, corresponding to gains of +16.4, +23.0, and +25.0 over the prior YouRefIt full model; at IoU 0.75, this closed 63.6% of the model-human gap (Li et al., 2022). AD-DINO later reported 76.3% at IoU 0.25, 72.4% at IoU 0.5, and 55.4% at IoU 0.75, with the paper noting that this surpasses human performance at the strict IoU 0.75 criterion for the first time on this benchmark (Guo et al., 2024). CAPE achieved 75.0 mAP@0.25, 65.4 [email protected], and 35.7 [email protected], improving especially at IoU 0.25 and 0.50, while DA-ERU reported 78.7, 67.6, and 38.1 mAP on YouRefIt and 62.7, 50.1, and 30.1 on the unseen ISL pointing dataset (Eyiokur et al., 29 Jul 2025, Eyiokur et al., 9 Oct 2025).
Perspective-aware modeling also produced measurable gains. REP surpassed the previous state of the art on YouRefIt by +5.22% absolute accuracy in terms of [email protected], with its 3D view rotation module alone improving the baseline by 2.8% / 3.4% / 5.1% on [email protected] / 0.50 / 0.75 (Shi et al., 2023). In 3D, Ges3ViG achieved 84.60 / 71.03 on the unique split, 67.57 / 55.77 on the multiple split, and 70.85 / 58.71 overall at [email protected] / [email protected] on ImputeRefer, exceeding the gesture-based ScanERU baseline by 29.87 points overall at [email protected] and surpassing the best language-only 3D grounding baseline M3DRefCLIP by 8.93 points overall (Mane et al., 13 Apr 2025). ScanERU’s own ablation already showed the same qualitative pattern: Gesture-only: 17.46 / 13.44, Language-only: 49.18 / 35.62, Full model: 54.45 / 40.94 on validation [email protected] / [email protected] (Lu et al., 2023).
Benchmark studies expose broader weaknesses. EmbSpatial-Bench found zero-shot human performance at 90.33%, while the best reported generation-based score was Qwen-VL-Max: 49.11%, the best open-source likelihood score was MiniGPT-v2: 43.85%, and GPT-4V: 36.07% under generation (Du et al., 2024). Fine-tuning MiniGPT-v2 on EmbSpatial-SFT improved overall accuracy from 23.93% to 32.97% under generation and from 43.85% to 78.10% under likelihood, but generation-based performance remained far below human level (Du et al., 2024). In multi-perspective referential communication, human-human pairs achieved 87.6% communicative success on average, while the best human-speaker/automated-listener setting reached 69.2% and GPT-4o as automated speaker with human listeners reached 64.9% (Tang et al., 2024). Multi-ref EC on SIGAR similarly found that multi-attribute references outperform single-attribute ones, with Qwen-VL reaching 54.4, 51.9, and 32.3 at IoU 0.25 / 0.5 / 0.75, and with intention + gesture emerging as the strongest dual-attribute combination at 53.2 / 47.3 / 28.1 (Guo et al., 25 Mar 2025).
The cumulative pattern is not that ERU is uniformly unsolved, but that performance is highly sensitive to which component is stressed: strict box precision, long-distance pointing, egocentric depth reasoning, multi-agent perspective mismatch, or dynamic scene persistence.
6. Recurrent limitations, controversies, and research directions
Several limitations recur across the literature. First, many methods focus primarily on pointing gestures and only partially model other nonverbal cues. Touch-Line Transformer explicitly notes that it does not fully model richer gaze direction, finger orientation, or lower-limb orientation (Li et al., 2022). AD-DINO likewise states that the current framework does not fully integrate gaze or richer hand gestures (Guo et al., 2024). Refer360 was created precisely because existing datasets showed weak coverage of nonverbal behavior and indoor-centric bias (Islam et al., 6 Dec 2025).
Second, reference-frame ambiguity remains a major source of error. EmbSpatial-Bench reports especially weak performance on depth relations close and far, attributing this to lack of depth-estimation training and more complex reasoning demands (Du et al., 2024). GSU shows that many models correctly compute a new heading and then fail to use it, reverting to a fixed Cartesian interpretation in longer egocentric sequences (Sidhu et al., 18 Mar 2026). Multi-perspective communication experiments similarly find that automated models underuse listener- and speaker-perspective references and instead over-rely on object-relative descriptions (Tang et al., 2024).
Third, realism and scalability in data remain contested. ScanERU and ImputeRefer both use synthetic or inserted human gestures in otherwise real 3D scenes (Lu et al., 2023, Mane et al., 13 Apr 2025). SIGAR depends on Claude-3.5-Sonnet-generated state and intention annotations that are manually filtered (Guo et al., 25 Mar 2025). This does not invalidate the datasets, but it does indicate that large-scale embodied annotation is still expensive and methodologically heterogeneous. Refer360’s multi-view, indoor-and-outdoor recording setup is one response to this limitation, and it shifts the field toward more natural embodied interaction data (Islam et al., 6 Dec 2025).
Fourth, many successful methods remain static or single-turn. The multi-perspective reference game is explicitly single-shot, without clarification dialogue (Tang et al., 2024). SIGAR does not model temporal interaction or sequential feedback (Guo et al., 25 Mar 2025). By contrast, SNOW and Embodied VideoAgent suggest that persistent, world-aligned memory may be necessary for resolving reference in dynamic environments over time (Sohn et al., 18 Dec 2025, Fan et al., 2024). This suggests a convergence between ERU and broader embodied world modeling.
Finally, there is an unresolved modeling question concerning how much structure should be built in. Gesture-line methods, body-centric coordinate systems, residual bottlenecks, and 4D scene graphs all encode strong priors. Position papers grounded in embodied cognition and image schemas argue that such structure may be necessary for interpretability, efficiency, and human-agent alignment (Tamari et al., 2020, Olivier et al., 31 Mar 2025). A plausible implication is that future ERU systems may combine foundation-model backbones with explicit representations for perspective, geometry, temporal persistence, and embodied conceptual structure rather than relying on scale alone.
In that sense, embodied reference understanding has evolved from a narrowly defined multimodal grounding task into a broader research program on how agents interpret situated human intent. The most concrete trajectory in the current literature is clear: richer datasets that capture natural embodied communication, stronger mechanisms for egocentric and multi-agent perspective-taking, better integration of gesture and language, and increasingly persistent object-centered world models for reference over time (Islam et al., 6 Dec 2025, Sohn et al., 18 Dec 2025).