Embodied Referring Expression Comprehension

• Embodied Referring Expression Comprehension is a multimodal task that integrates verbal directives and gesture cues to precisely localize objects in shared environments.
• It leverages advanced fusion techniques and perspective-taking modules, employing datasets like YouRefIt and Refer360 to benchmark model performance.
• Researchers focus on improving 3D view transformations, multi-attribute signal fusion, and interactive dialogue to enhance human–robot interaction.

Embodied Referring Expression Comprehension (EREC) is the task of grounding multimodal referring acts—comprising both natural language and embodied gesture—into physical object localization in a shared environment. Unlike classical referring expression comprehension (REC), EREC emphasizes the integration of temporally and semantically aligned nonverbal cues (e.g., hand/arm pose, pointing, gaze) with verbal utterances, and demands explicit or implicit perspective-taking by the model to account for the referrer's spatial frame. This domain is foundational in human-robot interaction, cognitive robotics, and multi-agent AI, and is characterized by a rapidly developing set of benchmarks, modeling paradigms, and evaluation protocols.

1. Formal Task Definition and Distinguishing Features

In EREC, the system receives as input a set of multimodal signals: a high-dimensional observation of the scene (RGB, optionally with depth), a referring natural language utterance $\mathcal{S}$, and an explicit or estimated gesture channel (e.g., arm/hand keypoints, pointing cone, or saliency heatmaps). The objective is to output a localization of the referred object, typically as a bounding box $\hat{b}$, ideally matching the target $b^*$ at a specified intersection-over-union (IoU) threshold.

EREC is defined by two essential axes:

• Multimodality: The necessity to fuse natural language and temporally/sequentially aligned gesture features for disambiguated grounding. Nonverbal channels are not auxiliary but critical; explicit ablation shows significant accuracy drops upon their removal.
• Perspective-Taking: The demand for the model to interpret egocentric or allocentric cues, necessitating inference of the referrer's spatial origin and orientation for correct reference interpretation, sometimes including 3D transformations from the receiver's to the sender's viewpoint (Chen et al., 2021, Shi et al., 2023).

2. Benchmark Datasets and Data Collection Protocols

YouRefIt

A canonical dataset for EREC, YouRefIt comprises 4,195 annotated reference acts across 432 diverse indoor scenes. Each instance provides synchronized video and canonical frames capturing the act (average length 2.81 s; ∼497k frames), bounding boxes for the referred object, a transcribed utterance with parsed semantic spans, and detailed hand/arm keypoints and part affinity fields (PAFs). The annotation pipeline includes:

• Amazon Mechanical Turk-driven scene setup, object selection, and reference performance.
• Segmentation into short, gesture-anchored clips and identification of canonical pointing frames.
• Manual parsing of utterances into target object, attribute, spatial, and comparative components.
• Extraction of 2D/3D gesture features using OpenPose and computation of pointing-cone heatmaps (σ = 15°/30°) and MSI-Net saliency maps (Chen et al., 2021).

Refer360

Refer360 expands the scope to multi-view (egocentric, exocentric, depth) capture, real-world indoor and outdoor scenes, and includes natural gaze, pointing, and speech from 66 participants over 392 sessions (13,990 interactions, 3.2M frames). Canonical moments are labeled for multimodal co-registration, and object referents span 75 everyday categories. The dataset addresses prior limitations—single-view bias, limited nonverbal coverage, and indoor-only focus—by capturing concurrent verbal, pointing, gaze, and 3D skeleton streams (Islam et al., 6 Dec 2025).

SIGAR

The State-Intention-Gesture Attributes Reference (SIGAR) dataset extends YouRefIt with 20,193 instances annotated with free-form state and intention sentences alongside gestures and object box labels, supporting studies of multi-attribute, embodied references (Guo et al., 25 Mar 2025).

3. Model Architectures and Multimodal Fusion Strategies

Canonical EREC Models

Typical pipelines adopt a feature extraction phase (vision: Darknet-53 or ResNet-50; language: BERT; gesture: PAF/pointing cone/MSI-Net), followed by multimodal fusion and anchor-based box regression/classification heads (Chen et al., 2021). Explicit fusion is achieved through attention mechanisms (e.g., text-conditioned visual attention), additive feature combination, and normalization layers.

Perspective-Taking and Relation Reasoning

REP (Shi et al., 2023) introduces a two-stage approach:

1. 3D View Rotation Module: Transforms the coordinate frame to the sender's position and orientation by computing a sender-centric 3D coordinate map (from monocular depth estimation and body segmentation) and encoding the sender's "body language" vector.
2. Relation Reasoning Module: Applies sequential spatial, nonverbal (gesture), and verbal attention blocks to localize the referred object. Fusion of spatial cues (e.g., projecting attention along the sender's orientation) and multi-layer transformer reasoning enables accurate perspective alignment.

Multi-Attribute Fusion

Recent models leverage transformers operating on visual, gestural, and linguistic embeddings, concatenated with multi-head attention. Ablative analyses show that intention and gesture provide the most complementary signals, while the ordering of state/intention/gesture in prompts influences performance by up to 4% (Guo et al., 25 Mar 2025).

Guided Residual Fusion

MuRes (Islam et al., 6 Dec 2025) is a lightweight adapter that uses cross-attention to extract salient signals from each modality and reinforce them within frozen vision–language encoder streams, yielding robust performance boosts on diverse datasets.

4. Evaluation Metrics and Experimental Results

EREC systems are evaluated using object localization accuracy at various IoU thresholds (typically 0.25, 0.50, 0.75), frame-level canonical detection F₁ (for videos), and breakdowns by object size.

Representative Results on YouRefIt (Chen et al., 2021):

Model Variant	[email protected] (All)	Notes
Language-only	16–25%	Pretrained or trained on inpainted scenes
Gesture-only	10–31%	PAF+saliency; spatial but non-disambiguating
Language+Gesture	up to 40.5%	Best: Ours_Full (fusion of PAF, saliency & text)
Human	85.8%	@IoU=0.50; 53.3% @IoU=0.75

Removing gesture channels halves performance. Gesture-only models highlight candidate regions but cannot resolve ambiguity; fusion of explicit gesture features is necessary for robust localization.

Perspective-Taking Models (Shi et al., 2023):

• REP achieves 45.7% @0.50 (all), an absolute gain of 5.2% over prior SOTA.
• Largest relative improvement occurs for small-object references (25.4% vs 16.3%).
• Each component (depth estimation, sender-centric rotation, gesture attention, verbal fusion) contributes incrementally to final performance.

Multi-Attribute Fusion (Guo et al., 25 Mar 2025):

Input Attributes	[email protected]	[email protected]
State only	35.9%	20.9%
Intention only	38.9%	22.2%
Gesture only	27.4%	17.9%
State + Intention	45.1%	26.1%
Intention + Gesture	47.3%	28.1%
Full (Multi-Attribute Fusion)	53.4%	37.2%

Correct prompt ordering (language attributes first, gesture last) significantly aids performance.

Multi-Perspective Benchmarks

Grounding Language in Multi-Perspective Referential Communication formalizes reference as a two-agent game with adversarial placement, establishing human speaker–listener performance (~90%) as the ceiling, with powerful models (e.g., LLaVA-1.5, GPT-4o) lagging by 20–30%, particularly on scenes requiring explicit viewpoint reasoning (Tang et al., 2024).

5. Methodological Advances, Challenges, and Limitations

Critical Methodological Advances

• Explicit modeling of 3D geometry and sender-centric frames is essential for robust perspective-taking (Shi et al., 2023).
• Guided residual fusion (MuRes) systematically improves multimodal representations by selectively amplifying salient modality-specific cues (Islam et al., 6 Dec 2025).
• Multi-attribute reference understanding (SIGAR) formalizes state, intention, and gesture fusion, demonstrating significant gains over single-attribute or unimodal approaches (Guo et al., 25 Mar 2025).
• Agent-based feedback and communicative success-driven learning regimes begin to close the gap between model and human performance in multi-agent scenes (Tang et al., 2024).

Remaining Challenges and Open Problems

• High-level human performance on EREC (85.8%+ on [email protected]) is not matched even by the best fusion or perspective-taking models (≤46%).
• Current gesture representations are fragile to occlusion, variance in viewpoint, and low frame rate. Robustness in multi-agent, multi-modal, and outdoor environments remains limited.
• Most datasets rely on single-turn references; complex, multi-round or situated dialogue is an open research direction.
• Fine-grained temporal modeling, e.g., for continuous gesture sequences or dynamic gaze, is not yet widely exploited.

6. Extensions: Manipulation, Interactive QA, and Multi-Agent Communication

Embodied Manipulation and Interactive QA

REMQA fuses EREC with manipulation-oriented embodied question answering. After grounding the referent, the agent must navigate to the object, manipulate it (e.g., open or pick up), and answer follow-up questions (existence, counting, spatial) using post-manipulation sensory input. Benchmarks demonstrate that this pipeline achieves substantially higher task completion (e.g., SPL = 0.595 vs. 0.178 over prior RL methods) and validates the criticality of physical interaction for unoccluding referents and resolving object permanence (Sima et al., 2022).

Towards Natural Communication

Recent experimental setups evaluate models as both “speakers” and “listeners” in multi-agent, perspective-divergent environments, with adversarial placement and preference-learning-based optimization. Data shows that only a minority of references by current models adopt true listener- or speaker-centric language; this is a dominant error mode. Fine-tuning for communicative success improves performance up to 69%, but a substantial gap with humans persists (Tang et al., 2024).

7. Future Directions and Research Priorities

Several directions are emerging as priorities for advancing EREC:

• Temporal Reasoning: Expansion of models to operate over continuous sequences of gesture and language, leveraging spatio-temporal transformers or recurrent modules.
• Interactive Dialogue: Enabling systems to explicitly ask clarifying follow-up questions in cases of ambiguity, leveraging uncertainty over region proposals or score thresholds (Dogan et al., 2021, Guo et al., 25 Mar 2025).
• Embodied Sensing: Integration of gaze, proxemics, and posture alongside pointing; further use of active perception and scene exploration.
• Dataset Scale and Diversity: Standardization of splits and experimental regimes across large-scale multi-view, multi-modal datasets (e.g., Refer360), including extension to complex environments and top-down/multi-robot settings.
• Plug-and-play Adaptation: Adapter-based fine-tuning of large foundation vision-LLMs with modules such as MuRes, targeting transfer to real-robot settings.
• Manipulation-Integrated Grounding: Expanding REMQA-style tasks to include multi-stage, composite instructions, and closing the simulation-to-reality gap.

The field is increasingly recognizing the necessity of fusing information from diverse embodied modalities, reasoning about perspective, and supporting dynamic, context-dependent reference resolution and grounding for practical deployment in human–robot interaction and multi-agent AI systems (Chen et al., 2021, Shi et al., 2023, Islam et al., 6 Dec 2025, Tang et al., 2024, Guo et al., 25 Mar 2025, Sima et al., 2022).