---
title: Dialogue-Based GREC in Multi-Turn Scenes
url: https://www.emergentmind.com/topics/dialogue-based-generalized-referring-expressions-comprehension-grec
type: topic
---

# Dialogue-Based GREC in Multi-Turn Scenes

Dialogue-Based Generalized Referring Expressions Comprehension (GREC) encompasses the set of methods and benchmarks where a system receives a dialogue containing one or more referring expressions and must precisely ground each expression—possibly ambiguous, co-referential, or relating to multiple targets—onto a set of visual entities in a scene. The objective extends classic referring expression comprehension beyond one-shot, single-turn utterances to handle multi-turn, potentially open-ended conversations where expressions may refer to any number of objects, require disambiguation, or depend on context or clarifications. Contemporary research focuses on both the foundational definitions underpinning dialogue-based GREC and the practical challenges of dataset construction, modeling, evaluation metrics, and cross-domain generalization.

## 1. Formal Task Definition in Dialogue-Based GREC

Dialogue-Based GREC generalizes classic REC by introducing multi-turn conversational context, potentially resolving unlimited targets, ambiguity, and coreference. Formally, given:
- An input image $I$, typically containing $N$ candidate objects $O = \{o_1, \ldots, o_N\}$, each associated with visual features $x_o = \varphi(I, o)$ via a convolutional or transformer encoder $\varphi$.
- A multi-turn dialogue context $D = \{u_1, \ldots, u_T\}$, where each $u_t$ is a free-form natural language utterance.
- At turn $T_{\text{ref}} \leq T$, an utterance $r$ containing one or more referring expressions.

The model seeks a grounding function $f: (I, D, r) \rightarrow \hat{O} \subseteq O$ which outputs the (possibly empty) set of target objects $\hat{O}$ whose bounding boxes or segmentations correspond to the referents in $r$ as resolved in the context $D$ [2512.02791]. The model may either score each candidate $o$ via $s(o \mid I, D, r)$ and select those exceeding threshold $\tau$, or output $\arg\max_{o \in O} P(o \mid I, D, r)$ for single-target cases. This formalism subsumes cases with zero, one, or multiple targets, and necessitates scalable set prediction.

A distinctive property is the integration of contextual dialogue—including clarificational exchanges, corrections, and anaphoric terms ("that one", "the others")—which demands a memory-augmented or context-sensitive model capable of accumulating, updating, and refining entity sets over turns [2308.16182].

## 2. Dataset Construction Strategies

Annotated dialogue-grounding data is sparse; thus, synthetic data frameworks play a critical role in research progress. The three-tier data synthesis methodology [2512.02791] trades off controllability and realism as follows:

**Tier 1: Template-Based Synthesis**
- Exhaustively produces single-turn expressions covering color, position, and ordinal patterns via a finite grammar, paired with bounding-box metadata. For example, "the red column of blocks" or "the first blue block of the tall tower."
- Yields $N_{\text{template}} \approx 19k$ samples under strict control.

**Tier 2: Prompted LLM-Based Synthesis**
- Introduces linguistic variability through controlled prompts to GPT-4, generating utterances and semantic slots in structured JSON. Each retained sample must satisfy parser checks for downstream interpretability.

**Tier 3: Full Dialogue with Coreference**
- Synthesizes multi-turn dialogues by chaining Tier 2 expressions, simulating natural clarification, correction, and coreference. Qwen2-VL is fine-tuned on external corpora (VisPro) using a LoRA adapter, generating contextually bound chains of intent and referent [2512.02791].
- Synthetic data is style-aligned with human-authored dialogues via a discriminator loss $L_{DA}$ to minimize KL-divergence in textual style.

All tiers combine annotated scenes (e.g., Minecraft), visual crops, object IDs, and bounding boxes, supporting slot-labeled supervision for both language and grounding objectives.

## 3. Model Architectures and Algorithms

No fundamentally new architecture is proposed exclusively for GREC; rather, existing state-of-the-art vision-language grounding frameworks are adapted to dialogue-based set prediction [2308.16182, 2512.02791, 2307.15554]:

- **Multi-Crop Network (MCN)**, **Vision-Language Transformer (VLT)**, **MDETR/Longformer**, and **Unified Next (UNINEXT)**: baseline REC models extended to output variable-sized candidate sets via a confidence threshold on detection scores.
- Vision encoder: ResNet-101, DarkNet-53, or Detectron2 region features.
- Text encoder: GRU, BERT/RoBERTa, or transformer-based; additionally, cross-attention modules or multimodal feature concatenation.
- Coreference and slot prediction: Multi-label cross-entropy is used for object identification, augmented by attribute-slot prediction (e.g., color, size, location) and margin-based latent disentanglement [2307.15554].

Relational reasoning is implemented by embedding object-pair relationships (e.g., spatial adjacency), and multi-task objectives ensure attribute-specific disentanglement in latent space—particularly important for clarification and repair in dialogue.

No explicit set-prediction or multi-label Hungarian matching losses are introduced for dialogue in [2308.16182], but future extensions are suggested.

## 4. Evaluation Metrics and Protocols

Dialogue-based GREC inherits and extends precision metrics from classic REC:

- **Precision@$(F_1 = 1, \text{IoU} \geq 0.5)$**: For each sample, predicted boxes are matched to ground-truth via IoU $\geq 0.5$, and $F_1$ is computed as $2 \cdot TP/(2 \cdot TP + FP + FN)$. A sample is only correct if $F_1 = 1.0$—all refs found, no extras.
- **No-Target Accuracy (N-acc)**: For cases where the correct referent set is empty, N-acc is the proportion of samples where no boxes are predicted [2308.16182].
- **Object-level F1**: Used for multi-label referent identification in each turn, especially critical for evaluating context-sensitive clarification, with $\Delta F_1$ measuring improvement before and after clarificational replies [2307.15554].
- **Style alignment loss $L_{DA}$**: In synthetic data generation, discriminator-based domain adaptation quantifies textual realism conditional on human-vs.-synthetic dialogue style [2512.02791].

Average precision (COCO-style) is tracked for completeness but is noted to be less suitable for dialogue GREC due to its tolerance of low-confidence extras.

## 5. Empirical Results and Analysis

Comparative performance across synthesis tiers, model architectures, and evaluation splits reveals several key findings:

| Training Data           | Size   | Mean F1 | Prec@($F_1=1$) |
|------------------------|--------|---------|---------------|
| gRefCOCO (out-of-domain)| 209k   | 19.1    | 13.5          |
| Tier 1 (Template)      | 19k    | 42.8    | 24.5          |
| Tier 2 (AI-Short)      | 1k     | 22.8    | 15.7          |
| Tier 3 (AI-Dialogue)   | 1k     | 24.8    | 11.2          |

Fine-tuning on in-domain template data yields $+23.7$ F1 absolute gain over large, out-of-domain sets (gRefCOCO). At constant data size (1k), LLM-generated short expressions and full dialogues yield greater Precision and F1 than pure templates. Notably, models perform better on "mention-only" scenarios than on full multi-turn coreferential dialogues, highlighting the difficulty of context-sensitive grounding [2512.02791].

Multi-modal, relational BART-based models excel at processing clarificational exchanges, achieving the largest $\Delta F_1$ improvements after ambiguous referents are explicitly repaired. Disentangled object-centric representations and explicit attribute-slot supervision are shown to be essential for dialogue GREC [2307.15554]. Simpler language-only models fail to exploit dialogue context and may worsen after clarification exchanges.

Ablation analysis combining synthetic tiers demonstrates that Tier 1 + Tier 2 maximizes F1, while Tier 3 can hurt performance due to style or domain mismatch unless alignment losses are deployed.

## 6. Disambiguation, Clarificational Exchanges, and Coreference Resolution

Dialogue-Based GREC critically depends on a system’s ability to repair referential ambiguity through clarificational exchanges (CEs), which are formalized as the triples $\langle U_b, CR_t, U_a \rangle$—user turn introducing ambiguity, system clarification request, and user's disambiguating reply [2307.15554]. Core steps include:

- Detection of referential ambiguity in $U_b$.
- Generation of $CR_t$ by the system.
- Contextual update after $U_a$ (quantified by $\Delta F_1$ and improvement percentages).

Relational models, especially those with attribute-slot and spatial relation modules, outperform pure language systems in exploiting clarificational repairs. Architectural constraints for high CE-processing capacity include multi-task slot losses and relational encoders. Disentanglement objectives—margin-based or multi-task slot constraints—produce embeddings most amenable to coreferential grounding across turns.

## 7. Future Directions and Open Challenges

Current limitations include insufficient set-prediction objectives, lack of explicit mechanisms for no-object prediction, and difficulties with dialogue-style domain adaptation [2308.16182, 2512.02791]. Prospective research themes include:

- Reinforcement-triggered, targeted data synthesis to address model failure cases (active synthesis).
- Advanced adversarial domain alignment to close the gap between synthetic and human dialogue styles.
- Extension to open-vocabulary, dynamic visual environments, and meta-learning for cross-scene generalization.
- Benchmark construction for turn-based, dialogue-intensive GREC, including richer anaphora, coreference, and interactive grounding protocols.
- Integration of dialog managers and context modules for incremental entity set updates and interactive clarification handling.

This suggests dialogue-based GREC is an active research area that unifies visually-grounded language understanding, robust referential grounding, and sequential, context-sensitive entity selection, offering substantial complexity distinct from classic REC.

Source: https://www.emergentmind.com/topics/dialogue-based-generalized-referring-expressions-comprehension-grec