Using Machine Mental Imagery for Representing Common Ground in Situated Dialogue

Abstract: Situated dialogue requires speakers to maintain a reliable representation of shared context rather than reasoning only over isolated utterances. Current conversational agents often struggle with this requirement, especially when the common ground must be preserved beyond the immediate context window. In such settings, fine-grained distinctions are frequently compressed into purely textual representations, leading to a critical failure mode we call \emph{representational blur}, in which similar but distinct entities collapse into interchangeable descriptions. This semantic flattening creates an illusion of grounding, where agents appear locally coherent but fail to track shared context persistently over time. Inspired by the role of mental imagery in human reasoning, and based on the increased availability of multimodal models, we explore whether conversational agents can be given an analogous ability to construct some depictive intermediate representations during dialogue to address these limitations. Thus, we introduce an active visual scaffolding framework that incrementally converts dialogue state into a persistent visual history that can later be retrieved for grounded response generation. Evaluation on the IndiRef benchmark shows that incremental externalization itself improves over full-dialog reasoning, while visual scaffolding provides additional gains by reducing representational blur and enforcing concrete scene commitments. At the same time, textual representations remain advantageous for non-depictable information, and a hybrid multimodal setting yields the best overall performance. Together, these findings suggest that conversational agents benefit from an explicitly multimodal representation of common ground that integrates depictive and propositional information.

• The paper presents an active visual scaffolding framework for situated dialogue, demonstrating that externalizing common ground as a sequence of schematic visual artifacts improves question-answering performance.
• Both Agentic-Image and Agentic-Text variants of the framework outperform baseline models, with Agentic-Image achieving higher accuracy on Temporal (0.50) and Inferred (0.58) queries, and text excelling slightly on Spatial (0.26) questions.
• The hybrid condition maintaining paired visual and textual artifacts per frame shows the best overall performance, indicating the modalities preserve complementary rather than redundant evidence with best overall performances Temporal 0.58, Spatial 0.44, Attributive 0.64 and Inferred 0.52

This paper introduces an active visual scaffolding framework for situated dialogue, in which a conversational agent incrementally externalizes its evolving interpretation of the common ground into a persistent sequence of schematic visual artifacts, and later retrieves from this multimodal memory bank to answer referential questions. The central hypothesis is that purely textual representations of shared context suffer from representational blur—the collapse of similar but distinct entities into interchangeable descriptions—and that depictive intermediate representations, analogous to human mental imagery, can mitigate this failure mode (2604.21144).

Motivation and problem framing

The authors ground their argument in conversational grounding theory: human interlocutors routinely rely on underspecification, recovering meaning from shared context, gaze, and an evolving model of the physical environment (Clark & Brennan). LLM-based agents, constrained to text and the context window, exhibit locally appropriate grounding behaviors such as acknowledgments but fail to reliably reuse established common ground over long horizons. Prior work by the same group showed that when common ground is stored textually—as summaries or ontological structures—it is vulnerable to compression, semantic drift, and ambiguity.

The paper distinguishes its approach from two existing paradigms of visual scaffolding: attention-based methods that direct model focus within fixed input images, which cannot recover spatial information absent from the original pixels; and latent scaffolding methods that form internal visual representations in hidden states, which are not inspectable. The proposed framework instead prioritizes explicit generation, producing inspectable artifacts that record what the model has inferred, preserved, and grounded over time—a verifiable history of conversational interpretation.

Task and evaluation setting

The framework is evaluated on the MeetUp subset of the IndiRef benchmark. MeetUp! consists of real-time cooperative navigation dialogues on a 2D grid of rooms rendered with ADE20K images, under partial observability and symmetric visibility: neither participant has a global map, forcing active negotiation of a shared spatial understanding. IndiRef augments these transcripts with QA pairs probing four semantic dimensions—Temporal, Spatial, Attributive, and Inferred—with strict perspective-taking requirements (deictic markers such as "my car" versus "your car"). Evaluation uses an LLM-as-a-Judge protocol with reported high agreement with human annotators, with $N=50$ per category ($N_{total}=200$).

Framework

The pipeline has two phases. In the first, three modules incrementally transform dialogue $\mathcal{D}$ into a memory bank $\mathcal{M} = \{\alpha_1, \dots, \alpha_K\}$:

• Observer: at each turn, decides among discrete edit actions $e_t \in \{New, Continue, Skip\}$, factoring each update into a scene descriptor $\delta_t$ (depictable content) and metadata $m_t$ (non-depictable content such as negations, intentions, and movement cues).
• Constructor: materializes the current interpretation into an artifact—either a schematic image $\mathcal{I}_t$ (visual condition) or a dense scene summary $\mathcal{S}_t$ (textual condition)—using rejection sampling with $J=3$ candidates scored by a faithfulness function $N_{total}=200$0, defined as the proportion of verified atomic facts satisfied.
• Linker: extracts sparse cross-artifact triplets $N_{total}=200$1 capturing topological relations between scenes.

A key design principle is Style-as-Semantics: artifacts are deliberately schematic rather than photorealistic, with color-coded outlines encoding epistemic status (black = confirmed with grounded position; red = confirmed but unresolved position; blue = assumption). This keeps uncertainty explicit rather than collapsing it into an over-committed interpretation. A pixel-normalization step after each edit stabilizes iterative image editing against cumulative drift.

In the second phase, a Reasoner performs planned RAG: it produces a plan of POV, RAG[N], PROCESS, and FINAL_ANSWER commands, retrieving selectively via a hybrid scoring function combining CLIP-style visual similarity against artifacts ($N_{total}=200$2) with text similarity against metadata. Two separate artifact sequences are maintained per speaker perspective, since grounding is speaker-relative.

All modules use Qwen3-VL-32B-Thinking; image editing uses an 8-step Lightning LoRA variant of Qwen-Image-Edit; judging uses Llama 3.1 8B.

Results

Framework	Temporal	Spatial	Attributive	Inferred
FD (Qwen3-VL-Thinking)	0.18	0.10	0.20	0.24
FD (Qwen-QwQ)	0.32	0.38	0.40	0.40
Agentic-Image	0.50	0.24	0.44	0.58
Agentic-Text	0.42	0.26	0.44	0.46

Three findings stand out. First, both agentic variants outperform the full-dialogue VL baseline across all relation types, indicating that incremental externalization itself—not merely agentic decomposition—is beneficial. Notably, this reverses the earlier finding of Mohapatra et al., where full-dialog reasoning beat a retrospective agentic representation pipeline; the critical factor appears to be constructing state progressively rather than retrospectively. Second, despite being built on the weaker VL backbone, Agentic-Image exceeds the stronger text-only Qwen-QwQ baseline on Temporal, Attributive, and Inferred queries. Third, the modality comparison reveals a structured trade-off rather than uniform advantage: images excel on Temporal (0.50) and Inferred (0.58), while text retains a slight edge on Spatial (0.26 vs. 0.24).

Analysis by reasoning scope shows Local questions reach roughly 50% accuracy versus 30–35% for Relational ones across conditions. Within the Spatial category, the image condition degrades more sharply because connectivity must be recovered through sparse Linker triplets; textual summaries preserve transitions propositionally. The faithfulness analysis is particularly instructive: higher Constructor faithfulness $N_{total}=200$3 correlates positively with accuracy on Temporal queries, but negatively on Spatial queries—a retrieval mismatch in which highly faithful images act as strong retrieval targets that lock the system onto the correct starting room without following cross-frame links to the answer room.

The qualitative mechanisms behind image-condition gains are twofold. Visual scaffolding makes representational blur structurally impossible: conflicting attributes mentioned across distant turns (e.g., a red rug and a yellow rug in two bathrooms) remain distinct elements in one explicit state, whereas textual compression merges them. It also enforces epistemic boundaries: when one speaker's clarification question implicitly reveals their own environment, the update routes exclusively to that speaker's canvas, preventing the linguistic cross-contamination observed in the textual condition where one speaker's correction overwrites the other's established state.

Hybrid setting

An Agentic-Both condition maintaining paired visual and textual artifacts per frame achieves the best overall performance: Temporal 0.58, Spatial 0.44, Attributive 0.64, Inferred 0.52. The large gains on Spatial (0.44 vs. 0.24/0.26) and Attributive (0.64 vs. 0.44) indicate the modalities preserve partially non-overlapping evidence rather than redundant views. The sole exception is Inferred grounding, where pure Agentic-Image remains stronger (0.58 vs. 0.52), suggesting the added textual channel may reintroduce ambiguity and dilute the forced commitment benefit of visual externalization.

Limitations and open questions

The paper concedes several limitations directly tied to its results. The framework handles positive, visually expressible commitments well but is substantially weaker on negation, denial, uncertainty, and implicature: negative clarification questions ("It's not yellow?") are pragmatically informative yet treated as non-depictable and skipped, losing grounding signals entirely. Retrieval also fails on indirect requests—an "Do you remember X?" prompt yielded "yes" despite correct artifact retrieval. The Spatial bottleneck is attributed to Linker triplet quality rather than local visual grounding, though this diagnosis is inferential. Evaluation rests on a single benchmark subset of 200 questions with author-annotated complexity labels vetted only via GPT-4o-mini, and the faithfulness–accuracy relationship depends on the verifier's reliability. Open questions include whether integrated multimodal memory (rather than retrieval-time combination) would outperform the hybrid setting, and how low-confidence or conflicting multimodal evidence could be connected to dialogue policy for clarification and repair.

Conclusion

The paper demonstrates that incremental externalization of common ground improves situated question answering over full-transcript reasoning, and that within incremental memory, depictive and propositional representations carry complementary evidence: images reduce representational blur and enforce concrete scene commitments, while text better preserves topology, negation, and other non-depictable information. The hybrid condition achieving the strongest overall performance supports modeling persistent common ground as an explicitly multimodal structure, while leaving open how relational link quality and non-depictable grounding signals should be handled within such memories.