MM-Conv: 3D Context-Aware Dialogue Benchmark

Updated 6 July 2026

MM-Conv is a multimodal dataset, benchmark, and two-stage pipeline for context-aware grounding of referring expressions in dynamic 3D dialogue.
It leverages synchronized VR-captured speech, gaze, motion, and scene geometry to resolve ambiguous and pronominal references.
Contextual rewriting boosts grounding performance by up to 22 percentage points, underscoring the need for explicit disambiguation.

Searching arXiv for the primary paper and closely related prior datasets/models to ground citations. arxiv_search(query="(Deichler et al., 20 May 2026) MM-Conv context-aware grounding 3D dialogue", max_results=5) arxiv_search(query="ScanRefer 3D grounding dataset arXiv", max_results=5) arxiv_search(query="ReferIt3D arXiv", max_results=5) arxiv_search(query="YouRefIt arXiv gesture third-person video", max_results=5) arxiv_search(query="TEACh embodied dialogue arXiv", max_results=5) arxiv_search(query="MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue", max_results=10) arxiv_search(query="(Deichler et al., 2024) MM-Conv Multi-modal Conversational Dataset for Virtual Humans", max_results=5) arxiv_search(query="(Kang et al., 2015) Majorization Minimization MM-Conv", max_results=5) MM-Conv is a multimodal dataset, benchmark, and two-stage grounding pipeline for context-aware grounding of referring expressions in dynamic, egocentric 3D dialogue. It targets a setting in which references such as “it,” “that one,” “the yellowish thing,” or part-whole mentions like “the cloud (in the painting)” must be resolved from multi-turn linguistic context together with evolving nonverbal cues and scene state. The resource is built from 6.7 hours of dyadic VR interaction with synchronized speech, egocentric vision, motion, gaze, facial signals, and 3D scene geometry, and contains 4,211 manually verified referring expressions, with 4,001 remaining after filtering out non-visible targets at referring-expression time. The associated experiments argue that conversational ambiguity resolution and visual localization are better handled as distinct subproblems: contextual rewriting raises grounding performance by 11–22 percentage points on average, and a pure detector reaches 56.7% [email protected] on pronominals after rewriting, nearly double the best end-to-end baseline with context (Deichler et al., 20 May 2026).

1. Problem setting and position in the literature

MM-Conv addresses context-aware grounding of referring expressions that arise spontaneously during multi-turn dialogue in dynamic, egocentric 3D environments (Deichler et al., 20 May 2026). The central problem is not object recognition in isolation, but reference resolution under discourse dependence, ellipsis, anaphora, multimodal deixis, and temporal change. In formal terms, the task is to select the correct object instance $o^* \in O(S)$ for a referring expression $r_t$ given conversational history $H_t$ , scene state $S$ , and optional nonverbal signals such as gaze $g_t$ and motion $m_t$ :

$o^* = \arg\max_{o \in O(S)} P(o \mid r_t, H_t, S, g_t, m_t).$

The benchmark evaluates three classes of referring expression. “Full NPs” are explicit noun phrases intended to be uniquely identifying. “Partitive/attribute NPs” are underspecified references to parts, features, subsets, spatial cues, or deictic forms. “Pronominals” include forms such as “it,” “that,” and “those.” This decomposition is consequential because nearly half of the expressions in the main egocentric view are pronominal, and pronominals are precisely the cases in which static image-style grounding is least adequate.

Relative to prior 3D grounding resources, MM-Conv is designed around spontaneous dialogue rather than scripted, single-turn text. The paper contrasts it with ScanRefer, ReferIt3D, YouRefIt, and TEACh: earlier datasets emphasize static scenes, scripted descriptions, third-person video, or embodied dialogue without continuous synchronized gaze and motion streams, whereas MM-Conv synchronizes speech, motion, gaze, facial signals, and scene geometry under egocentric perspective (Deichler et al., 20 May 2026). The paper characterizes the benchmark as the first multimodal, egocentric benchmark unifying speech, gaze, motion, and scene geometry for spontaneous 3D dialogue grounding.

2. Data collection, environments, and synchronized modalities

The corpus was captured in dyadic VR sessions in which an instruction-giver, designated the main actor, introduced and described objects while an instruction-follower responded but avoided introducing novel referents, thereby increasing the density of referential language (Deichler et al., 20 May 2026). Scenarios were instantiated in apartment rooms and included “introducing a new apartment,” “landlord inspection,” and “interior designer tips.” Each room contained 2–3 such scenarios.

The capture stack combined OptiTrack motion capture with 16 Prime x 41 cameras, a 50-marker skeleton for both participants, MANUS Quantum MetaGloves for finger tracking, and a Meta Quest Pro headset for binocular gaze and 52 facial blendshapes. AI2-THOR, running in Unity, supplied the apartment environments, object metadata, physics, and scene graphs. SMPTE timecode from Tentacle Sync E was injected into Unity, and headset pose was calibrated to mocap frames. As a result, audio, motion capture, gaze, facial signals, and simulator outputs share a frame-accurate temporal reference, allowing word-level timestamps to be aligned with egocentric RGB, depth, and instance-mask frames.

The environments comprise five AI2-THOR apartment rooms spanning common household categories such as sofas, chairs, lamps, desks, paintings or posters, plants, vases, pillows, tables, televisions, boxes, and garbage cans. The average number of interactable objects is $38 \pm 3.16$ per room. Scene representation is available at several levels: per-frame scene graphs with canonical object identifiers and metadata, per-pixel instance segmentation masks, metric depth renderings, and Unity/AI2-THOR scene graphs and meshes with per-frame transforms. Point clouds are explicitly not part of the starter release.

For each referring expression, the dataset provides an egocentric RGB image, per-pixel metric depth, a per-pixel segmentation mask with object IDs, and JSON annotations containing utterance text, token indices for the referring expression span, the referring-expression category, and the referent object ID. The released multimodal assets also include WhisperX transcripts with word-level timings for approximately 250k words, full-body motion for both participants, finger tracking, binocular gaze, facial blendshapes, and scene graphs with object metadata (Deichler et al., 20 May 2026).

3. Annotation pipeline, corpus statistics, and benchmark protocol

Speech transcription uses WhisperX for automatic speech recognition with word-level alignment, followed by human correction in Label Studio while preserving spontaneity, including filled pauses and repetitions (Deichler et al., 20 May 2026). Reference identification and initial categorization were assisted by GPT-4o, which produced utterance-level topic annotations and an initial split into full NPs, partitive/attribute NPs, and pronominals. All classifications and all groundings were then manually verified and corrected. Grounding itself was tied to the egocentric frame at expression time through raycasting and Unity-derived instance masks, again with manual verification.

The corpus contains 4,211 referring expressions in total; after removing references whose targets are not visible at the utterance timestamp, 4,001 remain for analysis. In the main egocentric view, pronominals account for 2,078 expressions, or 49.3% of 4,211, divided into 1,591 single and 487 multiple references. In the interlocutor’s view, pronominals account for 1,070 of 1,547 expressions, or 69.2%, divided into 901 single and 169 multiple references. Room-wise averages of unique objects are approximately 23–30, and the filtered set is constructed to exclude non-visible references at the utterance timestamp.

A human grounding study, used for validation rather than dataset labeling, reports three-way majority agreement with 75–83% unanimous judgments and Krippendorff’s $\alpha = 0.42–0.66$ (Deichler et al., 20 May 2026). Miss patterns include near-misses due to conservative masks at 15–20%, coherent alternative referents at approximately 30%, and scattered confusion at 24–51%, with the highest confusion for pronouns without context.

The evaluation protocol supplies, at time $t$ , conversational context $r_t$ 0, the current referring expression $r_t$ 1, the 3D scene $r_t$ 2, and optionally gaze and motion. Context is defined as the previous $r_t$ 3 utterances, with $r_t$ 4 topic-matched utterances or, if none match, the previous approximately 20 seconds of transcribed text. Performance is measured by IoU between predicted and ground-truth bounding boxes derived from instance masks, together with $r_t$ 5, $r_t$ 6, and mean IoU:

$r_t$ 7

$r_t$ 8

If a model is trained, the paper notes a standard cross-entropy objective,

$r_t$ 9

although the reported experiments are zero-shot or prompt-based and use no fine-tuning.

4. Ambiguity-first architecture: contextual rewriting and visual grounding

The methodological core of MM-Conv is a two-stage grounding pipeline that separates linguistic disambiguation from visual localization (Deichler et al., 20 May 2026). The stated motivation is that end-to-end VLMs conflate coreference resolution, ellipsis recovery, part-whole inference, and deixis with the downstream problem of placing a box on the correct object. The pipeline instead rewrites an ambiguous conversational expression into an explicit noun phrase and only then performs visual grounding.

Stage 1 is contextual rewriting. It takes the referring expression $H_t$ 0, conversational history $H_t$ 1, and the list of visible objects $H_t$ 2 extracted from the scene graph. The model is Qwen2.5-VL run strictly in text mode with a prompt requiring a single unambiguous noun phrase of 3–8 words. The rewriter is instructed to use attributes and spatial cues available in $H_t$ 3, preserve qualifiers and part-whole relations, and prefer the closest semantic match if exact names mismatch. The mapping is formalized as

$H_t$ 4

where $H_t$ 5 is intended to be a definite description suitable for grounding.

Stage 2 is visual localization. It takes the rewritten phrase $H_t$ 6 together with the egocentric scene state $H_t$ 7 at time $H_t$ 8, including RGB, depth, instance masks, and scene graph. The evaluated models are Florence-2, GroundingDINO, and Qwen2.5-VL, all used without fine-tuning. The second stage is written as

$H_t$ 9

with predicted box $S$ 0 maximizing a grounding score $S$ 1.

The paper’s illustrative examples clarify the intended behavior. In one case, discourse mentions a fireplace and then “The painting above it,” after which the expression “Can you move it a bit to the left?” is rewritten as “the wall painting above the fireplace,” enabling GroundingDINO to localize the painting correctly. In another, the utterance “The cloud is darker near the top” is interpreted against prior mentions of a landscape painting and rewritten as “the cloud-like shape in the wall painting,” allowing the grounding model to map a depicted element to the object instance corresponding to the painting. This suggests that the benchmark treats many difficult references as failures of identification rather than failures of low-level spatial localization.

5. Quantitative results, baseline behavior, and error structure

The empirical results show a marked gap between static or monolithic grounding behavior and ambiguity-aware grounding in dialogue (Deichler et al., 20 May 2026). Human text-only performance already indicates that context is differentially useful across expression types: for full NPs, performance is 73.18% without context and 62.45% with context; for partitive NPs, 47.93% without and 60.99% with; for pronominals, 37.43% without and 55.42% with. A common assumption that additional context uniformly helps is therefore not supported; in full NPs, extra dialogue can introduce competing interpretations.

Among zero-shot end-to-end VLMs with context, Qwen2.5-VL is the strongest baseline across the three categories, reaching 53.2/62.0 [email protected]/0.3 on full NPs, 29.6/34.2 on partitive NPs, and 30.4/38.0 on pronominals. Florence-2 obtains 25.8/28.5 on pronominals, GroundingGPT 10.7/16.0, and the other baselines are lower. Without context, pronominal grounding falls to 4.7–9.2% [email protected], which the paper characterizes as near-chance.

After contextual rewriting, performance increases substantially:

RE type	Best end-to-end with context ([email protected])	Best after rewriting ([email protected])
Full NPs	53.2	61.1
Partitive NPs	29.6	≈49.5
Pronominals	30.4	56.7

GroundingDINO, despite having no conversational capability, becomes the strongest overall model after rewriting. On pronominals it reaches 56.7% [email protected]; Florence-2 rises to 48.9%, a gain of 23.1 points over its baseline; Qwen2.5-VL rises to 50.3%, a gain of 19.9 points. On partitive NPs, GroundingDINO reaches approximately 49.5%, Florence-2 39.7% for a gain of 18.1 points, and Qwen2.5-VL 40.8% for a gain of 11.2 points. On full NPs, GroundingDINO reaches 61.1% after rewriting, while Florence-2 reaches 49.1% and Qwen2.5-VL 54.4%. Mean IoU when matched is 77–89%, indicating precise localization once the correct object has been identified.

The rewrite-quality ablation reinforces that qualifiers and part-whole relations are critical. [email protected] is 55.0% for good rewrites, 32.5% for borderline rewrites, and 23.4% for bad rewrites, with an overall figure of 50.5% (Deichler et al., 20 May 2026). Error flags show sizable drops from wrong_object at $S$ 2 percentage points, missing_part at $S$ 3, and lost_qualifier at $S$ 4. The strongest general lesson is explicitly stated in the paper: identification, not localization, is the bottleneck. Typical failure modes include grounding to the speaker’s hand or nearby salient regions rather than following deictic cues, confusion among visually similar items such as multiple lamps or chairs, and missed part-whole inference such as a detail “in the painting.” The observation that semantically incorrect predictions are often spatially close to ground truth suggests that temporal and nonverbal cues remain underexploited.

6. Limitations, release conditions, ethics, and terminological ambiguity

The benchmark has several stated limitations (Deichler et al., 20 May 2026). Its scenes are simulated apartment interiors, and vision is egocentric but rendered rather than natural video, so transfer to real-world robotics is not guaranteed. The current task focuses on single-object references per frame rather than multi-object or temporally extended grounding. Gaze and motion are captured but not incorporated into the baseline VLM evaluations. Annotation includes LLM-assisted classification, albeit with human verification.

The release is planned under CC BY-NC 4.0 for research use. The starter pack, limited to at most 20 MB, contains schemas, 10–20 samples, and the evaluation script; the full release is planned after the camera-ready version, together with documentation and a datasheet. Standardized evaluation scripts implement IoU and $S$ 5 together with prediction and ground-truth schemas. Ethical controls include informed consent, anonymized transcripts, simulated egocentric frames rather than real faces, omission of identity and location metadata, and an explicit prohibition on re-identification.

Future work is framed around exploiting the synchronized temporal and nonverbal streams already present in the corpus. The paper lists video-based grounding, gaze-conditioned attention, body and hand motion cues, fuller 3D scene reasoning, teacher–student distillation from ambiguous to explicit grounded forms, and extension to real-world AR and robotics with richer semantics and longer dialogues (Deichler et al., 20 May 2026).

The label “MM-Conv” is also terminologically overloaded in arXiv usage. A 2024 paper used the same name for a VR-based multimodal conversational dataset for virtual humans oriented toward gesture generation and referential communication (Deichler et al., 2024). In unrelated optimization literature, the label has also been used in discussions of majorization–minimization convergence (Kang et al., 2015). In the present context, however, MM-Conv specifically denotes the 2026 dataset, benchmark, and ambiguity-first grounding pipeline for spontaneous 3D dialogue grounding (Deichler et al., 20 May 2026).