MDSEval: Meta-Evaluation Benchmark for MDS

• MDSEval is a meta-evaluation benchmark designed to assess automatic evaluation methods for multimodal dialogue summarization by integrating text and images.
• It curates challenging image-sharing dialogues using a MEKI filtering framework to ensure key information is exclusive to the respective modalities.
• The benchmark provides expert annotations across eight quality dimensions, exposing biases and limitations in current multimodal evaluators.

MDSEval is a meta-evaluation benchmark for Multimodal Dialogue Summarization (MDS) whose purpose is to measure and improve automatic evaluation methods rather than to train summarization systems. It targets image-sharing dialogues in which critical information is distributed across text and images, and it provides corresponding summaries together with expert human judgments across eight quality aspects. In the formulation reported for the benchmark, MDSEval is presented as the first large-scale meta-evaluation benchmark specifically designed for MDS and as the first work to identify and formalize key evaluation dimensions specific to this task (Liu et al., 2 Oct 2025).

1. Scope and motivation

MDSEval is situated at the intersection of dialogue summarization, multimodal reasoning, and evaluation methodology. The benchmark addresses a problem that arises when a dialogue contains both textual exchanges and shared images: summary quality cannot be assessed adequately by text-only criteria alone. Existing text-only evaluation metrics such as ROUGE and BERTScore ignore visual content entirely, while MDS systems must capture information that may be split across modalities (Liu et al., 2 Oct 2025).

The benchmark’s goals are explicitly meta-evaluative. It is designed to supply a diverse collection of real image-sharing dialogues, generate multiple candidate summaries per dialogue from state-of-the-art MLLMs, collect expert human annotations across eight fine-grained dimensions, and benchmark existing automatic metrics to reveal their weaknesses. This framing is central to MDSEval: the dataset functions as a gold standard for assessing how closely automatic evaluators align with human judgment, thereby supporting the development of more reliable and human-aligned multimodal metrics (Liu et al., 2 Oct 2025).

A common misconception is to treat MDSEval as a training corpus for MDS models. The benchmark is instead defined as a resource for calibration and comparison of evaluation methods. Its role is therefore analogous to a reference standard in meta-evaluation rather than a supervision set for generation.

2. Dataset composition and curation pipeline

The source dialogues were drawn from PhotoChat and DialogCC, with reported source scales of approximately 12K and 45K image-sharing conversations, respectively. An initial filtering stage retained examples longer than 150 tokens and satisfying cross-modal coherence under a zero-shot check with Claude-3.5, leaving approximately 3K candidate dialogues. MDSEval then applied its MEKI filtering framework to select the top 300 most challenging dialogues, after which 198 dialogues were curated for annotation (Liu et al., 2 Oct 2025).

For each dialogue, five candidate summaries were retained. These were produced from four recent MLLMs—Molmo-72B, GPT-4o-mini, Gemini-1.5-flash, and Qwen-vl-max—combined with three prompting strategies: zero-shot, in-context learning, and guided prompts. To maximize diversity, the final five generation setups were selected to maximize the total pairwise embedding distance among their outputs.

Human annotation was performed by expert annotators. Each summary received, on average, 2.9 annotations; 93.1% of summaries were annotated by three experts and 6.9% by two. The benchmark reports the following core descriptive statistics (Liu et al., 2 Oct 2025).

Statistic	Value
Total dialogues	198
Summaries per dialogue	5
Avg turns per dialogue	17.1
Avg tokens per dialogue	209.0
Evaluation aspects	8
Avg annotators per summary	2.9
Avg sentences per summary	4.5

This curation procedure is intended to yield a compact but difficult benchmark. The emphasis on challenge is not incidental: the retained dialogues are meant to require genuine multimodal understanding rather than extraction from a single modality.

3. Annotation schema and evaluation dimensions

MDSEval defines eight quality aspects. These comprise coherence, conciseness, three forms of coverage, modality balance, topic progression, and faithfulness. With the exception of faithfulness and modality balance, the dimensions are rated on Likert scales.

Multimodal Coherence (COH) measures how smoothly the summary integrates and sequences text and visual information. The scale runs from 1, denoting poor integration, to 5, denoting seamless and natural integration.

Conciseness (CON) measures how efficiently the summary conveys essential information without redundancy. The scale runs from 1, extremely verbose, to 5, perfectly concise.

Multimodal Coverage (COV) is decomposed into three sub-aspects: visual critical coverage (COV-I), textual critical coverage (COV-T), and overall critical coverage (COV-O). Let $K$ denote the set of all human-identified key facts in the dialogue-plus-image source, and let $K_s \subseteq K$ denote the facts present in summary $s$. The reported scoring rule is

$\text{Score} = 5 \times \frac{|K_s|}{|K|}$

capped to an integer from 1 to 5 (Liu et al., 2 Oct 2025).

Modality Information Balance (BAL) measures whether the summary avoids over-focusing on either text or image. It is rated on a bipolar 1–7 scale, where 1 denotes entirely text-only, 7 denotes pure image caption, and the ideal value is 4.

Topic Progression (PROG) measures how accurately the summary reflects the sequence of topics and image references in the dialogue. The scale runs from 1, disordered flow, to 5, exact capture of all transitions.

Multimodal Faithfulness (FAI) is defined differently from the scalar Likert dimensions. It is a sentence-level classification with four labels: Faithful, Not-faithful-to-Text, Not-faithful-to-Image, and Not-faithful-to-Both. The summary-level label is then aggregated under explicit rules: a summary is Faithful only if all sentences are Faithful; it is Not-faithful-to-Both if any sentence has that label, or if there are mixed image and text errors; otherwise it inherits the single-modality error type.

Taken together, these aspects define a task-specific ontology for MDS evaluation. A plausible implication is that MDSEval treats summary quality as structurally multidimensional rather than reducible to a single scalar notion of overlap or preference.

4. MEKI filtering and multimodal challenge selection

A central methodological contribution of MDSEval is the Mutually Exclusive Key Information (MEKI) criterion. MEKI is introduced to ensure that the benchmark genuinely requires multimodal understanding: neither the image nor the dialogue text alone should suffice. Information in one modality that cannot be inferred from the other is termed exclusive key information, and dialogues with high MEKI scores are described as containing vital details split across modalities (Liu et al., 2 Oct 2025).

The formulation uses unit-normalized CLIP embeddings for image, text, and a pseudo-summary. Let $I, T, S \in \mathbb{R}^N$ denote the image embedding, dialogue-text embedding, and pseudo-summary embedding, respectively. The image embedding is first orthogonalized with respect to the text embedding:

$$I_T^\perp = I - \mathrm{Proj}_T(I) = I - \frac{\langle I,T\rangle}{\langle T,T\rangle}T.$$

The exclusive key information of the image relative to the text is then defined as

$$\mathrm{EKI}(I\mid T;S) = \left\|\mathrm{Proj}_S(I_T^\perp)\right\| = \left\|\frac{\langle I_T^\perp,S\rangle}{\langle S,S\rangle}S\right\|.$$

A symmetric quantity $\mathrm{EKI}(T\mid I;S)$ is computed for the text relative to the image, and MEKI aggregates the two:

$$\mathrm{MEKI}(I,T;S) = \lambda\,\mathrm{EKI}(I\mid T;S) + (1-\lambda)\,\mathrm{EKI}(T\mid I;S),$$

with $\lambda = 0.3$ to balance typically stronger text exclusivity (Liu et al., 2 Oct 2025).

MEKI was computed for the approximately 3K filtered dialogues, and the top 300 were selected. The paper reports that a human study found a strong Spearman correlation, approximately $K_s \subseteq K$0, between MEKI scores and annotator judgments of modality exclusivity. This gives MEKI a dual role: it is both a filtering heuristic and an empirical operationalization of multimodal dependence.

5. Benchmarking methodology

MDSEval evaluates automatic methods under two paradigms: pointwise scoring and pairwise comparison. The benchmarked evaluator families are LLaVA-Critic, MLLM-as-a-Judge, Image-to-Prompt, and Checklist-CoT. Each family is instantiated with three underlying MLLMs: GPT-4o-mini, Gemini-1.5-flash, and Qwen-vl-max (Liu et al., 2 Oct 2025).

The evaluator families differ in how they access multimodal information. LLaVA-Critic is described as a supervised, fine-tuned critic model for multimodal evaluation. MLLM-as-a-Judge uses zero-shot ranking or scoring by an off-the-shelf MLLM. Image-to-Prompt converts images to textual descriptions and then applies a text-only LLM. Checklist-CoT uses chain-of-thought checklist prompting to guide the model through modality-specific checks before an overall judgment.

The reported metrics are aspect-sensitive. For pointwise evaluation, MDSEval uses Spearman’s rank correlation and Mean Squared Error (MSE). For pairwise evaluation, it uses the accuracy of predicting the human-preferred summary. For faithfulness, where class imbalance is salient, it uses balanced accuracy and macro-F1. The reported Spearman formula is

$K_s \subseteq K$1

This methodology treats evaluation models not as generators but as predictive systems whose outputs can be compared directly against human annotations at multiple granularities.

6. Empirical findings, failure modes, and research significance

The benchmarking results are consistently described as exposing substantial limitations in current multimodal evaluators. For pointwise scoring over coherence, conciseness, coverage, balance, and progression, Spearman correlations were generally low, often below $K_s \subseteq K$2, and MSE values were around $K_s \subseteq K$3–$K_s \subseteq K$4. The reported explanation is that evaluator scores were heavily hedged and clustered around a single value, typically 4, which yielded poor discrimination among high-quality summaries (Liu et al., 2 Oct 2025).

Under pairwise comparison, accuracy only barely exceeded 60% even on conciseness and coverage, while most aspects remained near chance at 50–55%. Visual coverage was reported as slightly more reliable, which the paper attributes to MLLMs’ pedigree in visual question answering and image captioning. Faithfulness proved particularly difficult: both summary-level and sentence-level balanced accuracy were below 30%.

MDSEval also identifies three recurrent bias patterns. Score Distribution Bias refers to concentration of judgments in a narrow range. Positional Bias refers to systematic preference for “Option A” or “Option B” in pairwise settings. Modality Translation Loss refers to the degradation induced by converting images into text prompts, thereby losing critical visual nuances. These observations are part of the benchmark’s core contribution, because they specify not merely that present evaluators underperform, but how they fail.

The broader significance of MDSEval lies in its calibration function for future multimodal metrics. The benchmark argues that effective multimodal evaluation requires score-distribution regularization, bias mitigation, and deeper cross-modal reasoning. The future directions it enumerates are correspondingly concrete: training evaluation models explicitly on the eight dimensions, incorporating bias-correction techniques such as fair preference prompting and positional randomization, extending the benchmark to other dialogue genres and richer modalities, and exploring architectures that jointly attend to image and dialogue context rather than relying on pipeline conversion (Liu et al., 2 Oct 2025).

Within the evaluation landscape, MDSEval therefore serves as a benchmark for benchmarkers: it is a human-grounded instrument for determining whether automatic MDS evaluators can distinguish fine-grained quality differences, capture modality-specific failures, and align with expert assessment in image-sharing conversations.