Dialogue-Level Multi-Dimensional Evaluation Metrics

• The paper demonstrates that modeling dialogue as a multi-turn, multi-dimensional interaction yields evaluation metrics that correlate more strongly with human judgments than traditional BLEU/ROUGE methods.
• It outlines decomposing dialogue quality into interpretable sub-metrics—such as coherence, engagement, and safety—to enable configurable evaluation tailored for diverse dialogue systems.
• It emphasizes challenges and future directions, including enhancing multilingual generalization, calibrating LLM-based evaluators, and improving robustness against adversarial perturbations.

Dialogue-level, multi-dimensional automatic evaluation metrics are automatic methods that estimate the quality of an entire conversation along multiple human-relevant axes, rather than scoring isolated responses or relying on surface word overlap. The paradigm emerged from the observation that standard metrics such as BLEU and ROUGE correlate poorly with human judgements in dialogue, especially for open-domain systems, and that holistic dialogue quality is not exhausted by a single scalar notion of appropriateness. Contemporary formulations therefore model dialogue-level properties such as coherence, relevance, engagement, informativeness, user satisfaction, empathy, safety, and overall quality, either as separate outputs or as structured components of a composed score (Zhang et al., 2022, Zhang et al., 2022, Mendonça et al., 16 Sep 2025).

1. Conceptual scope and problem formulation

Dialogue evaluation shifted from response matching toward learned assessment because open-domain dialogue admits many valid continuations with little lexical overlap. Early learned evaluators such as ADEM formulated automatic evaluation as a supervised prediction problem over dialogue context, reference response, and candidate response, with appropriateness as the primary target (Lowe et al., 2017). Later reference-free evaluators replaced reference matching with context–response scoring and semi-supervised training, achieving stronger correlations with human judgement and better robustness to missing references and cross-corpus transfer (Zhao et al., 2020).

The dialogue-level, multi-dimensional setting introduces two additional requirements. First, the evaluator must model the conversation as a multi-turn interaction rather than an average of turn-local scores. DynaEval made this point explicitly by arguing that effective evaluation metrics should reflect the dynamics of multi-turn interaction, not only turn-level quality (Zhang et al., 2021). Second, the evaluator must distinguish among different qualities. The comprehensive benchmark of 23 metrics over 10 datasets evaluated metrics at turn level and dialog level, across different dialog lengths, qualities such as coherence and engagement, and different model families, showing that metric behavior depends strongly on evaluation regime and target quality (Yeh et al., 2021).

This conceptualization was formalized in benchmark design. DSTC12 Track 1 defined a subtask on “Dialogue-level, Multi-dimensional Automatic Evaluation Metrics” centered on ten dialogue dimensions: comprehensibility, relevance, engagement, consistency, fluency, informativeness, politeness, safety/toxicity, humanness, and overall quality or user satisfaction (Mendonça et al., 16 Sep 2025). A recurrent misconception is that dialogue-level evaluation can be obtained simply by averaging utterance-level metrics. The DSTC12 overview identifies this averaging strategy as a common baseline, but also emphasizes that it often misses longer-range dependencies, topic continuity, and context-sensitive issues (Mendonça et al., 16 Sep 2025).

2. Decomposed and compositional metric architectures

A major design line decomposes dialogue quality into interpretable sub-metrics and then composes them. USL-H grouped qualities into understandability, sensibleness, and likability, with likability defined as a task-specific combination of qualities such as specificity, empathy, or diversity. Its hierarchical composition encodes the assumption that if a response is not understandable, higher-level qualities are irrelevant; the simplified form is

$$s_{USL\text{-}H} = \alpha_1 s_U + \alpha_2 s_S + \alpha_3 s_S s_L,$$

with

$$s_L = \sum_j \beta_j q_j, \qquad \sum_j \beta_j = 1.$$

This made configurability explicit: specificity can be emphasized in food-ordering dialogue, whereas empathy can be substituted for support dialogue (Phy et al., 2020).

FineD-Eval also adopted decomposition, but at dialogue level and with self-supervised training. It selected three relatively independent dimensions—coherence, likability, and topic depth—after analyzing fine-grained human ratings on FED. Each sub-metric was trained with a dimension-specific objective, and the three were combined either by metric ensemble or multitask learning. The combined metrics significantly outperformed individual sub-metrics and achieved around 16% relative improvement on average across three dialogue-level benchmarks (Zhang et al., 2022).

MME-CRS generalized decomposition further by constructing five groups of parallel sub-metrics—fluency, relevance, topic coherence, engagement, and specificity—and then learning a data-driven composition rule through Correlation Re-Scaling. For each quality $q_j$, CRS computes non-negative weights from development-set Spearman correlations and composes the final score as

$S_{ij} = \sum_k w_{jk}\cdot S_{ijk}.$

The framework ranked first in the DSTC10 Track 5 Automatic Open-domain Dialogue Evaluation Challenge and achieved an average Spearman correlation of 31.04% across five test datasets, compared with 27.55% for simple averaging and 18.38% for Deep AM-FM (Zhang et al., 2022).

Framework	Dimensions or groups	Composition mechanism
USL-H	Understandability, Sensibleness, Likability	Hierarchical composition with configurable $\alpha_i$ and $\beta_j$
FineD-Eval	Coherence, Likability, Topic Depth	Ensemble averaging or multitask learning
MME-CRS	Fluency, Relevance, Topic Coherence, Engagement, Specificity	Correlation Re-Scaling from development-set correlations

These systems share a common principle: overall dialogue quality is not treated as primitive, but as a structured function of sub-qualities. A plausible implication is that multidimensionality is not only descriptive but operational, because composition itself becomes part of the metric design.

3. Graph, ranking, and interaction-based modeling of dialogue structure

Another design line models discourse structure and conversational dynamics directly. GRADE proposed that topic transition dynamics are better captured by a topic-level dialogue graph than by surface similarity or utterance-level semantics alone. It combines utterance-level BERT representations with graph representations derived from keywords, ConceptNet embeddings, k-hop neighbors, and hop-attention over topic links, and then predicts coherence from the fused representation (Huang et al., 2020). The method was trained with margin ranking loss using hard negatives, and it outperformed prior metrics on Pearson and Spearman correlation with human judgements for dialogue coherence (Huang et al., 2020).

DynaEval extended graph modeling from topic transitions to dialogue interaction structure. It represents each utterance as a graph node and encodes speaker dependency and temporal dependency as typed edges in a directed utterance graph. A relation-aware GCN and a contrastive loss distinguish original dialogues from corrupted ones generated by utterance replacement or speaker-level utterance shuffling (Zhang et al., 2021). On FED, DynaEval attained the highest dialogue-level correlation in 8 of 11 aspects and the highest turn-level correlation in 5 of 9 aspects, illustrating that explicit modeling of inter-utterance dynamics can unify turn-level and dialogue-level evaluation (Zhang et al., 2021).

QuantiDCE addressed a narrower but important problem: dialogue coherence is typically annotated on a Likert scale, whereas many automatic metrics are trained only for binary coherent-versus-incoherent discrimination. It introduced a two-stage framework consisting of Multi-Level Ranking pre-training and Knowledge Distillation fine-tuning. The pre-training stage learns coarse coherence degrees from automatically constructed multi-level data, while the fine-tuning stage uses a small amount of human annotation and a distillation regularizer to align with human rating standards without overfitting (Ye et al., 2021). On ConvAI2 and EmpatheticDialogues, QuantiDCE outperformed GRADE and substantially surpassed BLEU, ROUGE, METEOR, ADEM, and BERT-RUBER on average correlation (Ye et al., 2021).

Taken together, these approaches define dialogue-level evaluation as a structured inference problem over discourse topology, speaker interaction, or rating-scale calibration. They differ from purely compositional systems in where the structure is encoded: in the dialogue representation itself rather than only in the score aggregation layer.

4. Human-centered proxy targets: satisfaction, user reaction, empathy, and task success

A separate trajectory evaluates dialogue quality through human-centered proxy variables. In multi-domain conversation quality evaluation, user satisfaction was estimated from domain-independent behavioral features and a new Response Quality annotation scheme. Response Quality rates each system response on a 5-point scale while considering the turn in full dialogue context but without requiring annotators to track cumulative satisfaction. The scheme achieved Spearman’s $\rho = 0.94$ inter-annotator agreement and $\rho = 0.76$ correlation with explicit turn-level user ratings, compared with $\rho = 0.32$ for Interaction Quality (Bodigutla et al., 2019). Using five new feature sets—user request paraphrasing indicators, cohesion between request and response, diversity of topics, un-actionable user request indicators, and aggregate topic/domain popularity—a Gradient Boosting Regression model achieved linear correlation around 0.79 on seen domains and 0.67 on a new multi-turn domain; at dialogue level, adding predicted turn-level satisfaction as a feature improved satisfactory/dissatisfactory accuracy from 68% to 79% (Bodigutla et al., 2019).

The paper on user response and sentiment prediction proposed a related idea: use the sentiment of the next user utterance for turn- or dialog-level evaluation. It described three methods, including direct next-sentiment prediction and two variants that first predict the next user utterance and then classify its sentiment, with experiments reported to outperform existing automatic evaluation metrics on written and spoken open-domain dialogue datasets (Ghazarian et al., 2021).

Empathy evaluation introduced an explicitly two-perspective multidimensional design. The framework for empathetic dialog responses distinguished between expressed empathy from the speaker’s perspective and perceived empathy from the listener’s perspective. Perceived empathy was decomposed into engagement, understanding, sympathy, and helpfulness, annotated on a Likert scale, and these dimensions were found to correlate significantly with overall conversation satisfaction (Xu et al., 2024). Automatic measurement via prompting with GPT-4 and Flan-family models performed relatively poorly, whereas instruction-finetuned Flan-T5 classifiers consistently achieved the best Macro-F1 and accuracy across public and internal datasets (Xu et al., 2024).

For task-oriented dialogue, TD-EVAL combined turn-level precision with dialogue-level comparisons. Each turn is scored on conversation cohesion, backend knowledge consistency, and policy compliance on a 5-point Likert scale by an LLM-as-judge, while holistic dialogue quality is obtained through pairwise comparisons in “TOD Agent Arena” with Elo-style ranking updates (Acikgoz et al., 28 Apr 2025). Against traditional metrics and an adapted LMUnit baseline, TD-EVAL achieved the highest agreement with human judgements at both turn and dialogue level, with dialogue-level Gwet AC1 of 0.57 and Randolph’s $\kappa$ of 0.52 (Acikgoz et al., 28 Apr 2025).

These methods reframe evaluation dimensions in terms of downstream user state, perceived relational quality, or policy-faithful task execution. This suggests that multidimensional evaluation is not limited to linguistic well-formedness and coherence, but also includes interactional outcomes.

5. LLMs as multidimensional evaluators

LLMs made multidimensional dialogue evaluation more operationally uniform by turning dimension definitions into prompts. LLM-Eval proposed a single prompt-based method with an embedded unified evaluation schema that scores multiple dimensions in one model call. Its main experimental dimensions were appropriateness, content, grammar, and relevance, with extensibility to other dimensions such as engagement and coherence depending on dataset (Lin et al., 2023). The schema-based design supports both reference-free and reference-based evaluation and dialogue-level assessment, returning structured outputs such as JSON scores on either a 0–5 or 0–100 scale (Lin et al., 2023). Across a range of datasets, LLM-Eval was reported to very consistently outperform baselines including BLEU, ROUGE, BERTScore, DEB, GRADE, USR, USL-H, DynaEval, FlowScore, GPTScore, and MME-CRS (Lin et al., 2023).

A larger comparative study examined 30 LLMs over 12 meta-evaluation datasets at both turn and dialogue levels. Proprietary instruction-tuned models such as ChatGPT and Palm-2 were the strongest overall, GPT-4 outperformed all other models on reported subsets, and instruction-tuned open-source 13B models were the best among open models but remained behind proprietary systems (Zhang et al., 2023). The study also found that dimension-wise ensembling can improve prediction of overall quality; for example, averaging dimension scores for Chimera-13B improved dialogue-level overall correlation from 0.563 to 0.680 (Zhang et al., 2023). However, robustness remained incomplete: no model was robust to all adversarial perturbations, and blind spots were especially apparent for dullness, pronoun or named-entity replacements, and several dialogue-level engagingness or informativeness attacks (Zhang et al., 2023).

LLM-Rubric introduced a calibrated variant of LLM-based multidimensional evaluation. A manually constructed rubric defines several questions, including naturalness, citation presence and quality, conciseness, redundancy, efficiency, and overall user satisfaction. An LLM is prompted separately with each rubric question to obtain a distribution over responses, and a small feed-forward network with judge-specific and judge-independent parameters combines these distributions to predict each human judge’s annotations, including the summary question (Hashemi et al., 2024). In a human-AI information-seeking task, LLM-Rubric with nine questions predicted overall user satisfaction on a 1–4 scale with RMS error below 0.5, a twofold improvement over the uncalibrated baseline (Hashemi et al., 2024).

Results from DSTC12 system submissions reinforced both the promise and fragility of LLM judging. Under a constraint of fewer than 13 billion parameters, prompting-based evaluators achieved only modest correlations with human judgements but still ranked second on the test set, while smaller regression and classification models obtained stronger validation correlations on some dimensions and then degraded on the test set because score ranges differed substantially across splits (Elizabeth et al., 31 Aug 2025). The general lesson is that LLM-based multidimensional evaluation can be strong, efficient, and schema-driven, but it remains sensitive to model choice, decoding, prompt design, and calibration regime (Lin et al., 2023, Zhang et al., 2023).

6. Benchmarks, multilingual generalization, and meta-evaluation

Benchmarking has become central because metric performance varies sharply with data regime. The large comparative study of dialog evaluation metrics concluded that no single universal metric exists, that metric choice should depend on dialog length, domain, and model type, and that ensembles of diverse metrics often perform as well as or better than any single metric (Yeh et al., 2021). It also observed that dialog-level specialists such as FlowScore, PredictiveEngage, DynaEval, and FED can outperform turn-level metrics on datasets with long contexts and holistic annotations (Yeh et al., 2021).

Multilingual generalization was addressed by xDial-Eval, a benchmark built from open-source English dialogue evaluation datasets and extended to nine additional languages by machine translation. It contains 12 turn-level and 6 dialogue-level English datasets, totaling 14,930 annotated turns and 8,691 annotated dialogues, with dialogue-level analysis concentrating on coherence while preserving other dataset-specific dimensions such as engagement, consistency, appropriateness, fluency, empathy, and overall quality (Zhang et al., 2023). On average Pearson correlation over all datasets and languages, the best baseline outperformed ChatGPT by absolute improvements of 6.5% at turn level and 4.6% at dialogue level, despite using far fewer parameters (Zhang et al., 2023). Ensemble combinations of multilingual BERT-based and LLM-based metrics yielded the best dialogue-level performance, reaching 0.422 average Pearson correlation compared with 0.376 for ChatGPT (Zhang et al., 2023).

DSTC12 made multidimensional dialogue-level evaluation itself the object of competition. The track focused on ten dialogue dimensions and reported that a Llama-3-8B baseline achieved the highest average Spearman correlation of 0.1681 in Task 1, indicating substantial room for improvement (Mendonça et al., 16 Sep 2025). The same overview linked evaluation to language, culture, and safety, and its second subtask on multilingual and multicultural safety detection showed that strong multilingual safety performance did not automatically transfer to the cultural subset (Mendonça et al., 16 Sep 2025). A plausible implication is that future dialogue-level evaluation benchmarks will increasingly couple multidimensional quality scoring with multilinguality and culturally sensitive safety.

7. Persistent limitations and research frontiers

Several limitations recur across the literature. The DSTC12 overview identifies insufficient dialogue context modeling, subjectivity and label noise, dimension overlap, data and domain limitations, and weak metric generalizability across systems or conversational styles (Mendonça et al., 16 Sep 2025). These are not benchmark-specific. The comprehensive assessment of metrics likewise found strong dependence on training data similarity, degradation with longer dialog contexts for many BERT-based reference-free metrics, and different behavior across generative versus retrieval systems (Yeh et al., 2021).

LLM-based evaluators add new failure modes. LLM-Eval explicitly notes model dependence, prompt or schema sensitivity, possible bias, and the fact that single-number scores may miss subtleties in subjective dimensions (Lin et al., 2023). The large-scale LLM analysis shows that even strong evaluators remain vulnerable to adversarial perturbations and that robustness is far from solved (Zhang et al., 2023). Smaller-model DSTC12 submissions further show that annotation inconsistencies and score-range drift between train, validation, and test splits can collapse generalization even when within-split correlations are high (Elizabeth et al., 31 Aug 2025).

Another unresolved issue concerns the relation among dimensions. FineD-Eval selected dimensions partly by inter-dimensional independence, whereas DSTC12 reports that some dimensions are not clearly delineated, such as fluency versus comprehensibility (Zhang et al., 2022, Mendonça et al., 16 Sep 2025). MME-CRS, by contrast, assumes that some qualities are best assessed by combining multiple sub-metrics, as when specificity also depends on fluency and relevance (Zhang et al., 2022). The field therefore does not yet converge on whether dimensions should be maximally disentangled, hierarchically dependent, or compositionally overlapping.

Current research directions follow directly from these tensions. Better holistic modeling of dialogue, richer and more consistent annotation protocols, stronger multilingual and cross-domain benchmarks, more robust and calibrated LLM judges, and unified yet flexible frameworks for overlapping qualities are all recurrent priorities (Mendonça et al., 16 Sep 2025, Zhang et al., 2023, Hashemi et al., 2024). Dialogue-level, multi-dimensional automatic evaluation has thus evolved from a critique of BLEU-style scoring into a research area concerned simultaneously with representation, aggregation, calibration, robustness, and human interpretability.