---
title: Automatic Evaluation Metrics (AEM)
url: https://www.emergentmind.com/topics/automatic-evaluation-metrics-aem
type: topic
---

# Automatic Evaluation Metrics (AEM)

Automatic evaluation metrics (AEMs) are methods for automatically assessing the quality of generated outputs. In the literature summarized here, they are described as tools that compare a system output with a human reference, assess consistency between source and target without a reference, or, in process-aware settings, aggregate multi-step evaluations across heterogeneous workflows rather than produce only a single response score [2504.14804][2601.11903]. Their role spans natural language generation, dialog, summarization, image captioning, speech generation, image transcreation, medical note generation, re-speaking, document-level translation, and interpreting, yet no single metric has emerged as a definitive solution; instead, reported usefulness depends on task, dataset, quality dimension, and validation methodology [2508.13816].

## 1. Conceptual scope and major families

A recurring taxonomy divides AEMs along two overlapping axes. One axis distinguishes **reference-based** evaluation from **reference-free** evaluation: reference-based methods compare a candidate against one or more human references, whereas reference-free methods assess consistency between source and output directly or score the output without a gold reference [2504.14804]. The second axis groups metrics into **lexical similarity**, **semantic similarity or model-based metrics**, and **LLM-based evaluation** [2508.13816].

Lexical metrics compare surface overlap between generated and reference text. Representative examples in the surveyed literature include **BLEU**, **ROUGE**, **METEOR**, **TER**, **CIDEr**, and **SPICE**. Their advantages are simple calculation and fast speed, but they mainly evaluate lexical matching and have a weaker ability to understand semantics [2504.14804]. Semantic and model-based metrics move to embeddings or learned scoring functions. The literature summarized here names **WMD**, **SMS/SMD**, **WISDM**, **BERTScore**, **BLEURT**, **BEM**, **COMET**, **MoverScore**, **BARTScore**, and domain-specific variants such as **MIST**, **MedBARTScore**, **MedBERTScore**, and **SpeechBERTScore** [2508.13816][2305.17364][2401.16812]. LLM-based evaluation includes rubric-based scoring, pairwise preference judgments, and long-form evaluators such as **GPTScore**, **G-EVAL**, **Prometheus**, and **AlpacaEval-LC**, while process-aware frameworks such as **AEMA** treat evaluation itself as a multi-stage orchestration problem [2508.13816][2601.11903].

A plausible implication is that “AEM” now denotes not only a scalar metric but also a broader evaluation apparatus. This is explicit in work on agentic systems, where the evaluation object is a structured workflow with planning, tool use, validation, and reporting, and in adaptive metric-construction frameworks that synthesize task-specific evaluators from sparse human feedback [2601.11903][2512.17267].

| Family | Representative examples | Primary emphasis |
|---|---|---|
| Lexical similarity | BLEU, ROUGE, METEOR, TER, CIDEr, SPICE | Surface overlap and string matching |
| Semantic/model-based | BERTScore, BLEURT, COMET, MoverScore, WMD, MIST, SpeechBERTScore | Embeddings, learned scoring, semantic or concept similarity |
| LLM-based/process-aware | GPTScore, G-EVAL, Prometheus, AlpacaEval-LC, AEMA, AutoMetrics | Rubric-based judging, adaptive composition, multi-step auditing |

## 2. Validation, meta-evaluation, and the problem of what metrics really measure

The dominant validation convention is to correlate metric outputs with human judgments, but several papers argue that this is insufficient or methodologically optimistic. In summarization, standard system-level correlation is usually defined by averaging automatic and human scores over the human-judged subset and correlating those system means across systems. Re-examination of this practice shows two mismatches with actual use: metrics are validated on the judged subset although systems are scored on the full test set in papers, and correlations are computed over many easy system pairs although practical comparisons usually involve very small score differences [2204.10216]. The standard system-level correlation is written as
$$
sys = Corr\left(\left\{\left(\frac{1}{jud}\sum_j^{jud} x^j_i, \frac{1}{jud}\sum_j^{jud} z^j_i\right)\right\}_{i=1}^N\right),
$$
whereas the revised formulation computes automatic system scores on the full test set:
$$
r_{sys} = Corr\left( \left\{ \left( \frac{1}{test}\sum_{j=1}^{test} x_i^j,\; \frac{1}{jud}\sum_{j=1}^{jud} z_i^j \right) \right\}_{i=1}^N \right).
$$
Under this correction, and especially when attention is restricted to close system pairs through \(sys\Delta(\ell,u)\), the estimated usefulness of common metrics drops sharply; for realistic score differences, the best estimate of **ROUGE’s** correlation with human judgments is near \(0\) [2204.10216].

A second line of critique argues that human evaluation is itself multi-dimensional, so correlation with one criterion is a weak validation target. Across 6 NLG tasks, 25 metrics, and 18 criteria, human scores on different criteria are often only weakly or moderately correlated, and there is no single metric that correlates well with all desirable criteria for most tasks. The proposed remedy is **Perturbation CheckLists**, which apply criterion-targeted perturbations and compare human and metric responses at a fine-grained level [2109.05771]. A third line of work studies metric similarity directly in the space of rankings rather than raw scores. On 9 datasets across 4 NLG tasks, automatic metrics are reported to be much more similar to one another than to human metrics, and human metrics predict each other better than the combination of all automatic metrics predicts a human metric, which is the basis for the “glass ceiling” claim [2208.14585].

Meta-evaluation has also been generalized beyond human-score correlation. For corpus-level semantic similarity, one proposal builds **known similarity corpora** by mixing source corpora in controlled proportions and then measures whether a corpus-level metric is monotonic, separable, linear, sample-efficient, and robust to size imbalance [2211.16259]. This suggests that AEM research increasingly distinguishes between evaluating models and evaluating the evaluators themselves.

## 3. Task-specific developments in text generation and dialogue

In summarization, one major development is the shift from opaque similarity scores to explicitly interpretable content-unit scoring. The **AutoACU** framework decomposes a reference summary into **Atomic Content Units (ACUs)**, checks each ACU against the candidate with an NLI model, and aggregates the results into a recall-style summary score:
$$
R(S_2 \mid S_1) = \frac{\sum_{a \in A} l_a}{|A|}.
$$
The two-stage **A2CU** metric offers fine-grained and summary-level interpretability, while the one-stage **A3CU** approximation predicts the ACU-based score directly and requires only about \(6\%\) of A2CU’s inference time [2303.03608]. This line of work is important because it treats interpretability and efficiency as first-class design goals rather than afterthoughts.

Dialog evaluation has produced a large ecosystem of specialized metrics, but comparative evidence remains strongly setting-dependent. A comprehensive assessment of 23 dialog metrics on 10 datasets reports that **USR, GRADE, DEB, and USL-H** are the strongest metrics for turn-level response evaluation, whereas **FlowScore, PredictiveEngage, DynaEval, and FED** perform best on dialog-level evaluation in FED and DSTC9. The same study shows that metric effectiveness changes with dialog length, target quality dimension, and whether the systems are retrieval-based or generative [2106.03706]. A related multilingual and robustness-oriented benchmark in DSTC11 Track 4 reframes dialog AEM as a problem of cross-language and perturbation-invariant evaluation; the winning system combined an **XLM-R** submetric group with a **ChatGPT** submetric group and substantially outperformed the provided multilingual **deep AM-FM** baseline on multilingual turn-level and dialog-level tasks [2306.12794].

At a broader corpus scale, the question is not only whether a response matches its reference but whether two corpora exhibit similar semantic distributions. In that setting, **MAUVE** and **DC** are reported to behave more like semantic distributional metrics, whereas **CHI** and **ZIPF** are fast and highly surface-sensitive [2211.16259]. This suggests that AEM design diverges substantially once the evaluation target changes from a single instance to a distribution of instances.

## 4. Multimodal, speech, and specialized-domain expansions

Image captioning has been a major testbed for both classical and learned AEMs. **LCEval** addresses a long-standing gap between system-level and caption-level evaluation by learning a composite metric from lexical, semantic, and syntactic similarity features. It is trained as a binary classifier that distinguishes human-generated from machine-generated captions, and its inference score is the probability that a caption is human-generated. On COMPOSITE, the reported caption-level Kendall correlation is \(0.418\), above **SPIDEr** at \(0.357\) and **NNEval** at \(0.396\), while system-level correlation also remains strong [2012.13136]. The paper’s feature ablations further show that lexical features are the backbone of the learned metric, semantic features add a meaningful boost, and syntactic features are complementary rather than primary.

Image transcreation extends AEM beyond text by treating evaluation as explicitly multidimensional. One framework defines three dimensions—**cultural relevance**, **semantic equivalence**, and **visual similarity**—and instantiates them with **object-based**, **embedding-based**, and **VLM-based** metrics. On a benchmark spanning 7 countries and 3543 instances, the best metrics are dimension-specific: **Gemini-1.5-Pro** is strongest on cultural relevance with average segment-level Kendall correlation \(0.81\), **GPT-4o** is strongest on semantic equivalence with \(0.86\), and **SigLIP** is strongest on visual similarity with \(0.87\) [2412.13717]. The paper’s operational conclusion is a hybrid strategy rather than a single score.

Speech evaluation shows an analogous transfer from NLP metrics into a new modality. **SpeechBERTScore** computes BERTScore-style precision over self-supervised speech embeddings:
$$
\text{SpeechBERTScore} = \frac{1}{N_{\mathrm{gen}}}\sum_{i=1}^{N_{\mathrm{gen}}} \max_j \cos(\hat{z}_i, z_j),
$$
with the practical advantage that generated and reference speech may have different lengths. On SOMOS, the reported correlations are \(0.581\) LCC and \(0.563\) SRCC at utterance level, above **MCD** and **UTMOS**; on noisy speech, **SpeechBERTScore** reaches \(0.824\) LCC and \(0.868\) SRCC and has the highest SRCC among the compared metrics [2401.16812].

In highly specialized domains, AEM design often becomes task-specific. For **re-speaking**, a comparative study of **BLEU, EBLEU, NIST, METEOR, METEOR-PL, TER, and RIBES** uses backward linear regression against the human **NER** metric and reports that **BLEU, NIST, and EBLEU** are the significant predictors in the final model, with adjusted \(R^2 = 0.761\) [1601.02789]. For **medical note generation**, evaluation is organized around **factual precision**, **factual recall**, **hallucination rate**, and **omission rate**, with new metrics such as **MIST**, **MedBARTScore**, **MedBERTScore**, **ClinicalBLEURT**, and ensemble combinations. Across seven annotated datasets, the most stable subset is **MIST-Comb1**, **MIST-Comb2**, **BERTScore-R**, and **ROUGE-1-R**, while **MedBARTScore** is especially useful for factual precision and hallucination sensitivity [2305.17364].

## 5. Fairness, robustness, interpretability, and trustworthy evaluation

AEMs are increasingly treated as objects of fairness and trust analysis in their own right. For reference-based NLG metrics, one paper adapts **WEAT** and **SEAT** to the metric-scoring setting by defining a symmetrized matching score
$$
S(x,y)=\frac{1}{2}(M(x,y)+M(y,x)),
$$
and then computing differential associations over target and attribute sets. The main empirical result is that model-based metrics such as **BARTScore**, **BLEURT**, **BERTScore**, **MoverScore**, and **WMD** widely exhibit nontrivial and often statistically significant social associations, while most n-gram metrics stay near zero. In image captioning and summarization meta-evaluation, gender swapping further shows that model-based metrics may prefer male hypotheses when references are gender-neutral, and metric-human correlations can shift under swapping [2210.08859].

A related fairness audit focuses on image captioning and shows that **CLIPScore** is biased on over half of the tested profession, activity, and object concepts in **PAO-EvalBias**, while **BLEU-4**, **METEOR**, **ROUGE**, **CIDEr**, and **SPICE** are not biased under that controlled setup. The paper also demonstrates two downstream harms: biased metrics fail to distinguish biased from debiased generations, and using **CLIPScore** as an RL reward increases gender prediction error. A simple hybrid metric,
$$
\mathcal{H}(c_i, r_i, I_i)=\text{CLIPScore}(c_i, I_i)+\text{CIDEr}(c_i, r_i),
$$
eliminates the benchmarked bias signal in that study and improves correlation with human judgments on Flickr8K-Expert [2305.14711].

Interpretability and control motivate a different strand of work. **AEMA** treats evaluation of multi-agent LLM systems as a planned, multi-stage, auditable process with four coordinated roles: **Planning Agent**, **Prompt-Refinement Agent**, **Evaluation Agents**, and **Final Report Agent**. It records plan creation, parameter refinement, scoring, and reporting as a verifiable artifact, and in the reported finance-domain experiments it shows lower dispersion and closer agreement with human judges than a fair single-LLM baseline, with average absolute error \(0.018\) versus \(0.077\) on good-quality invoices and \(0.037\) versus \(0.108\) on blurry invoices [2601.11903]. **AutoMetrics** pursues a complementary goal: it synthesizes task-specific evaluators from sparse human feedback by generating candidate judge criteria, retrieving relevant metrics from **MetricBank**, and composing them with **Partial Least Squares** regression. Across five tasks, it reports improvements in Kendall correlation over LLM-as-a-Judge while using fewer than 100 feedback points, and it can also function as a proxy reward for optimization [2512.17267].

A plausible implication is that “trustworthy AEM” increasingly means more than a better scalar correlation. It includes fairness auditing, perturbation robustness, interpretable decomposition, auditable traces, and explicit control over what is being measured.

## 6. Limits, controversies, and future directions

Several surveys and critical papers converge on a shared conclusion: AEMs do not provide absolute performance measures. In document-level translation, the evaluation space is described as being constrained by lack of reference diversity, dependence on sentence-level alignment information, and the bias, inaccuracy, and lack of interpretability of LLM-as-a-Judge. The recommended future directions include reducing dependency on sentence-level information, introducing multi-level and multi-granular evaluation, and training models specifically for machine translation evaluation [2504.14804]. A broader synthesis reaches a similar conclusion for NLG at large: metrics at best highlight specific dimensions of text quality, their effectiveness varies by task and dataset, and the most recent type of metric, LLM-as-a-Judge, still inherits inconsistent human correlation, unstructured validation, and task dependence [2508.13816].

The strongest theoretical challenge to the “one best metric” program comes from two complementary directions. One is the “glass ceiling” argument that automatic metrics are much more similar to one another than to human metrics, are not complementary, and rank systems similarly; the other is the multi-criteria argument that NLG quality dimensions such as fluency, coherence, coverage, relevance, and adequacy are often not strongly correlated with one another, so validation against a single human criterion is inadequate [2208.14585][2109.05771]. This suggests that future AEM work will likely need to optimize not only raw human-correlation scores but also complementarity, criterion coverage, robustness under perturbation, and clarity about the level of evaluation—segment, document, dialog, system, or workflow.

Interpreting studies make the limitation especially explicit. A critical analysis of AEMs for interpreting argues that automatic metrics are “backwards-looking”: they ask whether the target sufficiently reflects the source, or how closely it matches an earlier translation, but they do not ask whether the interpreting met the needs of the people involved, fulfilled its communicative purpose, supported decision-making, enabled turn-taking, conveyed affect, or handled culturally loaded and pragmatically complex material appropriately. The proposed shift is “towards situated quality,” where evaluation is localized, tied to the actual situation, and meaningful to stakeholders [2601.05864]. This suggests a broader implication for AEM research: even technically strong metrics may remain partial instruments unless their assumptions align with the communicative setting they are meant to judge.

Source: https://www.emergentmind.com/topics/automatic-evaluation-metrics-aem