Automatic Evaluation Metrics (AEM)

• AEMs are methods that automatically assess the quality of generated outputs by comparing them to human references or by evaluating source-output consistency.
• They include lexical, semantic/model-based, and LLM-based approaches, each offering different trade-offs in speed, interpretability, and task suitability.
• Recent developments focus on fairness, robustness, and multi-stage interpretability, with adaptive and process-aware frameworks for specialized domains.

Automatic evaluation metrics (AEMs) are methods for automatically assessing the quality of generated outputs. In the literature summarized here, they are described as tools that compare a system output with a human reference, assess consistency between source and target without a reference, or, in process-aware settings, aggregate multi-step evaluations across heterogeneous workflows rather than produce only a single response score (Guo et al., 21 Apr 2025, Lee et al., 17 Jan 2026). Their role spans natural language generation, dialog, summarization, image captioning, speech generation, image transcreation, medical note generation, re-speaking, document-level translation, and interpreting, yet no single metric has emerged as a definitive solution; instead, reported usefulness depends on task, dataset, quality dimension, and validation methodology (Oliva et al., 19 Aug 2025).

1. Conceptual scope and major families

A recurring taxonomy divides AEMs along two overlapping axes. One axis distinguishes reference-based evaluation from reference-free evaluation: reference-based methods compare a candidate against one or more human references, whereas reference-free methods assess consistency between source and output directly or score the output without a gold reference (Guo et al., 21 Apr 2025). The second axis groups metrics into lexical similarity, semantic similarity or model-based metrics, and LLM-based evaluation (Oliva et al., 19 Aug 2025).

Lexical metrics compare surface overlap between generated and reference text. Representative examples in the surveyed literature include BLEU, ROUGE, METEOR, TER, CIDEr, and SPICE. Their advantages are simple calculation and fast speed, but they mainly evaluate lexical matching and have a weaker ability to understand semantics (Guo et al., 21 Apr 2025). Semantic and model-based metrics move to embeddings or learned scoring functions. The literature summarized here names WMD, SMS/SMD, WISDM, BERTScore, BLEURT, BEM, COMET, MoverScore, BARTScore, and domain-specific variants such as MIST, MedBARTScore, MedBERTScore, and SpeechBERTScore (Oliva et al., 19 Aug 2025, Abacha et al., 2023, Saeki et al., 2024). LLM-based evaluation includes rubric-based scoring, pairwise preference judgments, and long-form evaluators such as GPTScore, G-EVAL, Prometheus, and AlpacaEval-LC, while process-aware frameworks such as AEMA treat evaluation itself as a multi-stage orchestration problem (Oliva et al., 19 Aug 2025, Lee et al., 17 Jan 2026).

A plausible implication is that “AEM” now denotes not only a scalar metric but also a broader evaluation apparatus. This is explicit in work on agentic systems, where the evaluation object is a structured workflow with planning, tool use, validation, and reporting, and in adaptive metric-construction frameworks that synthesize task-specific evaluators from sparse human feedback (Lee et al., 17 Jan 2026, Ryan et al., 19 Dec 2025).

Family	Representative examples	Primary emphasis
Lexical similarity	BLEU, ROUGE, METEOR, TER, CIDEr, SPICE	Surface overlap and string matching
Semantic/model-based	BERTScore, BLEURT, COMET, MoverScore, WMD, MIST, SpeechBERTScore	Embeddings, learned scoring, semantic or concept similarity
LLM-based/process-aware	GPTScore, G-EVAL, Prometheus, AlpacaEval-LC, AEMA, AutoMetrics	Rubric-based judging, adaptive composition, multi-step auditing

2. Validation, meta-evaluation, and the problem of what metrics really measure

The dominant validation convention is to correlate metric outputs with human judgments, but several papers argue that this is insufficient or methodologically optimistic. In summarization, standard system-level correlation is usually defined by averaging automatic and human scores over the human-judged subset and correlating those system means across systems. Re-examination of this practice shows two mismatches with actual use: metrics are validated on the judged subset although systems are scored on the full test set in papers, and correlations are computed over many easy system pairs although practical comparisons usually involve very small score differences (Deutsch et al., 2022). The standard system-level correlation is written as

$$sys = Corr\left(\left\{\left(\frac{1}{jud}\sum_j^{jud} x^j_i, \frac{1}{jud}\sum_j^{jud} z^j_i\right)\right\}_{i=1}^N\right),$$

whereas the revised formulation computes automatic system scores on the full test set:

$$r_{sys} = Corr\left( \left\{ \left( \frac{1}{test}\sum_{j=1}^{test} x_i^j,\; \frac{1}{jud}\sum_{j=1}^{jud} z_i^j \right) \right\}_{i=1}^N \right).$$

Under this correction, and especially when attention is restricted to close system pairs through $sys\Delta(\ell,u)$, the estimated usefulness of common metrics drops sharply; for realistic score differences, the best estimate of ROUGE’s correlation with human judgments is near $0$ (Deutsch et al., 2022).

A second line of critique argues that human evaluation is itself multi-dimensional, so correlation with one criterion is a weak validation target. Across 6 NLG tasks, 25 metrics, and 18 criteria, human scores on different criteria are often only weakly or moderately correlated, and there is no single metric that correlates well with all desirable criteria for most tasks. The proposed remedy is Perturbation CheckLists, which apply criterion-targeted perturbations and compare human and metric responses at a fine-grained level (Sai et al., 2021). A third line of work studies metric similarity directly in the space of rankings rather than raw scores. On 9 datasets across 4 NLG tasks, automatic metrics are reported to be much more similar to one another than to human metrics, and human metrics predict each other better than the combination of all automatic metrics predicts a human metric, which is the basis for the “glass ceiling” claim (Colombo et al., 2022).

Meta-evaluation has also been generalized beyond human-score correlation. For corpus-level semantic similarity, one proposal builds known similarity corpora by mixing source corpora in controlled proportions and then measures whether a corpus-level metric is monotonic, separable, linear, sample-efficient, and robust to size imbalance (Kour et al., 2022). This suggests that AEM research increasingly distinguishes between evaluating models and evaluating the evaluators themselves.

3. Task-specific developments in text generation and dialogue

In summarization, one major development is the shift from opaque similarity scores to explicitly interpretable content-unit scoring. The AutoACU framework decomposes a reference summary into Atomic Content Units (ACUs), checks each ACU against the candidate with an NLI model, and aggregates the results into a recall-style summary score:

$R(S_2 \mid S_1) = \frac{\sum_{a \in A} l_a}{|A|}.$

The two-stage A2CU metric offers fine-grained and summary-level interpretability, while the one-stage A3CU approximation predicts the ACU-based score directly and requires only about $6\%$ of A2CU’s inference time (Liu et al., 2023). This line of work is important because it treats interpretability and efficiency as first-class design goals rather than afterthoughts.

Dialog evaluation has produced a large ecosystem of specialized metrics, but comparative evidence remains strongly setting-dependent. A comprehensive assessment of 23 dialog metrics on 10 datasets reports that USR, GRADE, DEB, and USL-H are the strongest metrics for turn-level response evaluation, whereas FlowScore, PredictiveEngage, DynaEval, and FED perform best on dialog-level evaluation in FED and DSTC9. The same study shows that metric effectiveness changes with dialog length, target quality dimension, and whether the systems are retrieval-based or generative (Yeh et al., 2021). A related multilingual and robustness-oriented benchmark in DSTC11 Track 4 reframes dialog AEM as a problem of cross-language and perturbation-invariant evaluation; the winning system combined an XLM-R submetric group with a ChatGPT submetric group and substantially outperformed the provided multilingual deep AM-FM baseline on multilingual turn-level and dialog-level tasks (Rodríguez-Cantelar et al., 2023).

At a broader corpus scale, the question is not only whether a response matches its reference but whether two corpora exhibit similar semantic distributions. In that setting, MAUVE and DC are reported to behave more like semantic distributional metrics, whereas CHI and ZIPF are fast and highly surface-sensitive (Kour et al., 2022). This suggests that AEM design diverges substantially once the evaluation target changes from a single instance to a distribution of instances.

4. Multimodal, speech, and specialized-domain expansions

Image captioning has been a major testbed for both classical and learned AEMs. LCEval addresses a long-standing gap between system-level and caption-level evaluation by learning a composite metric from lexical, semantic, and syntactic similarity features. It is trained as a binary classifier that distinguishes human-generated from machine-generated captions, and its inference score is the probability that a caption is human-generated. On COMPOSITE, the reported caption-level Kendall correlation is $0.418$, above SPIDEr at $0.357$ and NNEval at $0.396$, while system-level correlation also remains strong (Sharif et al., 2020). The paper’s feature ablations further show that lexical features are the backbone of the learned metric, semantic features add a meaningful boost, and syntactic features are complementary rather than primary.

Image transcreation extends AEM beyond text by treating evaluation as explicitly multidimensional. One framework defines three dimensions—cultural relevance, semantic equivalence, and visual similarity—and instantiates them with object-based, embedding-based, and VLM-based metrics. On a benchmark spanning 7 countries and 3543 instances, the best metrics are dimension-specific: Gemini-1.5-Pro is strongest on cultural relevance with average segment-level Kendall correlation $0.81$, GPT-4o is strongest on semantic equivalence with $$r_{sys} = Corr\left( \left\{ \left( \frac{1}{test}\sum_{j=1}^{test} x_i^j,\; \frac{1}{jud}\sum_{j=1}^{jud} z_i^j \right) \right\}_{i=1}^N \right).$$0, and SigLIP is strongest on visual similarity with $$r_{sys} = Corr\left( \left\{ \left( \frac{1}{test}\sum_{j=1}^{test} x_i^j,\; \frac{1}{jud}\sum_{j=1}^{jud} z_i^j \right) \right\}_{i=1}^N \right).$$1 (Khanuja et al., 2024). The paper’s operational conclusion is a hybrid strategy rather than a single score.

Speech evaluation shows an analogous transfer from NLP metrics into a new modality. SpeechBERTScore computes BERTScore-style precision over self-supervised speech embeddings:

$$r_{sys} = Corr\left( \left\{ \left( \frac{1}{test}\sum_{j=1}^{test} x_i^j,\; \frac{1}{jud}\sum_{j=1}^{jud} z_i^j \right) \right\}_{i=1}^N \right).$$2

with the practical advantage that generated and reference speech may have different lengths. On SOMOS, the reported correlations are $$r_{sys} = Corr\left( \left\{ \left( \frac{1}{test}\sum_{j=1}^{test} x_i^j,\; \frac{1}{jud}\sum_{j=1}^{jud} z_i^j \right) \right\}_{i=1}^N \right).$$3 LCC and $$r_{sys} = Corr\left( \left\{ \left( \frac{1}{test}\sum_{j=1}^{test} x_i^j,\; \frac{1}{jud}\sum_{j=1}^{jud} z_i^j \right) \right\}_{i=1}^N \right).$$4 SRCC at utterance level, above MCD and UTMOS; on noisy speech, SpeechBERTScore reaches $$r_{sys} = Corr\left( \left\{ \left( \frac{1}{test}\sum_{j=1}^{test} x_i^j,\; \frac{1}{jud}\sum_{j=1}^{jud} z_i^j \right) \right\}_{i=1}^N \right).$$5 LCC and $$r_{sys} = Corr\left( \left\{ \left( \frac{1}{test}\sum_{j=1}^{test} x_i^j,\; \frac{1}{jud}\sum_{j=1}^{jud} z_i^j \right) \right\}_{i=1}^N \right).$$6 SRCC and has the highest SRCC among the compared metrics (Saeki et al., 2024).

In highly specialized domains, AEM design often becomes task-specific. For re-speaking, a comparative study of BLEU, EBLEU, NIST, METEOR, METEOR-PL, TER, and RIBES uses backward linear regression against the human NER metric and reports that BLEU, NIST, and EBLEU are the significant predictors in the final model, with adjusted $$r_{sys} = Corr\left( \left\{ \left( \frac{1}{test}\sum_{j=1}^{test} x_i^j,\; \frac{1}{jud}\sum_{j=1}^{jud} z_i^j \right) \right\}_{i=1}^N \right).$$7 (Wołk et al., 2016). For medical note generation, evaluation is organized around factual precision, factual recall, hallucination rate, and omission rate, with new metrics such as MIST, MedBARTScore, MedBERTScore, ClinicalBLEURT, and ensemble combinations. Across seven annotated datasets, the most stable subset is MIST-Comb1, MIST-Comb2, BERTScore-R, and ROUGE-1-R, while MedBARTScore is especially useful for factual precision and hallucination sensitivity (Abacha et al., 2023).

5. Fairness, robustness, interpretability, and trustworthy evaluation

AEMs are increasingly treated as objects of fairness and trust analysis in their own right. For reference-based NLG metrics, one paper adapts WEAT and SEAT to the metric-scoring setting by defining a symmetrized matching score

$$r_{sys} = Corr\left( \left\{ \left( \frac{1}{test}\sum_{j=1}^{test} x_i^j,\; \frac{1}{jud}\sum_{j=1}^{jud} z_i^j \right) \right\}_{i=1}^N \right).$$8

and then computing differential associations over target and attribute sets. The main empirical result is that model-based metrics such as BARTScore, BLEURT, BERTScore, MoverScore, and WMD widely exhibit nontrivial and often statistically significant social associations, while most n-gram metrics stay near zero. In image captioning and summarization meta-evaluation, gender swapping further shows that model-based metrics may prefer male hypotheses when references are gender-neutral, and metric-human correlations can shift under swapping (Gao et al., 2022).

A related fairness audit focuses on image captioning and shows that CLIPScore is biased on over half of the tested profession, activity, and object concepts in PAO-EvalBias, while BLEU-4, METEOR, ROUGE, CIDEr, and SPICE are not biased under that controlled setup. The paper also demonstrates two downstream harms: biased metrics fail to distinguish biased from debiased generations, and using CLIPScore as an RL reward increases gender prediction error. A simple hybrid metric,

$$r_{sys} = Corr\left( \left\{ \left( \frac{1}{test}\sum_{j=1}^{test} x_i^j,\; \frac{1}{jud}\sum_{j=1}^{jud} z_i^j \right) \right\}_{i=1}^N \right).$$9

eliminates the benchmarked bias signal in that study and improves correlation with human judgments on Flickr8K-Expert (2305.14711).

Interpretability and control motivate a different strand of work. AEMA treats evaluation of multi-agent LLM systems as a planned, multi-stage, auditable process with four coordinated roles: Planning Agent, Prompt-Refinement Agent, Evaluation Agents, and Final Report Agent. It records plan creation, parameter refinement, scoring, and reporting as a verifiable artifact, and in the reported finance-domain experiments it shows lower dispersion and closer agreement with human judges than a fair single-LLM baseline, with average absolute error $sys\Delta(\ell,u)$0 versus $sys\Delta(\ell,u)$1 on good-quality invoices and $sys\Delta(\ell,u)$2 versus $sys\Delta(\ell,u)$3 on blurry invoices (Lee et al., 17 Jan 2026). AutoMetrics pursues a complementary goal: it synthesizes task-specific evaluators from sparse human feedback by generating candidate judge criteria, retrieving relevant metrics from MetricBank, and composing them with Partial Least Squares regression. Across five tasks, it reports improvements in Kendall correlation over LLM-as-a-Judge while using fewer than 100 feedback points, and it can also function as a proxy reward for optimization (Ryan et al., 19 Dec 2025).

A plausible implication is that “trustworthy AEM” increasingly means more than a better scalar correlation. It includes fairness auditing, perturbation robustness, interpretable decomposition, auditable traces, and explicit control over what is being measured.

6. Limits, controversies, and future directions

Several surveys and critical papers converge on a shared conclusion: AEMs do not provide absolute performance measures. In document-level translation, the evaluation space is described as being constrained by lack of reference diversity, dependence on sentence-level alignment information, and the bias, inaccuracy, and lack of interpretability of LLM-as-a-Judge. The recommended future directions include reducing dependency on sentence-level information, introducing multi-level and multi-granular evaluation, and training models specifically for machine translation evaluation (Guo et al., 21 Apr 2025). A broader synthesis reaches a similar conclusion for NLG at large: metrics at best highlight specific dimensions of text quality, their effectiveness varies by task and dataset, and the most recent type of metric, LLM-as-a-Judge, still inherits inconsistent human correlation, unstructured validation, and task dependence (Oliva et al., 19 Aug 2025).

The strongest theoretical challenge to the “one best metric” program comes from two complementary directions. One is the “glass ceiling” argument that automatic metrics are much more similar to one another than to human metrics, are not complementary, and rank systems similarly; the other is the multi-criteria argument that NLG quality dimensions such as fluency, coherence, coverage, relevance, and adequacy are often not strongly correlated with one another, so validation against a single human criterion is inadequate (Colombo et al., 2022, Sai et al., 2021). This suggests that future AEM work will likely need to optimize not only raw human-correlation scores but also complementarity, criterion coverage, robustness under perturbation, and clarity about the level of evaluation—segment, document, dialog, system, or workflow.

Interpreting studies make the limitation especially explicit. A critical analysis of AEMs for interpreting argues that automatic metrics are “backwards-looking”: they ask whether the target sufficiently reflects the source, or how closely it matches an earlier translation, but they do not ask whether the interpreting met the needs of the people involved, fulfilled its communicative purpose, supported decision-making, enabled turn-taking, conveyed affect, or handled culturally loaded and pragmatically complex material appropriately. The proposed shift is “towards situated quality,” where evaluation is localized, tied to the actual situation, and meaningful to stakeholders (Downie et al., 9 Jan 2026). This suggests a broader implication for AEM research: even technically strong metrics may remain partial instruments unless their assumptions align with the communicative setting they are meant to judge.