Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory (2305.14889v2)

Abstract: We address a fundamental challenge in Natural Language Generation (NLG) model evaluation -- the design and evaluation of evaluation metrics. Recognizing the limitations of existing automatic metrics and noises from how current human evaluation was conducted, we propose MetricEval, a framework informed by measurement theory, the foundation of educational test design, for conceptualizing and evaluating the reliability and validity of NLG evaluation metrics. The framework formalizes the source of measurement error and offers statistical tools for evaluating evaluation metrics based on empirical data. With our framework, one can quantify the uncertainty of the metrics to better interpret the result. To exemplify the use of our framework in practice, we analyzed a set of evaluation metrics for summarization and identified issues related to conflated validity structure in human-eval and reliability in LLM-based metrics. Through MetricEval, we aim to promote the design, evaluation, and interpretation of valid and reliable metrics to advance robust and effective NLG models.

Overview

The work introduces MetricEval, a framework that leverages principles from measurement theory—traditionally applied in educational test design—to rigorously assess the reliability and validity of NLG evaluation metrics. The contribution lies in formalizing the distinction between observed scores and latent quality, quantifying measurement errors, and proposing statistical methodologies that enable a more granular diagnosis of metric performance in NLG settings.

Measurement Theory Foundations

MetricEval is grounded in classical measurement theory, where the measurement process is characterized by the decomposition of observed scores into true scores and error components. This framework acknowledges that any evaluation metric, whether automated or human-judged, is susceptible to noise and biases. The conceptual underpinning adopts key measurement properties such as:

• Test–Retest Reliability: Operationalized via the Pearson correlation to capture metric stability over repeated observations on identical outputs.
• Internal Consistency: Quantified using coefficient alpha, which measures the agreement among different segments or items of the benchmark dataset.
• Construct Validity: Investigated through convergent and divergent validity in a multitrait-multimethod (MTMM) matrix, along with factorial validity through factor analysis.
• Concurrent Validity: Evaluated by comparing the target metric against a validated reference criterion with the use of a Pearson correlation coefficient.

By treating evaluation scores as stochastic variables with underlying measurement error, MetricEval systematically isolates and quantifies the sources of uncertainty in NLG evaluations.

Framework Components

MetricEval is built around four key desiderata that address both the reliability and validity of NLG evaluation metrics:

1. Metric Stability: Quantified using test–retest reliability coefficients, this facet measures the consistency of metric scores when the same model output is rescored. A high stability coefficient (close to +1) indicates minimal random fluctuations due to metric sensitivity or inherent model output variability.
2. Metric Consistency: Evaluated via coefficient alpha, this property examines the internal congruence among evaluations across different segments of a dataset. High internal consistency implies that the metric yields stable and coherent assessments across various subsets, which is critical in benchmarking exercises.
3. Metric Construct Validity:

This multidimensional component is further divided into: - Convergent Validity: Verification that metrics intended to measure similar constructs (e.g., coherence or relevance) are highly correlated. - Divergent Validity: Ensuring that unrelated dimensions are statistically uncorrelated, thereby confirming that the metric is not conflated across constructs. - Factorial Validity: The alignment of observed metric scores with theorized latent factors is assessed via factor analysis, which clarifies the underlying dimensionality of the construct space.

1. Metric Concurrent Validity: This aspect involves benchmarking a given metric against a pre-validated reference metric. Here, the Pearson correlation coefficient is used to assess how closely the new metric aligns with established measures, thus situating its performance within the broader evaluation landscape.

Statistical Tools and Implementation

MetricEval employs a suite of statistical tools, ensuring that each measurement property is quantitatively assessed:

• Test–Retest Reliability: Pearson correlation coefficient is used over repeated metric scores.
• Coefficient Alpha: A measure of internal consistency, often computed using Cronbach’s alpha, to capture agreement among different parts of a benchmark dataset.
• Multitrait-Multimethod (MTMM) Matrix Analysis: This is implemented to differentiate convergent from divergent validity, offering a structured approach to identifying overlaps between metrics of related constructs.
• Factor Analysis: Employed to validate the factorial structure and confirm that observed metric scores truly reflect the theoretical aspects of model quality.
• Pearson Correlation for Concurrent Validity: Used to measure the validity coefficient between the target metric and the reference criterion.

Implementation of these statistical tests typically leverages standard packages in Python (e.g., SciPy, statsmodels) or R, ensuring that empirical researchers can systematically integrate these analyses into existing evaluation pipelines. Such integration allows for quantification of uncertainty and enhanced interpretability of NLG evaluation results.

Empirical Case Study: Summarization Evaluation

The application of MetricEval in the context of summarization benchmarks reveals critical deficiencies in both human and LLM-based metrics:

• Human Evaluation (SummEval Data):

The analysis uncovered a conflated validity structure, particularly between coherent and relevant constructs. Expert ratings demonstrated overlapping scores that may originate from ambiguous guidelines or intrinsic subjectivity. Such conflation significantly impacts the interpretation of human judgment, undermining both convergent and discriminant validity.

• LLM-Based Metrics:

Automated metrics based on LLM scores showed markedly lower stability, indicating sensitivity to minor fluctuations in input data and potential measurement errors. Metrics such as BARTScore and G-Eval, while capturing quality signals, struggled to discriminate between distinct evaluative dimensions robustly. This instability is quantified by lower test–retest coefficients, signifying the need for refined metric formulations.

Discussion and Implications

MetricEval’s integration of measurement theory allows for an unprecedented level of rigor in NLG metric evaluation. By decomposing scores into true quality and error components, it facilitates a nuanced interpretation of metric outputs and their variability. The implications for real-world deployment include:

• Enhanced Interpretability: Stakeholders are better equipped to understand the reliability and construct validity of evaluation scores, leading to more informed model comparisons.
• Metric Refinement: Empirical insights from MTMM and factor analysis can direct future metric development to address observed deficiencies, ensuring that evaluation instruments measure intended constructs without conflation.
• Quantification of Uncertainty: The framework provides a method to quantify the uncertainty associated with evaluation metrics, which is crucial for robust performance reporting and decision-making in production systems.

Users implementing MetricEval should account for computational overhead associated with repeated evaluations and the necessity for sufficiently large datasets to reliably estimate coefficients. While the framework employs standard statistical toolkits, careful attention must be paid to the assumptions inherent in each statistical method (e.g., normality assumptions for Pearson correlations and validity of factor model interpretations).

Conclusion

MetricEval presents a systematic and quantitatively rigorous framework for the evaluation of NLG evaluation metrics. By embedding measurement theory into the analysis process, it provides a structured approach to quantifying metric stability, consistency, and validity. The framework's application in summarization evaluation exposes significant challenges in current human and LLM-based metrics, most notably in terms of validity conflations and sensitivity to measurement error. Overall, MetricEval offers researchers and practitioners a robust set of tools to improve metric design, interpretation, and ultimately, the development of next-generation NLG systems.