Calibrating Model-Based Evaluation Metrics for Summarization

Abstract: Recent advances in summary evaluation are based on model-based metrics to assess quality dimensions, such as completeness, conciseness, and faithfulness. However, these methods often require LLMs, and predicted scores are frequently miscalibrated, limiting their reliability. Moreover, evaluating the average quality across different summaries for a single document typically requires access to multiple reference summaries. Here, we propose a general framework that generates individual and average proxy scores without relying on reference summaries, human annotations, or expensive model-based metrics. We also propose group isotonic regression binning (GIRB), a calibration method that adjusts the raw predictions to better align with ground-truth evaluation metrics. While we focus on continuous-value scenarios, such as summarization, the method is applicable to discrete-value tasks, such as question answering. Experiments on seven datasets demonstrate that our approach consistently outperforms existing baselines.

• The paper presents GIRB, a novel post hoc calibration method using group isotonic regression binning to generate accurate proxy scores for summaries.
• It demonstrates significant improvements in calibration error, Brier score, and group calibration across multiple datasets compared to baseline methods.
• The framework’s model-agnostic design enables scalable, reference-free evaluation of summarization and QA outputs, closely aligning with human judgments.

Calibrating Model-Based Evaluation Metrics for Summarization

Motivation and Problem Background

Model-based metrics are now central to the evaluation of LLM-generated summaries, supporting multidimensional assessments such as completeness, conciseness, and faithfulness. Existing metric generators—often LLMs themselves—frequently output miscalibrated scores. Such miscalibration leads to untrustworthy downstream decisions, undermining reliability in automated quality assessment. Furthermore, obtaining reference-based or human-annotated ground-truth for calibration is computationally infeasible at scale. An efficient, reference-free, and accurate calibration architecture is therefore critical.

Core Methodological Contributions

The paper introduces a general-purpose calibration framework that generates per-summary and aggregate proxy scores across multiple dimensions without any need for reference summaries or human input. Central to this is the novel post hoc calibration method, group isotonic regression binning (GIRB), which operates as follows.

Each document-summary pair is embedded via a frozen or lightly-tuned encoder. These embeddings are then clustered to assign optimization groups, intended to capture content-specific biases in model-based predictions—e.g., summarization output in the medical domain may need a distinct calibration profile compared to that in legal or news contexts. Within each group, GIRB fits a monotonic isotonic regression mapping from raw model predictions to proxy scores. This flexible, continuous mapping strictly enforces monotonicity and circumvents the information loss associated with classic histogram-based binning strategies.

The framework architecture is depicted in the following overview:

Figure 1: A unified scoring model maps each document-summary pair to raw scores $\hat{y}$ via shared embeddings that are reused for clustering; GIRB then clusters representations and performs post hoc groupwise isotonic regression calibration.

Critically, the model delivers both an individual proxy score and an averaged proxy score—enabling the estimation, for any single summary, of both its own merit and its expected position relative to system-level baselines.

Experimental Evaluation

Experiments span document summarization (FeedSum with FineSurE and UniSumEval metrics) and question answering (six established QA datasets). GIRB is benchmarked against a strong baseline (QA binning, QAB [manggala2024qa]) and several other calibration strategies (histogram binning, Platt scaling, hierarchical scaling, etc.), each using either raw or group-conditioned information.

On summarization tasks, GIRB consistently improves calibration error, group calibration, Brier error, and discrimination over all baselines. For instance, GIRB achieves higher win counts and mean gain improvements over QAB on all primary metrics. The empirical coverage extends across conciseness, completeness, and multidimensional aspects such as fidelity, consistency, fluency, and coherence.

Figure 2: Results comparing calibration methods across dimensions and metrics; GIRB outperforms others most frequently, particularly on conciseness and completeness.

Furthermore, GIRB maintains strong performance when tested with lightweight embedding models (e.g., DistilBERT), demonstrating that large LLMs are not strictly necessary for effective calibration. Sensitivity analyses confirm that KMeans clustering and contextualized prompts for embedding extraction yield the highest calibration improvements, with only marginal degradation for less sophisticated variants.

In the QA domain, evaluated using MMLU and others, GIRB again reliably ranks at or near the top on group conditional calibration error, expected calibration error, and Brier score (see Figure 3). On several datasets, GIRB’s groupwise isotonic regression results in both the highest number of per-metric wins and the largest mean performance gains relative to the uncalibrated baseline and binning-based methods.

Figure 3: Results comparing calibration methods over all metrics in the MMLU dataset, with GIRB generally achieving the strongest calibration.

Human alignment is also assessed: ground-truth calibration against human-annotated summary scores confirms that GIRB achieves the closest match with human judgment, outperforming QAB and raw metric outputs on ECE, group-ECE, and Brier score.

Theoretical Implications and Model Generality

The approach formalizes continuous calibration for summarization, extending statistical calibration concepts to proxy metrics with continuous-valued targets—contrasted with previous work that focused on discrete/binary calibration. Groupwise calibration isolates local domains of semantic homogeneity, enabling unbiased adjustment of content-specific systematic errors. Isotonic regression, in turn, promotes finer-grained and smoother transformations than piecewise-constant histogram binning, delivering calibration mappings that are robust to changes in the underlying distribution.

Unlike global calibration routines that risk overcorrecting or undercorrecting in heterogeneous content regimes, GIRB accommodates heterogeneity via modular, group-conditioned monotonic transformations. This design is model-agnostic and functions as a pure post hoc recalibration, preserving architectural flexibility.

Practical Implications and Limitations

Practically, GIRB enables scalable, data-efficient, and reliable evaluation of LLM-generated summaries and QA answers, without the overhead of reference-based methods or additional model-based metric queries. By providing both absolute and relative scores (against system averages), it offers versatile utility for automated system evaluation, online deployment pipelines, and adaptive summary improvement strategies.

However, GIRB's performance depends critically on both the quality of content-grouping (clustering) and the representational power of embeddings. In settings with limited post hoc calibration data or poorly separated semantic spaces, calibration improvements may diminish. Exploring adaptive or neural clustering, or end-to-end trainable grouping strategies, may offer further gains. Broader extension to multimodal or hierarchical evaluation schemes is plausible.

Conclusion

This work establishes a robust, efficient, and theoretically grounded methodology for calibrating continuous model-based evaluation metrics in LLM summarization and QA. Groupwise isotonic regression binning effectively resolves semantic heterogeneity in miscalibrated predictions. The method yields strong quantitative calibration improvements, validated across multiple datasets, evaluation dimensions, and baseline systems. GIRB’s groupwise, monotonic, model-agnostic approach sets a new calibration standard for reference-free evaluation regimes and offers a foundation for further developments in scalable, trustworthy evaluation of generative LLM outputs.

Figure 4: Workflow Overview showing dual score types (individual and average) produced for each summary, enabling efficient reference-free assessment of summary quality.