CAT Module: Evaluating LLM Consistency

• CAT Module is a metric-driven framework that quantifies how LLM accuracy trades off against consistency using controlled input perturbations.
• It introduces key metrics such as Minimum-Consistency Accuracy, CAR curves, and the CORE index to measure and visualize performance under varying conditions.
• The framework supports both multiple-choice and open-ended tasks, offering actionable insights for robust evaluation in high-stakes applications.

A CAT module, in the context of LLM evaluation, denotes the unified metric-driven framework introduced in “CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variations” (Cavalin et al., 26 Nov 2025). CAT provides a principled, rigorous methodology for quantifying how LLM accuracy trades off against consistency when exposed to controlled, meaning-preserving input perturbations. It does this using new metrics—Minimum-Consistency Accuracy (MCA), Consistency-Accuracy Relation (CAR) curves, and the Consistency-Oriented Robustness Estimate (CORE)—and supports systematic benchmarking for both multiple-choice and open-ended evaluation settings.

1. Motivation and Problem Formulation

Evaluation of LLMs in high-stakes or real-world deployment scenarios increasingly demands not only raw accuracy but also robust response consistency under input variations that should not affect “ground truth.” Traditional accuracy metrics, such as MCQA+ or majority voting, either ignore answer consistency or implicitly conflate it with accuracy. CAT addresses a critical gap by defining and visualizing the interaction between accuracy and consistency in a tunable and transparent manner.

Formally, given a set $Q = \{q_1, \dots, q_N\}$ of multiple-choice questions and per-question, per-variant model answers $\{ a_i^j \}$, CAT systematically quantifies how often a model’s answer agrees with the gold label $o^*_i$ across controlled “divergence sets” (variations in formatting, order, etc.). The aim is to move beyond single-point metrics and illuminate how imposing stricter consistency requirements modulates apparent model performance.

2. Core Metrics: Minimum-Consistency Accuracy (MCA) and CAR Curves

The foundational metric in CAT is Minimum-Consistency Accuracy (MCA), defined for a tunable threshold $c \in [0,1]$. Response consistency per item is measured as

$$\text{RC}_i = \frac{1}{M}\sum_{j=1}^M \Lambda(a_i^j = o^*_i),$$

where $M$ is the number of variants, and $\Lambda(\cdot)$ is the 0–1 indicator.

For a chosen $c$, MCA counts items where consistency meets or exceeds $c$:

$$\boxed{ \text{MCA}(c) = \frac{1}{N} \sum_{i=1}^N \Lambda( \text{RC}_i \geq c ). }$$

Two extremal regimes:

• $\{ a_i^j \}$0: classic “averaged accuracy” over all variants (i.e., MCQA+).
• $\{ a_i^j \}$1: fraction of items answered correctly on every variant (i.e., perfect consistency).

CAT visualizes the entire consistency–accuracy tradeoff by computing MCA across a grid $\{ a_i^j \}$2 and plotting the resulting Consistency-Accuracy Relation (CAR) curve:

$\{ a_i^j \}$3

This provides a continuous curve from fully permissive ($\{ a_i^j \}$4) to maximally strict ($\{ a_i^j \}$5) evaluation paradigms.

3. Consistency-Oriented Robustness Estimate (CORE) Index

To distill the CAR curve into a single, interpretable robustness scalar, CAT introduces the CORE metric:

$\{ a_i^j \}$6

where

• $\{ a_i^j \}$7 is the area under the CAR curve,

$\{ a_i^j \}$8

• $\{ a_i^j \}$9 is a normalized Dynamic Time Warping similarity between the observed CAR and the ideal horizontal line at $o^*_i$0:

[ \text{norm-DTW} = 1 - \frac{\mathrm{DTW}\text{model}}{\mathrm{DTW}\text{worst}} \in [0,1].