---
title: 'CAT Module: Evaluating LLM Consistency'
url: https://www.emergentmind.com/topics/cat-module
type: topic
---

# CAT Module: Evaluating LLM Consistency

A CAT module, in the context of Large Language Model (LLM) evaluation, denotes the unified metric-driven framework introduced in “CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variations” [2512.23711]. CAT provides a principled, rigorous methodology for quantifying how LLM accuracy trades off against consistency when exposed to controlled, meaning-preserving input perturbations. It does this using new metrics—Minimum-Consistency Accuracy (MCA), Consistency-Accuracy Relation (CAR) curves, and the Consistency-Oriented Robustness Estimate (CORE)—and supports systematic benchmarking for both multiple-choice and open-ended evaluation settings.

## 1. Motivation and Problem Formulation

Evaluation of LLMs in high-stakes or real-world deployment scenarios increasingly demands not only raw accuracy but also robust response consistency under input variations that should not affect “ground truth.” Traditional accuracy metrics, such as MCQA+ or majority voting, either ignore answer consistency or implicitly conflate it with accuracy. CAT addresses a critical gap by defining and visualizing the interaction between accuracy and consistency in a tunable and transparent manner.

Formally, given a set \( Q = \{q_1, \dots, q_N\} \) of multiple-choice questions and per-question, per-variant model answers \( \{ a_i^j \} \), CAT systematically quantifies how often a model’s answer agrees with the gold label \( o^*_i \) across controlled “divergence sets” (variations in formatting, order, etc.). The aim is to move beyond single-point metrics and illuminate how imposing stricter consistency requirements modulates apparent model performance.

## 2. Core Metrics: Minimum-Consistency Accuracy (MCA) and CAR Curves

The foundational metric in CAT is Minimum-Consistency Accuracy (MCA), defined for a tunable threshold \( c \in [0,1] \). Response consistency per item is measured as

\[
\text{RC}_i = \frac{1}{M}\sum_{j=1}^M \Lambda(a_i^j = o^*_i),
\]

where \( M \) is the number of variants, and \( \Lambda(\cdot) \) is the 0–1 indicator.

For a chosen \( c \), MCA counts items where consistency meets or exceeds \( c \):

\[
\boxed{
    \text{MCA}(c) = \frac{1}{N} \sum_{i=1}^N \Lambda( \text{RC}_i \geq c ).
}
\]

Two extremal regimes:

- \( \text{MCA}(0) \): classic “averaged accuracy” over all variants (i.e., MCQA+).
- \( \text{MCA}(1.0) \): fraction of items answered correctly on every variant (i.e., perfect consistency).

CAT visualizes the entire consistency–accuracy tradeoff by computing MCA across a grid \( C = \{c_1,\dots,c_K\} \) and plotting the resulting Consistency-Accuracy Relation (CAR) curve:

\[
\text{CAR} = \left\{ (c_k, \;\text{MCA}(c_k)) \bigm| k = 1, \dots, K \right\}.
\]

This provides a continuous curve from fully permissive (\( c=0 \)) to maximally strict (\( c=1 \)) evaluation paradigms.

## 3. Consistency-Oriented Robustness Estimate (CORE) Index

To distill the CAR curve into a single, interpretable robustness scalar, CAT introduces the CORE metric:

\[
\text{CORE} = \text{AUCAR} \times \text{norm-DTW}, \quad 0 \leq \text{CORE} \leq 1,
\]

where

- \( \text{AUCAR} \) is the area under the CAR curve,

\[
\text{AUCAR} \approx \sum_{k=1}^{K-1} \frac{\text{MCA}(c_k) + \text{MCA}(c_{k+1}) }{2} (c_{k+1} - c_k),
\]

- \( \text{norm-DTW} \) is a normalized Dynamic Time Warping similarity between the observed CAR and the ideal horizontal line at \( y=1 \):

\[
\text{norm-DTW} = 1 - \frac{\mathrm{DTW}_\text{model}}{\mathrm{DTW}_\text{worst}} \in [0,1].

Source: https://www.emergentmind.com/topics/cat-module