Minimum-Consistency Accuracy (MCA) Metric

• Minimum-Consistency Accuracy (MCA) is defined as the fraction of questions for which a model’s response consistency exceeds a specified threshold, combining accuracy with stability.
• The metric is computed by creating divergence sets of meaning-equivalent variants and thresholding per-question response consistency to capture robust performance.
• MCA serves as the foundational measure in the CAT framework, underpinning CAR curves and the CORE index to evaluate consistency-accuracy trade-offs in multiple-choice benchmarks.

Minimum-Consistency Accuracy (MCA) is a tunable evaluation metric that measures accuracy under a minimum required level of response consistency across controlled input variations. In the CAT framework, MCA is defined for multiple-choice benchmarks by constructing, for each question, a divergence set of meaning-equivalent variants and then counting the fraction of questions whose response consistency meets or exceeds a threshold $c\in[0,1]$. MCA therefore treats correctness as conditional on stability, rather than as a single-shot event, and it functions as the basic quantity from which the Consistency-Accuracy Relation (CAR) curves and the Consistency-Oriented Robustness Estimate (CORE) index are derived (Cavalin et al., 26 Nov 2025).

1. Definition and conceptual scope

In CAT, MCA is introduced to evaluate the interplay of accuracy and response consistency of LLMs under controllable input variations, using multiple-choice benchmarks as a case study. The motivating claim is that current evaluation practices primarily focus on accuracy or benchmark scores, whereas consistency is increasingly treated as an essential property for deployment in high-stake, real-world applications. CAT argues that both dimensions should still be evaluated independently, but that their inter-dependency also needs to be considered for a more nuanced evaluation (Cavalin et al., 26 Nov 2025).

The metric formalizes a specific question: how often is the model correct among samples whose responses satisfy at least a chosen consistency threshold? Standard accuracy asks whether the model was correct on a single input. Consistency asks how stable and correct the model is across controlled variations. MCA combines these by making accuracy thresholded by consistency. This thresholding is the central feature of the metric.

The CAT paper describes MCA as evaluating the accuracy of an LLM on a multiple-choice benchmark while considering correct only those responses which meet or exceed a specified minimum consistency threshold $c$. This definition places MCA between raw accuracy and consistency-only statistics: it is not merely a mean over perturbed variants, and it is not an all-purpose robustness score detached from correctness. It is instead an accuracy-like measure indexed by a consistency requirement (Cavalin et al., 26 Nov 2025).

2. Formalization and computation

For a benchmark with questions $Q=\{q_1,\ldots,q_N\}$, let $o_i^*$ denote the correct option for question $q_i$, and let $a_i$ denote the model’s standard single-shot prediction. For controlled variations, CAT defines a divergence set

$$Q^*=\{\hat{Q}_1^*,\ldots,\hat{Q}_N^*\}, \qquad \hat{Q}_i^*=\{q_i^1,\ldots,q_i^M\},$$

where each $\hat{Q}_i^*$ contains $M$ meaning-equivalent variants of question $i$, and $c$0 is the model prediction on variant $c$1 (Cavalin et al., 26 Nov 2025).

Per-question response consistency is defined as

$c$2

where $c$3 is the indicator function. Thus, $c$4 is the fraction of variants for which the model answers correctly (Cavalin et al., 26 Nov 2025).

MCA at threshold $c$5 is then defined as

$c$6

Operationally, the computation consists of five steps: create divergence sets; query the model on every variant; compute $c$7 for each question; test whether $c$8; and average the resulting indicators over all $c$9 questions. In the CAT experiments on multiple-choice benchmarks, the controlled variations are produced by reordering answer choices (Cavalin et al., 26 Nov 2025).

The threshold $Q=\{q_1,\ldots,q_N\}$0 determines how strict the metric is. At low $Q=\{q_1,\ldots,q_N\}$1, weak consistency is sufficient. At high $Q=\{q_1,\ldots,q_N\}$2, only strongly stable behavior is counted. At $Q=\{q_1,\ldots,q_N\}$3, only fully consistent responses count. This threshold parameter is not incidental; it is the mechanism that makes MCA a family of metrics rather than a single fixed score (Cavalin et al., 26 Nov 2025).

3. Position within the CAT framework

MCA is the building block of CAT. The CAT framework consists of three components: MCA, CAR curves, and CORE. MCA provides the score at a specific consistency threshold; the CAR curve is obtained by evaluating MCA over a range of thresholds; and CORE is a scalar summary of the resulting curve (Cavalin et al., 26 Nov 2025).

If $Q=\{q_1,\ldots,q_N\}$4 is an ordered set of thresholds, the CAR curve is defined as

$Q=\{q_1,\ldots,q_N\}$5

The curve visualizes how accuracy changes as the consistency requirement becomes stricter. Low-threshold regions correspond to permissive consistency requirements; high-threshold regions correspond to strict requirements; and $Q=\{q_1,\ldots,q_N\}$6 isolates fully consistent behavior (Cavalin et al., 26 Nov 2025).

CORE is built from the CAR curve rather than directly from single-shot accuracy. CAT defines an area term, AUCAR, using a numerical approximation

$Q=\{q_1,\ldots,q_N\}$7

and then combines this with a normalized dynamic time warping similarity to an ideal perfectly consistent model: $Q=\{q_1,\ldots,q_N\}$8 The paper’s rationale is that area alone can be overly optimistic and shape alone can overestimate similarity, whereas their product yields a better-considered summary of the CAR curve (Cavalin et al., 26 Nov 2025).

This organization gives MCA a dual role. It is a standalone metric at a chosen threshold, and it is also the ordinate of the CAR curve. A plausible implication is that CAT treats MCA not merely as a robustness adjunct, but as the primitive quantity through which the consistency-accuracy trade-off becomes analyzable across operating points.

4. Relation to adjacent metrics and thresholded consistency evaluation

CAT positions MCA against several adjacent metrics used on multiple-choice benchmarks. Standard MCQA accuracy is

$Q=\{q_1,\ldots,q_N\}$9

which uses only a single prompt per question. MCQA+ averages correctness over controlled variants, and MV applies a majority-vote rule over variant predictions. CAT argues that these statistics do not expose how much accuracy survives as consistency requirements become stricter (Cavalin et al., 26 Nov 2025).

Quantity	Basis	Role in the literature
MCQA	Single-shot correctness	Standard benchmark accuracy
MCQA+	Mean correctness over variants	Global mean over altered inputs
MV	Most frequent answer across variants	Relatively permissive decision rule
MCA	Threshold on per-item consistency	Thresholded benchmark metric
CORE	Area and shape of CAR	Global summary of the trade-off

A close neighboring formulation appears in CoRA, which introduces Bare-Minimum-Consistency Accuracy (BMCA): $o_i^*$0 That paper states that BMCA generalizes majority voting by replacing the fixed $o_i^*$1 threshold with an adjustable minimum consistency level $o_i^*$2, and that this mechanism is exactly what would correspond to a “minimum-consistency accuracy” idea. The paper itself does not use MCA as the formal metric name; its main proposed metric is Consistency-Rebalanced Accuracy (CoRA) (Cavalin et al., 26 Nov 2025).

Another related line of work studies contrast set consistency and relative consistency. In that setting, consistency is bundle-level and all-or-nothing, and the key theoretical point is that achievable consistency depends on accuracy. Relative consistency is then defined as a cumulative probability over the achievable-consistency distribution at fixed accuracy. That framework does not define MCA, but it argues that raw consistency is not directly comparable across models with different accuracies. This suggests a broader interpretive context for MCA-style metrics: thresholded consistency scores are most informative when the interaction between accuracy and consistency is made explicit, rather than assumed away (Johnson et al., 2023).

5. Threshold behavior, interpretation, and practical meaning

The meaning of MCA depends strongly on the threshold $o_i^*$3. CAT explicitly discusses $o_i^*$4 as the strictest version, since only fully consistent responses count. Lower thresholds such as $o_i^*$5 are more permissive and are described as closer in spirit to majority-vote-style behavior. The paper notes that high $o_i^*$6 can overly penalize the model, while low $o_i^*$7 can resemble majority voting and may overestimate accuracy at moderate or high consistency levels. It also notes that $o_i^*$8 may understate useful performance, which is one reason CORE is proposed as a balance between extremes (Cavalin et al., 26 Nov 2025).

This threshold sensitivity is central to the interpretation of the metric. A single MCA value is an operating-point statistic: it answers how much benchmark performance remains once a specific minimum consistency requirement has been imposed. A full sweep of thresholds produces the CAR curve, which reveals whether a model’s apparent competence is stable or collapses under stricter consistency demands (Cavalin et al., 26 Nov 2025).

The metric is also presented as policy-relevant. CAT states that MCA is useful in settings where regulations or application needs impose minimum consistency requirements. In that sense, MCA can be read as a compliance-style score: it does not ask merely whether the model can answer correctly, but whether it can do so while satisfying a specified stability condition under controlled variation (Cavalin et al., 26 Nov 2025).

A common misconception is to treat MCA as synonymous with standard multiple-choice accuracy on perturbed inputs. CAT’s formulation is stricter. It first computes per-question response consistency and only then converts those values into a benchmark-level score via thresholding. This is why MCA differs both from raw MCQA and from global means such as MCQA+.

6. Empirical use and terminological ambiguity

CAT reports experiments with eight LLMs on four multiple-choice benchmarks: MedQA, MMLU-Redux, ARC, and TruthfulQA. It reports $o_i^*$9 and states that this strict score often grows more gradually than MCQA+ or MV, making it less prone to early saturation. The paper further states that $q_i$0 can differentiate models that look similar under standard accuracy, that CAR curves reveal cases in which some models perform close to chance at low consistency thresholds and only improve under stricter conditions, and that models with similar average scores can exhibit different consistency-accuracy trade-offs (Cavalin et al., 26 Nov 2025).

The reported $q_i$1 ranges illustrate the contraction from single-shot accuracy to fully consistent performance. On MedQA, values range from near $q_i$2 up to about $q_i$3; on MMLU-Redux, around $q_i$4–$q_i$5; on ARC, around $q_i$6–$q_i$7; and on TruthfulQA, around $q_i$8–$q_i$9. The paper also notes that some models, such as Llama-1, are close to chance baseline, that in TruthfulQA some models even underperform chance at certain low-consistency regions, that finetuning often makes CAR curves of previously different base models more similar, and that expert medical models can outperform base models on general benchmarks under varying consistency requirements (Cavalin et al., 26 Nov 2025).

The acronym “MCA” is, however, terminologically overloaded in the arXiv literature. In the 2D-3D retrieval paper “MCA: 2D-3D Retrieval with Noisy Labels via Multi-level Adaptive Correction and Alignment,” MCA denotes “Multi-level cross-modal adaptive Correction and Alignment,” the name of a retrieval framework rather than an evaluation metric (Zou et al., 8 Aug 2025). In “MCA: Boolean Networks Control Algorithm,” MCA denotes a control algorithm for Boolean networks (Moradi et al., 2016). In “MCA-based Rule Mining Enables Interpretable Inference in Clinical Psychiatry,” MCA refers to Multiple Correspondence Analysis rather than Minimum-Consistency Accuracy (Gao et al., 2018).

For the topic of Minimum-Consistency Accuracy specifically, the relevant formalization is the CAT metric. Neighboring work uses closely related constructions—most explicitly BMCA in CoRA—but does not standardize the same term. The resulting picture is that MCA, in the sense of Minimum-Consistency Accuracy, names a thresholded consistency-conditioned accuracy metric for controlled-input evaluation of multiple-choice reasoning, and not a universally adopted acronym across machine learning more broadly (Cavalin et al., 26 Nov 2025).