Consistency Index (CI): Evaluation & Applications

• Consistency Index (CI) is a reliability measure that quantifies structural coherence in tasks, from LLM multiple-choice benchmarks to pairwise comparison matrices.
• In LLM evaluations, CI is derived from the difference between raw accuracy and the bare-minimum consistency accuracy, recalibrating performance through the CoRA metric.
• In pairwise comparisons, CI assesses global inconsistency using either eigenvalue deviations (Saaty’s method) or average triad determinants (Peláez–Lamata method) for decision support.

to=arxiv_search 福利彩票天天彩json {"query":"(Cavalin et al., 26 Nov 2025) Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choices (1311.0748, Brunelli, 2015)", "max_results": 5} to=search_arxiv 下载彩神争霸 _日本毛片免费视频观看 code ਨਹੀਂjson {"query":"(Cavalin et al., 26 Nov 2025) Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choices", "max_results": 5} to=arxiv qq天天中彩票 code 微信上的天天中彩票json {"query":"(Cavalin et al., 26 Nov 2025)", "max_results": 3} Consistency Index (CI) denotes a class of measures that quantify whether answers, judgments, or comparisons remain coherent under the structural constraints of a task. In recent multiple-choice evaluation of LLMs, CI is defined as a reliability factor that measures how much raw multiple-choice accuracy survives systematic perturbations of the answer choices, and it is used to rebalance accuracy into Consistency-Rebalanced Accuracy (CoRA) (Cavalin et al., 26 Nov 2025). In pairwise comparison theory, the same label is used for at least two established inconsistency measures: Saaty’s spectral Consistency Index and the Peláez–Lamata triad-based Consistency Index for positive reciprocal matrices (1311.0748, Brunelli, 2015). The shared intuition is stability under a formal notion of consistency, but the mathematical object, admissible range, and practical interpretation are domain-specific.

1. Terminological scope and principal usages

The term “Consistency Index” is not univocal. In LLM benchmark evaluation, CI is defined on multiple-choice question answering (MCQA) performance under altered answer choices and takes values in $[0,1]$, with higher values indicating that correct answers remain stable across perturbations (Cavalin et al., 26 Nov 2025). In pairwise comparison matrices (PCMs), CI may refer either to Saaty’s eigenvalue-based inconsistency measure or to the Peláez–Lamata average of triad determinants, both of which equal zero under perfect multiplicative consistency but differ in aggregation logic and thresholding practice (1311.0748, Brunelli, 2015).

Usage	Mathematical object	Core interpretation
LLM MCQA CI	Gap-adjusted factor from $\mathrm{MCQA}$ and $\mathrm{BMCA}(1.0)$	Reliability of raw accuracy under altered answer choices
Saaty’s CI	$\frac{\lambda_{\max}(A)-n}{n-1}$ for a PCM $A$	Global spectral inconsistency
Peláez–Lamata CI	Average determinant of reciprocal $3\times 3$ triads	Average local inconsistency across triads

This terminological overlap is a recurrent source of confusion. A statement such as “the CI is low” is not interpretable without specifying whether the underlying object is an LLM benchmark score or a reciprocal comparison matrix. A further ambiguity arises inside the pairwise comparison literature itself, where “CI” may refer either to Saaty’s index or to Peláez–Lamata’s index; by contrast, the LLM metric is explicitly tied to BMCA and CoRA and is not derived from eigenvalues or triads.

2. CI in multiple-choice benchmark evaluation for LLMs

In the LLM setting, the benchmark is $\mathrm{MCQ}=\{mcq_1,\dots,mcq_N\}$, where each $mcq_i$ is a single multiple-choice question with one correct choice and several distractors. The correctness function $\mathrm{LLM}(mcq_i)$ returns $1$ if the model’s selected choice equals the correct alternative in $\mathrm{MCQA}$0, or $\mathrm{MCQA}$1 otherwise. Traditional multiple-choice accuracy is then

$\mathrm{MCQA}$2

To probe answer stability, each original item is associated with a divergence set $\mathrm{MCQA}$3 obtained by altering only the answer choices. Expanded accuracy averages correctness over all divergence sets:

$\mathrm{MCQA}$4

Per-question response consistency is

$\mathrm{MCQA}$5

and Majority Voting (MV) counts a question as correct if $\mathrm{MCQA}$6 (Cavalin et al., 26 Nov 2025).

The intermediate quantity that leads directly to CI is Bare-Minimum-Consistency Accuracy:

$\mathrm{MCQA}$7

where $\mathrm{MCQA}$8 is a minimum response consistency threshold. The paper emphasizes that $\mathrm{MCQA}$9 approximates MV-like permissiveness, whereas $\mathrm{BMCA}(1.0)$0 requires the model to be correct on every variant of the question. CI is then defined by

$\mathrm{BMCA}(1.0)$1

Because $\mathrm{BMCA}(1.0)$2 by construction, $\mathrm{BMCA}(1.0)$3. A value of $\mathrm{BMCA}(1.0)$4 means that every raw correct answer remains correct under every perturbation; a value near $\mathrm{BMCA}(1.0)$5 indicates that many raw correct answers fail under perturbations, so raw MCQA substantially overstates reliability.

The role of CI is operational rather than merely descriptive. It enters the consistency-aware score

$\mathrm{BMCA}(1.0)$6

which multiplicatively scales down raw MCQA when response consistency is low. This makes CI a quality factor on accuracy rather than a standalone replacement for accuracy.

3. Construction from altered answer choices

The LLM CI is built from synthetically generated variants that change only the answer choices while keeping the question text and correct answer intact. For a question with $\mathrm{BMCA}(1.0)$7 alternatives, the perturbation family comprises: Shuffled; With NOTA; With NOTA shuffled; Decoupled; Decoupled shuffled; Decoupled with NOTA; and Decoupled with NOTA shuffled (Cavalin et al., 26 Nov 2025). The paper states: “Our method generates a total of $\mathrm{BMCA}(1.0)$8 variations of the set of choices for each question. For instance, with with five alternatives, i.e. $\mathrm{BMCA}(1.0)$9, 26 variations are generated.”

The evaluation prompt is fixed across all variants and asks the model to output a single letter on the first line, followed by an explanation. The reported template is:

$\mathrm{MCQA}$65

Evaluation parses the first token, strips punctuation and spaces, and compares the resulting letter with the known correct choice. This protocol intentionally isolates instability induced by distractor replacement, shuffling, and subset decoupling, rather than by rephrasing the question stem.

The measurement pipeline proceeds per question and then per dataset. For each $\frac{\lambda_{\max}(A)-n}{n-1}$0, one constructs $\frac{\lambda_{\max}(A)-n}{n-1}$1, queries the LLM on each variant with the same base prompt, parses the first-letter answer, and computes $\frac{\lambda_{\max}(A)-n}{n-1}$2. Dataset-level reporting then includes $\frac{\lambda_{\max}(A)-n}{n-1}$3 on the original benchmark, $\frac{\lambda_{\max}(A)-n}{n-1}$4 for $\frac{\lambda_{\max}(A)-n}{n-1}$5, CI via $\frac{\lambda_{\max}(A)-n}{n-1}$6, and CoRA via $\frac{\lambda_{\max}(A)-n}{n-1}$7.

Several limitations are explicit. The scope is restricted to multiple-choice questions with exactly one correct answer. Some perturbations change the number of presented alternatives, which can alter guessing priors; the paper therefore also evaluates “same-number-of-alternatives only” variants as an ablation. Computational cost is linear in $\frac{\lambda_{\max}(A)-n}{n-1}$8 because each variant requires a separate LLM call, and the paper notes that CoRA’s computational burden is roughly equivalent to MCQA+ and MV.

4. Empirical behavior of CI, BMCA, and CoRA in LLM evaluation

The reported experiments cover MedQA, MMLU, ARC-C, and TruthfulQA with several 7–8B models and GPT4o (Cavalin et al., 26 Nov 2025). The central empirical finding is that high raw MCQA can coexist with low consistency under altered answer choices, and CI exposes this gap.

On MedQA, where $\frac{\lambda_{\max}(A)-n}{n-1}$9 and hence $A$0 variants in a 0-shot setting, the paper reports $A$1 values of GPT4o $A$2, MedL $A$3, BioML $A$4, and BMist $A$5. The corresponding CoRA values are GPT4o $A$6, MedL $A$7, BioML $A$8, and BMist $A$9. The associated $3\times 3$0 values make the mechanism explicit: GPT4o drops from $3\times 3$1 to $3\times 3$2, whereas MedL drops from $3\times 3$3 to $3\times 3$4, BioML from $3\times 3$5 to $3\times 3$6, and BMist from $3\times 3$7 to $3\times 3$8. The paper’s worked examples show the arithmetic directly: for GPT4o, $3\times 3$9 and $\mathrm{MCQ}=\{mcq_1,\dots,mcq_N\}$0; for MedL, $\mathrm{MCQ}=\{mcq_1,\dots,mcq_N\}$1 and $\mathrm{MCQ}=\{mcq_1,\dots,mcq_N\}$2.

The same pattern appears on general benchmarks. On MMLU, with $\mathrm{MCQ}=\{mcq_1,\dots,mcq_N\}$3–$\mathrm{MCQ}=\{mcq_1,\dots,mcq_N\}$4 choices and hence $\mathrm{MCQ}=\{mcq_1,\dots,mcq_N\}$5–$\mathrm{MCQ}=\{mcq_1,\dots,mcq_N\}$6 in a 5-shot setting, reported CI values are Mist $\mathrm{MCQ}=\{mcq_1,\dots,mcq_N\}$7, Llam $\mathrm{MCQ}=\{mcq_1,\dots,mcq_N\}$8, Gran $\mathrm{MCQ}=\{mcq_1,\dots,mcq_N\}$9, and DSeek $mcq_i$0; the corresponding CoRA values are $mcq_i$1, $mcq_i$2, $mcq_i$3, and $mcq_i$4. On ARC-C, with $mcq_i$5–$mcq_i$6 choices and $mcq_i$7–$mcq_i$8 in a 25-shot setting, CI values are Gran $mcq_i$9, Mist $\mathrm{LLM}(mcq_i)$0, DSeek $\mathrm{LLM}(mcq_i)$1, and Llam $\mathrm{LLM}(mcq_i)$2, while CoRA values are $\mathrm{LLM}(mcq_i)$3, $\mathrm{LLM}(mcq_i)$4, $\mathrm{LLM}(mcq_i)$5, and $\mathrm{LLM}(mcq_i)$6. The paper highlights Llam’s CI of $\mathrm{LLM}(mcq_i)$7 as a case of strong inconsistency that widens the gap in CoRA relative to MCQA. On TruthfulQA, with $\mathrm{LLM}(mcq_i)$8–$\mathrm{LLM}(mcq_i)$9 choices and $1$0–$1$1 in a 0-shot setting, CI values are Gran $1$2, DSeek $1$3, Mist $1$4, and Llam $1$5, and CoRA values are Mist $1$6, Llam $1$7, Gran $1$8, and DSeek $1$9.

Two robustness checks are reported. First, an ablation restricted to ten MedQA variants that preserve the original number of alternatives yields lower MCQA+ and MV than the all-variant runs, while CoRA slightly increases, suggesting reduced sensitivity with smaller $\mathrm{MCQA}$00 but similar overall conclusions. Second, bootstrap resampling on MedQA uses $\mathrm{MCQA}$01 runs, each aggregating $\mathrm{MCQA}$02 randomly sampled variant prompts with replacement from the full $\mathrm{MCQA}$03, and yields low standard deviations, with CI and CoRA standard deviations on the order of $\mathrm{MCQA}$04.

A recurrent misconception is that this CI merely duplicates self-consistency. The paper distinguishes them sharply: self-consistency aggregates multiple stochastic decodes of the same prompt, whereas CI measures stability under structural changes to answer choices, not sampling variability or temperature effects.

5. CI in pairwise comparison matrices

In pairwise comparison theory, a PCM $\mathrm{MCQA}$05 is positive and reciprocal: $\mathrm{MCQA}$06, $\mathrm{MCQA}$07, and $\mathrm{MCQA}$08. Consistency requires multiplicative transitivity,

$\mathrm{MCQA}$09

or equivalently the existence of a positive priority vector $\mathrm{MCQA}$10 such that $\mathrm{MCQA}$11 for all $\mathrm{MCQA}$12; in that case $\mathrm{MCQA}$13 (Brunelli, 2015).

Saaty’s Consistency Index is the spectral quantity

$\mathrm{MCQA}$14

where $\mathrm{MCQA}$15 is the Perron root of $\mathrm{MCQA}$16 (Brunelli, 2015). The associated Consistency Ratio is

$\mathrm{MCQA}$17

and a common recommendation is $\mathrm{MCQA}$18, with refinements $\mathrm{MCQA}$19 and $\mathrm{MCQA}$20 discussed in the literature (1311.0748, Brunelli, 2015). These thresholds are the best-established acceptance rules among PCM inconsistency measures.

The Peláez–Lamata Consistency Index is different in construction. It is triad-based: for $\mathrm{MCQA}$21, the index equals the determinant of the reciprocal $\mathrm{MCQA}$22 matrix; for $\mathrm{MCQA}$23, it is the average determinant over all $\mathrm{MCQA}$24 triads (1311.0748). For a triad with log-variables $\mathrm{MCQA}$25, if

$\mathrm{MCQA}$26

then consistency requires $\mathrm{MCQA}$27, and the determinant becomes

$\mathrm{MCQA}$28

which is minimized at $\mathrm{MCQA}$29 and increases symmetrically with $\mathrm{MCQA}$30. This yields several properties emphasized in the paper: nonnegativity, vanishing exactly on consistent matrices, monotonic increase with triad deviation, invariance under relabeling of items, and convexity in log-space.

The contrast between Saaty’s CI and Peláez–Lamata’s CI is substantive. Saaty’s index is global and spectral; Peláez–Lamata’s index is local-to-global, averaging triadic deviations. The latter has no generally accepted threshold in practice. The paper therefore treats a threshold $\mathrm{MCQA}$31 as a decision-maker-specified parameter rather than as a calibrated universal standard (1311.0748).

6. Optimization, thresholding, and decision support in the PCM setting

The 2013 treatment of Peláez–Lamata CI casts inconsistency reduction as mixed-integer optimization in logarithmic space (1311.0748). Let $\mathrm{MCQA}$32 be the log-transformed original PCM, let $\mathrm{MCQA}$33 denote the log of the modified comparison value, and let binary variables $\mathrm{MCQA}$34 indicate whether an upper-triangular entry is modified. Reciprocity is enforced by $\mathrm{MCQA}$35, and bounds $\mathrm{MCQA}$36 encode prior ratio-scale limits.

A threshold-driven problem minimizes the number of modified entries subject to a translated CI constraint. For Peláez–Lamata CI, the paper defines

$\mathrm{MCQA}$37

which converts the average-of-determinants threshold into a bound on the sum of exponential triad terms. The resulting formulation is a mixed 0–1 convex optimization problem. A second, budget-driven problem minimizes the inconsistency level achievable when at most $\mathrm{MCQA}$38 upper-triangular elements may be changed. The paper’s Theorem 9 states that if $\mathrm{MCQA}$39 denotes the optimum value of the budget-driven program, then $\mathrm{MCQA}$40 is the minimal value of inconsistency CI obtainable by modifying at most $\mathrm{MCQA}$41 elements above the main diagonal of $\mathrm{MCQA}$42 and their reciprocals.

These formulations support interactive decision revision. The threshold-driven program identifies the smallest set of judgments whose modification renders the PCM acceptable under a chosen $\mathrm{MCQA}$43; the budget-driven program quantifies how far inconsistency can be reduced under a fixed revision budget. Because the continuous relaxations are convex in log-space, branch-and-bound or branch-and-cut methods for mixed-integer convex programs are applicable. The same paper also describes how to enumerate all optimal modification patterns by fixing the objective at its optimal value and adding exclusion constraints to avoid previously found binary solutions.

An illustrative $\mathrm{MCQA}$44 example is given explicitly. For

$\mathrm{MCQA}$45

the four triad determinants are approximately $\mathrm{MCQA}$46, $\mathrm{MCQA}$47, $\mathrm{MCQA}$48, and $\mathrm{MCQA}$49, so $\mathrm{MCQA}$50. If the decision maker sets $\mathrm{MCQA}$51, the matrix is not acceptable. Modifying a single entry, $\mathrm{MCQA}$52 from $\mathrm{MCQA}$53 to $\mathrm{MCQA}$54 and reciprocally $\mathrm{MCQA}$55 from $\mathrm{MCQA}$56 to $\mathrm{MCQA}$57, makes one affected triad exactly consistent and reduces another to approximately $\mathrm{MCQA}$58, yielding a new average $\mathrm{MCQA}$59.

7. Interpretation, debates, and recurrent points of confusion

Across domains, CI should be understood as a task-specific statistic rather than a universal scale. In the LLM benchmark framework, CI is a multiplicative quality factor on raw MCQA and is explicitly designed to expose cases where top raw accuracy masks low response consistency under altered answer choices (Cavalin et al., 26 Nov 2025). In pairwise comparisons, CI is a measure of inconsistency in reciprocal judgments, but the literature contains competing philosophies: global spectral aggregation in Saaty’s CI, average triad inconsistency in Peláez–Lamata’s CI, and worst-triad diagnostics in Koczkodaj’s index $\mathrm{MCQA}$60 (1311.0748, Brunelli, 2015).

A central controversy in the pairwise comparison literature concerns global versus local inconsistency. The commentary argues that a “local worsening” of one triad need not increase a global inconsistency measure, because one modified entry belongs to $\mathrm{MCQA}$61 distinct triads and can worsen some while improving others (Brunelli, 2015). This is used to defend global measures such as CI against criticisms based on worst-case reasoning. The same commentary also notes that CI is nonnegative and unbounded above in general, whereas Koczkodaj’s index is bounded in $\mathrm{MCQA}$62 and focuses exclusively on the worst triad. Neither perspective dominates categorically; rather, they support different diagnostic aims.

Thresholding is another source of misunderstanding. For PCMs, only Saaty’s CR has widely used acceptance thresholds, notably the “ten percent rule.” Peláez–Lamata CI has no generally accepted threshold, so any admissibility level must be selected contextually by the decision maker (1311.0748). In the LLM setting, CI itself already lies in $\mathrm{MCQA}$63, but it is not an “accept/reject” thresholding device; it is a scalar that dampens inflated raw scores through CoRA.

A plausible implication is that CI is best treated as part of a reporting bundle rather than as a solitary verdict. The LLM paper recommends reporting MCQA, MCQA+, MV, the BMCA curve, CI, and CoRA together, with particular attention to $\mathrm{MCQA}$64 and CI for interpreting how much raw accuracy persists under perturbations (Cavalin et al., 26 Nov 2025). The pairwise comparison literature similarly supports mixed use of global and local indices: CI or CR for overall screening, and a local triadic index when targeted repair or quality control is required (Brunelli, 2015).

In that sense, “Consistency Index” names a family resemblance rather than a single construct. Its common function is to quantify the gap between observed performance or judgment data and an ideal of structural coherence; its concrete instantiation depends on whether coherence means robustness to altered answer choices, multiplicative transitivity in reciprocal matrices, or spectral proximity to rank-one consistency.