Confidence-Consistency Evaluation (CCE)

• Confidence-Consistency Evaluation (CCE) is a framework that defines confidence as a local invariance property under perturbations and semantic variations.
• It employs metrics such as Expected Calibration Error (ECE) and Within-Question Discrimination to distinguish reliable from unreliable predictions.
• CCE addresses domain ambiguities by integrating evaluation and training strategies to enhance model robustness and calibration.

Searching arXiv for papers on confidence-consistency evaluation and related formulations. Confidence-Consistency Evaluation (CCE) denotes a family of evaluation, calibration, and training ideas in which confidence is judged not only by aggregate confidence–accuracy matching but also by some notion of stability. In this literature, the central question is whether a high-confidence prediction remains coherent under local perturbations, among competing answers to the same question, across semantically equivalent prompt or answer variations, or within a structured neighborhood of related facts (Tao et al., 2024, Taubenfeld et al., 10 Feb 2025, Xia et al., 12 Jan 2026, Xu et al., 9 Jan 2026). Across papers, the term is not fully standardized: some works present CCE as an explicit metric, some as a viewpoint on calibration, and others use the acronym for different objects such as “Consensus Cross-Entropy” or the unrelated AFLOW-CCE materials framework (Hejabi et al., 16 Oct 2025, Friedrich et al., 2023).

1. Reliability-based calibration and the turn to consistency

Classical calibration treats confidence as a reliability problem. In the formulation emphasized by the consistency-calibration literature, reliability requires

$\mathbb{P}(\hat{y}=y \mid \hat{p}=p)=p,$

and Expected Calibration Error (ECE) approximates the mismatch between confidence and empirical accuracy through confidence bins: $$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|A_m-C_m\right|.$$ This is a global, bin-based view: samples are grouped by similar confidence values, and calibration is assessed by how well empirical correctness matches reported confidence inside each bin (Tao et al., 2024).

Several papers argue that this perspective is insufficient for the way confidence is actually used. In reasoning-time aggregation, a confidence score need not merely be calibrated across questions; it must distinguish correct and incorrect sampled answers to the same question. That motivates the “within-question confidence evaluation” view and the Within-Question Discrimination metric,

$$\text{WQD}(c) \equiv \frac{1}{N} \cdot \sum_{q} \sum_{(r, a)\in R^+_q} \sum_{(r', a')\in R^-_q} [c(r,a) > c(r',a')],$$

which measures whether correct responses receive higher confidence than incorrect responses for the same prompt (Taubenfeld et al., 10 Feb 2025).

An analogous critique appears in language-variation studies. Calibration and discrimination can both look strong while confidence still fluctuates under semantically equivalent prompt reformulations, varies across equivalent answers, or fails to react to semantically different answers. This motivates three additional axes: robustness to prompt perturbations, stability across semantic-equivalent answers, and sensitivity to semantic differences (Xia et al., 12 Jan 2026). The common theme is a shift from marginal correctness-frequency matching toward local, semantic, or structural coherence.

2. Local, neighborhood, and structural formulations

In image classification, consistency is formalized as prediction stability under perturbation. For a sample $x$, with perturbed neighbors $\tilde{x}_t$ satisfying $d(\tilde{x}_t,x)<\epsilon^*$, the consistency score for class $k$ is

$c_k(x)=\frac{1}{T}\sum_{t=1}^{T}\mathbbm{1}(\hat{y}(\tilde{x}_t)=k),$

and the consistency-calibrated ideal is

$\hat{p}_k(x)=c_k(x).$

Consistency Calibration then replaces the original confidence by a perturbation-frequency estimate,

$\hat{p}'_k = \frac{1}{T}\sum_{t=1}^{T}\mathbbm{1}\left(\argmax q(\widetilde{h(x)}^t)=k\right),$

with perturbations applied at the data, feature, or logit level. The paper reports that hard-vote consistency aggregation is slightly better than a softmax-averaging baseline, especially on larger datasets, and that logit-level perturbation provides the best efficiency–performance trade-off (Tao et al., 2024).

A stronger version appears in work on truthful belief in LLMs. There, point-wise self-consistency is treated as inadequate because a model can answer correctly in all samples for a target question yet collapse under contextual interference. The proposed Neighbor-Consistency Belief (NCB) score operationalizes belief robustness over a neighborhood of related facts: $$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|A_m-C_m\right|.$$0 The geometric-mean correction $$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|A_m-C_m\right|.$$1 is introduced to avoid exponential decay with neighborhood size. Empirically, the motivating example selects 995 questions on which Qwen3-30B-A3B-Instruct has perfect self-consistency, $$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|A_m-C_m\right|.$$2, yet accuracy collapses from $$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|A_m-C_m\right|.$$3 to $$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|A_m-C_m\right|.$$4 when contextual interference is added. In that setting, neighborhood consistency is intended to capture whether a fact is embedded in a coherent belief structure rather than reproduced in isolation (Xu et al., 9 Jan 2026).

These formulations share a common structure: confidence is reinterpreted as a local invariance property. The locality may be geometric, as in perturbed image neighborhoods; semantic, as in paraphrased prompts or equivalent answers; or conceptual, as in related prerequisite and implication facts. This suggests that “confidence” is increasingly being treated as a statement about neighborhoods rather than a scalar attached to a single forward pass.

3. Consistency-based estimators and post-hoc calibrators

Consistency can be used directly as a confidence estimator. In Consistency Calibration, confidence is computed from the fraction of perturbations preserving the predicted class, without extra labeled data, retrieval of external neighbors, or retraining. The method studies weak, moderate, and stronger image augmentations, reports that moderate perturbations often improve calibration while preserving accuracy, and finds that overly strong perturbations destroy recognizability and worsen calibration. Logit-level perturbation is especially attractive because only the argmax on perturbed logits must be repeated (Tao et al., 2024).

In reasoning tasks, Confidence-Informed Self-Consistency (CISC) replaces unweighted majority vote with a confidence-weighted vote: $$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|A_m-C_m\right|.$$5 The paper reports that CISC reduces the required number of reasoning paths by over $$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|A_m-C_m\right|.$$6 on average, and that the most calibrated confidence method is not the most useful for CISC. This is precisely the within-instance use case for which WQD was introduced (Taubenfeld et al., 10 Feb 2025).

Multi-Perspective Consistency (MPC) extends the same logic to LLM confidence estimation by combining internal self-verification and cross-model agreement. MPC-Internal counts how often an answer survives verifier-style self-reflection, $$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|A_m-C_m\right|.$$7, while MPC-Across fuses scores from another model, optionally with knowledge injection. The final score is

$$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|A_m-C_m\right|.$$8

with $$\text{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{N}\left|A_m-C_m\right|.$$9 in the main experiments. On eight datasets, MPC is reported to achieve state-of-the-art performance and to reduce overconfidence on incorrect answers (Wang et al., 2024).

CRUX adds an explicitly context-aware variant. It combines contextual entropy reduction,

$$\text{WQD}(c) \equiv \frac{1}{N} \cdot \sum_{q} \sum_{(r, a)\in R^+_q} \sum_{(r', a')\in R^-_q} [c(r,a) > c(r',a')],$$0

with unified consistency examination over the pooled set of context-conditioned and context-free answers, and then fuses both signals with a two-layer MLP: $$\text{WQD}(c) \equiv \frac{1}{N} \cdot \sum_{q} \sum_{(r, a)\in R^+_q} \sum_{(r', a')\in R^-_q} [c(r,a) > c(r',a')],$$1 The method is motivated by contextual question answering, where answer consistency alone can be misleading if it reflects memorized knowledge rather than faithfulness to the supplied context (Yuan et al., 1 Aug 2025).

4. Training objectives and shift-aware consistency

Some work moves from evaluation to direct optimization. In Flip-Flop Consistency, CCE stands for Consensus Cross-Entropy: for each input rendered by multiple prompt templates, a strict majority label becomes a hard pseudo-label, and every prompt variation is trained toward it. The pseudo-label is trusted only when $$\text{WQD}(c) \equiv \frac{1}{N} \cdot \sum_{q} \sum_{(r, a)\in R^+_q} \sum_{(r', a')\in R^-_q} [c(r,a) > c(r',a')],$$2; otherwise the example is skipped. CCE is then combined with representation-alignment losses that align lower-confidence or non-majority prompt variants toward the confident consensus set. The paper reports that full $$\text{WQD}(c) \equiv \frac{1}{N} \cdot \sum_{q} \sum_{(r, a)\in R^+_q} \sum_{(r', a')\in R^-_q} [c(r,a) > c(r',a')],$$3 raises observed agreement by $$\text{WQD}(c) \equiv \frac{1}{N} \cdot \sum_{q} \sum_{(r, a)\in R^+_q} \sum_{(r', a')\in R^-_q} [c(r,a) > c(r',a')],$$4, improves mean $$\text{WQD}(c) \equiv \frac{1}{N} \cdot \sum_{q} \sum_{(r, a)\in R^+_q} \sum_{(r', a')\in R^-_q} [c(r,a) > c(r',a')],$$5 by $$\text{WQD}(c) \equiv \frac{1}{N} \cdot \sum_{q} \sum_{(r, a)\in R^+_q} \sum_{(r', a')\in R^-_q} [c(r,a) > c(r',a')],$$6, and reduces performance variance across formats by $$\text{WQD}(c) \equiv \frac{1}{N} \cdot \sum_{q} \sum_{(r, a)\in R^+_q} \sum_{(r', a')\in R^-_q} [c(r,a) > c(r',a')],$$7, while the CCE-only variant already captures most of the agreement gains (Hejabi et al., 16 Oct 2025).

Under covariate shift, expectation consistency provides a different notion of confidence consistency. The key condition is

$$\text{WQD}(c) \equiv \frac{1}{N} \cdot \sum_{q} \sum_{(r, a)\in R^+_q} \sum_{(r', a')\in R^-_q} [c(r,a) > c(r',a')],$$8

which the paper proves is necessary and sufficient for preserving calibration across source and target domains. Expectation Consistency Loss (ECL) turns this condition into an unsupervised domain adaptation objective compatible with canonical, class-wise, and top-label calibration. The paper further states that computing ECL has the same sample complexity as ECE and gives a mini-batch trainable scheme based on an auxiliary-variable reformulation (Dong et al., 20 May 2026).

Neighborhood consistency has also been turned into a training signal for factual robustness. Structure-Aware Training (SAT) optimizes context-invariant belief structure by distilling a frozen teacher across Neighbor Contexts and General Contexts through a KL-divergence objective,

$$\text{WQD}(c) \equiv \frac{1}{N} \cdot \sum_{q} \sum_{(r, a)\in R^+_q} \sum_{(r', a')\in R^-_q} [c(r,a) > c(r',a')],$$9

The paper reports that SAT reduces long-tail knowledge brittleness by approximately $x$0 (Xu et al., 9 Jan 2026).

5. Benchmarks, metrics, and empirical regimes

The empirical literature now evaluates confidence consistency through dedicated benchmarks rather than through calibration plots alone. ConfProBench targets multimodal LLM process judges and defines three complementary scores: Confidence Robustness Score (CRS), Confidence Sensitivity Score (CSS), and Confidence Calibration Score (CCS). The benchmark perturbs reasoning steps by synonym substitution, syntactic transformation, and image perturbation, and evaluates 14 MLLMs. Its reported findings are that confidence robustness is imperfect even for strong models, syntactic transformation is the hardest perturbation type, calibration is often the weakest axis, and step-classification accuracy does not imply reliable confidence (Zhou et al., 6 Aug 2025).

For behavioral comparison of classifiers, error consistency (EC) had long been used without uncertainty quantification. Recent work adds bootstrap confidence intervals, significance tests, and a copy-model interpretation in which EC can be read as an implicit copying probability when marginals match. The paper revisits model-vs-human and Brain-Score analyses and concludes that many reported differences between deep vision models are statistically insignificant once uncertainty is propagated through the full pipeline. It recommends collecting at least 1000 trials per classifier as a rule of thumb (Klein et al., 9 Jul 2025).

In time-series anomaly detection, CCE is an explicit evaluation metric combining confidence and uncertainty consistency. Event-level confidence is multiplied by event-level consistency, global anomaly and normal scores are aggregated, and the final metric is

$x$1

Uncertainty is estimated through a Beta model with variance

$x$2

The paper proves strict boundedness, Lipschitz robustness against score perturbations, and linear time complexity $x$3, and introduces RankEval as a standardized pipeline for comparing the ranking capability of anomaly-detection metrics (Zhong et al., 1 Sep 2025).

Across these benchmarks, a recurring result is that well-calibrated confidence is neither sufficient for semantic robustness nor guaranteed to be useful for aggregation, belief robustness, or process judging. This has become one of the field’s central methodological claims.

6. Acronym ambiguity and adjacent usages

The acronym “CCE” is overloaded across domains, so interpretation depends on context.

Usage of “CCE”	Meaning	Representative paper
Confidence-consistency evaluation	Confidence assessed through perturbation, semantic, or neighborhood stability	(Tao et al., 2024)
Consensus Cross-Entropy	Majority-vote pseudo-label loss across prompt variations	(Hejabi et al., 16 Oct 2025)
Time-series anomaly-detection CCE	Explicit metric combining confidence and uncertainty consistency	(Zhong et al., 1 Sep 2025)
Conditional congruence / MCMD	Point-wise conditional distribution discrepancy for regressors	(Young et al., 2024)
AFLOW-CCE	Coordination Corrected Enthalpies for ionic materials	(Friedrich et al., 2023)

In regression, the relevant paper explicitly uses Maximum Conditional Mean Discrepancy (MCMD) as the practical estimator, but also frames the contribution as a conditional congruence error in a CCE-style sense: the goal is to measure, at any point $x$4, the discrepancy between the learned predictive distribution and the empirical conditional distribution rather than relying on marginal PIT-style calibration (Young et al., 2024). In materials science, AFLOW-CCE is unrelated to uncertainty evaluation; it denotes coordination corrected enthalpies and reduces formation-enthalpy deviations for oxides and nitrides to roughly the room-temperature thermal scale, about $x$5 meV/atom (Friedrich et al., 2023). In survey methodology, dynamic latent class analysis has likewise been used to assess the consistency of consumer confidence responses, but there “confidence” refers to economic sentiment rather than predictive uncertainty (Kumar et al., 2015).

The broader significance of CCE, in the machine-learning sense, is therefore conceptual rather than terminological. It marks a transition from treating confidence as a single scalar to treating it as a relational property: confidence should agree with correctness, but it should also remain stable under harmless variation and change when the underlying semantic or structural state changes.