Confidence vs. Accuracy: DCA

• Difference between Confidence and Accuracy (DCA) is a measure that quantifies the gap between a model’s predicted probability and its true accuracy, highlighting calibration issues.
• DCA employs metrics like Expected Calibration Error (ECE), Static Calibration Error (SCE), and Confidence-Weighted AUC to evaluate overconfidence and underconfidence in predictions.
• Empirical findings indicate that addressing DCA can enhance model reliability, improve calibration in safety-critical applications, and support effective domain adaptation.

The difference between confidence and accuracy (DCA) quantifies the discrepancy between a model’s stated belief in its own predictions (confidence) and the proportion of those predictions that are actually correct (accuracy). This distinction underpins the concept of calibration, central to the risk assessment and reliability of machine learning systems, especially in safety-critical or distribution-shifted settings. The DCA framework provides both theoretical formalisms and a practical suite of metrics for measuring, diagnosing, and improving calibration across classification architectures.

1. Formal Definitions: Confidence Versus Accuracy

Let a classifier $f$ output a probability vector $s(x)\in[0,1]^K$ for input $x$, with predicted label $\hat{y}(x) = \arg\max_j s_j(x)$. The confidence $c(x)$ is the highest probability assigned to the predicted class: $c(x) = \max_j s_j(x)$ The accuracy $A$ over a dataset $D = \{(x_i,y_i)\}_{i=1}^N$ is the empirical fraction of correct predictions: $$A = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{\hat{y}(x_i) = y_i\}$$ A model is perfectly calibrated if, for any confidence $p\in[0,1]$, the likelihood of correctness among all predictions made with confidence $s(x)\in[0,1]^K$0 is exactly $s(x)\in[0,1]^K$1: $s(x)\in[0,1]^K$2 The difference between confidence and accuracy for a set of predictions is thus: $s(x)\in[0,1]^K$3 A positive DCA indicates overconfidence; a negative value signals underconfidence (Ruijs et al., 11 Mar 2026, Ao et al., 2023, Kivimäki et al., 2024).

2. Calibration Metrics and the Measurement of DCA

The Expected Calibration Error (ECE) remains the classical metric for quantifying DCA. The dataset is partitioned into $s(x)\in[0,1]^K$4 bins by confidence. For bin $s(x)\in[0,1]^K$5 with confidence interval $s(x)\in[0,1]^K$6,

$s(x)\in[0,1]^K$7

where $s(x)\in[0,1]^K$8 is the set of samples with confidence in $s(x)\in[0,1]^K$9, $x$0 is the fraction of correct predictions, and $x$1 is the mean confidence in the bin (Martin-Maroto et al., 3 May 2026, Nixon et al., 2019, Ao et al., 2023). Extensions such as Static Calibration Error (SCE) and Adaptive Calibration Error (ACE) also account for full vector probabilities and adaptive binning (Nixon et al., 2019).

Class-wise miscalibration score (MCS) captures per-class DCA: for class $x$2, $x$3 (mean predicted confidence for $x$4) $x$5 (empirical frequency of correct $x$6 predictions) (Ao et al., 2023).

These metrics are often complemented by Maximum Calibration Error (MCE) and, for adaptive settings, versions such as adaECE (Penso et al., 2024).

3. Modern Metrics Beyond ECE: CSR, Risk, and Confidence-Weighted Utility

Despite its popularity, ECE is insensitive to dangerous, high-probability miscalibration (“tail risk”), as a few overconfident errors can have minimal impact on the average. The Calibrated Size Ratio (CSR) directly targets tail risk: $x$7 Under perfect calibration, $x$8; $x$9 signals overconfidence at high-probability regions. The associated risk probability $\hat{y}(x) = \arg\max_j s_j(x)$0 (from a Gaussian test) quantifies the one-sided $\hat{y}(x) = \arg\max_j s_j(x)$1-value for overconfidence (Martin-Maroto et al., 3 May 2026).

Confidence-weighted accuracy (cwA) and confidence-weighted AUC (cwAUC) capture the discriminative utility of confidence. cwA exceeds the raw accuracy when confidence correlates with correctness; cwAUC detects when high confidence specifically helps to separate correct and incorrect predictions (Martin-Maroto et al., 3 May 2026).

Metric	Captures	Key Application
ECE	Average DCA	Miscalibration screening
CSR, $\hat{y}(x) = \arg\max_j s_j(x)$2	Tail risk	Overconfidence alarm
cwA, cwAUC	Discrimination	Confidence utility, ranking

4. Theoretical Guarantees and Calibration in Practice

The Average Confidence (AC) estimator $\hat{y}(x) = \arg\max_j s_j(x)$3 is unbiased and consistent for true accuracy under perfect calibration and independence: $\hat{y}(x) = \arg\max_j s_j(x)$4 with variance decaying as $\hat{y}(x) = \arg\max_j s_j(x)$5. Under miscalibration ($\hat{y}(x) = \arg\max_j s_j(x)$6), AC becomes biased: overconfident models ($\hat{y}(x) = \arg\max_j s_j(x)$7 systematically too high) overestimate accuracy, and vice versa (Kivimäki et al., 2024).

Temperature scaling and class-wise temperature scaling (cwMCS-TS) can correct miscalibration. However, global scaling may overcompensate, causing class-imbalances or under-confidence for many classes, while class-wise approaches directly minimize both sides of DCA per class (Ao et al., 2023).

In domain adaptation, direct accuracy estimation on unlabeled target data is infeasible. Methods such as UTDC estimate target-domain accuracy and adjust confidences so that bin-level DCA is minimized (Penso et al., 2024).

Key theoretical implications:

• Perfect calibration $\hat{y}(x) = \arg\max_j s_j(x)$8 DCA $\hat{y}(x) = \arg\max_j s_j(x)$9 in all bins and globally.
• DCA serves as both a quantitative risk measure and a practical basis for monitoring and calibration adjustment.
• Variance bounds and Poisson-binomial intervals enable control charts for operational settings (Kivimäki et al., 2024).

5. Empirical Findings and Impact on Downstream Tasks

Empirical studies on CNNs across datasets such as Fashion-MNIST, CIFAR-100, and ImageNet show DCA ranging from 1–7% depending on model architecture and calibration method (Ruijs et al., 11 Mar 2026, Xia et al., 2021, Ao et al., 2023). Bayesian inference (MC Dropout), conformal prediction, or explicit MDCA regularization at train time can dramatically reduce DCA, improve reliability diagrams, and lower risk-coverage curves, all while preserving or negligibly affecting top-1 accuracy (Hebbalaguppe et al., 2022).

Overconfidence is tightly linked to robustness under quantization—overconfident bins are rarely flipped by noise, and swaps among low-confidence, low-accuracy predictions minimally harm accuracy, raising a tension in model design between quantization resilience and calibration (Xia et al., 2021).

Metacognitive sensitivity—how well confidence distinguishes correct from incorrect predictions—can actually outweigh raw accuracy in human-AI decision teams. Models with lower accuracy but better metacognitive AUC yield superior team performance, highlighting DCA as an operational lever beyond solo model metrics (Li et al., 30 Jul 2025).

6. Practical Recommendations for Model Monitoring and Calibration

• Regularly monitor both accuracy and DCA (ECE, SCE, ACE, MCS, CSR) on current data and after deployment (Kivimäki et al., 2024, Nixon et al., 2019).
• Employ adaptive, class-conditional or per-class metrics with L2 norm (e.g., ACE or SCE) for stable and discriminative DCA measurement (Nixon et al., 2019).
• For safety-critical applications, combine CSR + $c(x)$0 for overconfidence alarms, cwA/cwAUC for discriminative utility, and raw accuracy as a completeness check (Martin-Maroto et al., 3 May 2026).
• In domain adaptation, directly estimate target accuracy and calibrate to minimize DCA on unsupervised data (Penso et al., 2024).
• Hyperparameterize loss functions (e.g., MDCA, $c(x)$1) to control the tradeoff between calibration and raw accuracy; too strong a penalty may degrade accuracy (Hebbalaguppe et al., 2022).

7. Limitations and Open Challenges

DCA-based estimation and correction fundamentally presuppose calibration holds. Severe miscalibration, support shift, or concept drift will typically invalidate DCA-based monitoring and require complementary data-shift or retraining interventions (Kivimäki et al., 2024, Nixon et al., 2019). No single metric suffices; ECE lacks tail risk sensitivity, CSR may miss distributional issues with low-confidence bins, and cwA is uninformative under flat confidences. Full reporting should include multiple DCA metrics with details on binning, norm, and conditioning.

A plausible implication is that continued research on DCA must focus on robust, theoretically grounded, and discriminative utility measures for confidence—especially under distribution shift and high-stakes decision regimes. This includes both new metrics (e.g., CSR, cwAUC) and improved calibration correction algorithms targeting DCA at the instance and class level.

---

References:

• (Martin-Maroto et al., 3 May 2026)
• (Kivimäki et al., 2024)
• (Ao et al., 2023)
• (Nixon et al., 2019)
• (Xia et al., 2021)
• (Hebbalaguppe et al., 2022)
• (Penso et al., 2024)
• (Ruijs et al., 11 Mar 2026)
• (Li et al., 30 Jul 2025)