Confidence Uncertainty: Concepts & Methods

• Confidence uncertainty is a framework that links a model’s calibrated score to its underlying uncertainty, integrating probability calibration, uncertainty metrics, and task-specific assessments.
• Methodological families such as post-hoc calibration and probabilistic modeling implement confidence uncertainty to optimize predictions across diverse domains including ML and quantum mechanics.
• Empirical findings in fields like medical imaging and agent systems demonstrate that refined uncertainty quantification improves performance, reliability, and decision-making.

Confidence uncertainty denotes a family of formalisms that relate a system’s confidence score to the uncertainty of its prediction, decision, or physical state. In machine learning, confidence is typically treated as the estimated probability that a prediction is correct, while uncertainty quantification studies whether that estimate is calibrated, what sources of uncertainty it reflects, and how it should be propagated or acted upon in downstream computation. The recent literature uses the term across calibrated classification, LLM uncertainty elicitation, medical imaging, ranking, and confidence-interval propagation for PDEs; in quantum mechanics, it also appears as a distinct notion defined by the minimal support measure needed to contain probability at least $\theta$ (Liu et al., 20 Mar 2025, Lin et al., 6 May 2026).

1. Core definitions and uncertainty types

A standard calibration definition treats a model with raw score $r \in [0,1]$ as perfectly calibrated if $\Pr(y=1 \mid r=p)=p$ for all $p \in [0,1]$; miscalibration occurs when empirical accuracy at confidence $p$ deviates from $p$ (Küppers et al., 2021). In open-ended and multimodal settings, closely related definitions appear. In visual question answering, “model confidence” is defined as

$$P(\text{correct})=\sum_{i \in \text{correct}} P(i),$$

using top-$k$ token probabilities under greedy or very low-temperature decoding (Liu et al., 20 Jun 2025). In the confidence-first LLM paradigm, a model emits a score $s \in [0,1]$ before answering, and this score is interpreted as $\Pr(\text{correct}\mid x)$ under the current policy (Li et al., 6 Mar 2026). In embodied agents, the agent verbalizes a confidence $r \in [0,1]$0 for whether its next perception or action will be correct, equivalently $r \in [0,1]$1 (Yu et al., 13 Mar 2025).

The dominant decomposition separates aleatoric from epistemic uncertainty. In open-ended tasks with multiple valid answers, aleatoric uncertainty is the irreducible randomness among semantically distinct correct answers, whereas epistemic uncertainty reflects ignorance about which answers are correct versus incorrect. One formulation defines

$r \in [0,1]$2

and

$r \in [0,1]$3

with total entropy decomposing as

$r \in [0,1]$4

This same literature argues that LLMs require a broader taxonomy, including input ambiguity uncertainty, reasoning-path divergence uncertainty, and decoding-stochasticity uncertainty in addition to classical aleatoric and epistemic uncertainty (Liu et al., 20 Jun 2025, Liu et al., 20 Mar 2025).

Confidence is therefore not a single scalar concept. Across the literature, it can mean a calibrated probability of correctness, a distributional object derived from posterior or ensemble variation, a verbalized self-assessment, or a task-specific summary of structured ambiguity. This suggests that “confidence uncertainty” is best understood as a relation between a score, the mechanism producing it, and the uncertainty class it is intended to encode.

2. Methodological families

One major family is post-hoc calibration. Geometric calibration methods replace the model’s internal score by a distance-based signal computed from the training set, then calibrate that signal with standard methods such as isotonic regression or Platt scaling. In the real-time geometric formulation, for a test input $r \in [0,1]$5 with predicted label $r \in [0,1]$6, one defines training points of the same predicted class $r \in [0,1]$7 and all others $r \in [0,1]$8, and uses the fast-separation proxy

$r \in [0,1]$9

where $\Pr(y=1 \mid r=p)=p$0. Positive values signal high confidence and negative values low confidence; isotonic regression on $\Pr(y=1 \mid r=p)=p$1 then yields calibrated probabilities (Chouraqui et al., 2022). A closely related formulation introduces maximal geometric separation $\Pr(y=1 \mid r=p)=p$2 and its fast approximation $\Pr(y=1 \mid r=p)=p$3 as a raw uncertainty signal in a fixed representation space, again followed by isotonic or sigmoid calibration (Chouraqui et al., 2023).

A second family makes the calibrator or predictor itself probabilistic. Bayesian confidence calibration places a posterior distribution over calibration parameters $\Pr(y=1 \mid r=p)=p$4, approximated with stochastic variational inference, and returns a posterior predictive distribution

$\Pr(y=1 \mid r=p)=p$5

rather than a single calibrated estimate (Küppers et al., 2021). Uncertainty-aware Dirichlet networks instead predict concentration parameters $\Pr(y=1 \mid r=p)=p$6 with $\Pr(y=1 \mid r=p)=p$7, so that expected class probabilities are $\Pr(y=1 \mid r=p)=p$8 and the oracle true-class probability becomes $\Pr(y=1 \mid r=p)=p$9 (Tsiligkaridis, 2020). Hyperspherical Confidence Mapping uses a different geometric reformulation: the network output $p \in [0,1]$0 is decomposed into a magnitude $p \in [0,1]$1 and a direction vector $p \in [0,1]$2, with uncertainty defined by violation of the unit-hypersphere constraint through scores such as

$p \in [0,1]$3

and confidence $p \in [0,1]$4 (Choi et al., 7 May 2026).

A third family is confidence-first or interface-centric. CoCA formulates sequence generation as an MDP in which the model must first emit a confidence segment $p \in [0,1]$5 and only then an answer segment $p \in [0,1]$6, optimizing answer correctness and calibration jointly with segmented credit assignment under GRPO (Li et al., 6 Mar 2026). In agent systems, the central problem is no longer only score estimation but preservation of upstream fragility across module boundaries. This view defines an uncertain decision handoff as commitment from an internal distribution $p \in [0,1]$7 to a single artifact $p \in [0,1]$8, and proposes augmenting the handoff to

$p \in [0,1]$9

where $p$0 is a latent uncertainty carrier derived from the hidden state and passed downstream for recovery or rerouting (Shi et al., 9 Jun 2026).

These families differ in where uncertainty is represented: in a calibrated scalar, a posterior predictive distribution, a structured latent embedding, or an interface contract. The variation is not terminological only; it determines what downstream decisions are possible.

3. Domain-specific implementations

Several application areas instantiate confidence uncertainty with domain-specific objects rather than generic confidence scores.

Domain	Formal object	Key mechanism
Dermatological lesion classification	$p$1	Per-feature confidence $p$2 and confidence pooling of latent features
Medical semantic segmentation	$p$3 with $p$4	Two nested contours encode “definitely in,” “possibly in,” and “definitely out”
Back-translation for NMT	Word- and sentence-level uncertainty-based confidence	MC Dropout estimates expectation and variance of translation probabilities
Deterministic PDEs	Joint $p$5-confidence interval $p$6	Bi-level propagation of boundary-data confidence intervals through a PDE solver

In dermatological lesion classification, the backbone produces a feature vector $p$7, a confidence branch predicts variances $p$8, and latent features are modeled as $p$9. Per-feature confidence is defined by $p$0, normalized as $p$1, and the pooled feature is

$p$2

Training is two-stage: first minimize weighted cross-entropy for classification, then freeze the backbone and optimize the uncertainty loss over genuine same-class pairs (Lee et al., 2021).

In medical semantic segmentation, “Confidence Contours” replace a single binary mask with two nested masks. $p$3 contains pixels definitely in the structure, $p$4 contains pixels possibly in the structure, and the image is partitioned as $p$5 with $p$6. Standard segmentation networks can learn this representation with two output channels and the summed loss

$p$7

using Dice loss for both channels in the paper (Ye et al., 2023).

In neural machine translation, uncertainty-based confidence estimation is used to control synthetic data quality in back-translation. Monte Carlo Dropout approximates the expectation and variance of sentence-level and token-level translation probabilities, and variance-derived confidence is used both to down-weight noisy synthetic sentence pairs and to attenuate attention to low-confidence source tokens (Wang et al., 2019).

In deterministic PDEs, Physics-Informed Confidence Propagation defines a joint $p$8-confidence interval for the exact solution $p$9 as random functions $$P(\text{correct})=\sum_{i \in \text{correct}} P(i),$$0 satisfying

$$P(\text{correct})=\sum_{i \in \text{correct}} P(i),$$1

Given a confidence interval for clean boundary data and a solver bias $$P(\text{correct})=\sum_{i \in \text{correct}} P(i),$$2, the propagated bounds at each query point are obtained by outer optimization over admissible boundary data:

$$P(\text{correct})=\sum_{i \in \text{correct}} P(i),$$3

The paper proves validity of the resulting interval under the assumption that the clean boundary data lie in the constructed set with probability at least $$P(\text{correct})=\sum_{i \in \text{correct}} P(i),$$4 (Shen et al., 2023).

Embodied agents add a further variant. Confidence is elicited through Elicitation Policies—Vanilla Self-Assessment, Self-Intervention, Chain-of-Thought, Plan-and-Solve, and Top-K Reasoning—and refined through Execution Policies such as action sampling, scenario reinterpretation, and hypothetical reasoning (Yu et al., 13 Mar 2025).

4. Calibration, validation, and evaluation

Confidence uncertainty is evaluated with both calibration metrics and task-specific validity criteria. Expected Calibration Error is repeatedly defined by binning predictions into confidence bins and computing

$$P(\text{correct})=\sum_{i \in \text{correct}} P(i),$$5

Brier score is likewise standard:

$$P(\text{correct})=\sum_{i \in \text{correct}} P(i),$$6

or, for scalar correctness labels, $$P(\text{correct})=\sum_{i \in \text{correct}} P(i),$$7 (Li et al., 6 Mar 2026, Chouraqui et al., 2022). AUROC is used to measure discrimination between correct and incorrect predictions or between successes and failures (Yu et al., 13 Mar 2025). Detection settings introduce Detection ECE, as well as uncertainty interval metrics such as Prediction Interval Coverage Probability and Mean Prediction Interval Width (Küppers et al., 2021).

Some work argues that evaluation itself must reflect a probabilistic interpretation of uncertainty rather than a deterministic oracle. Confidence curves are constructed by sorting examples by predicted uncertainty and recomputing an error statistic after removing the highest-uncertainty subset. The common oracle reference corresponds to a perfect deterministic error predictor such as $$P(\text{correct})=\sum_{i \in \text{correct}} P(i),$$8, but this paper replaces it with a probabilistic reference generated by sampling $$P(\text{correct})=\sum_{i \in \text{correct}} P(i),$$9 and forming Monte Carlo confidence-curve bands. In that framework, the confidence curve can test both calibration and tightness of prediction uncertainties (Pernot, 2022).

Other domains validate uncertainty through set-valued inference rather than scalar calibration. In ranking, interval estimates $k$0 induce a partial order $k$1 whenever $k$2, and the set of admissible full rankings is the set of linear extensions $k$3. The resulting set estimator is a valid confidence set for the overall ranking, and its size quantifies ranking uncertainty (Rising, 2021).

A consistent theme is that calibration and usefulness are not identical. A score may be numerically calibrated yet fail to preserve uncertainty across interfaces, distinguish aleatoric from epistemic effects, or support operational decisions such as abstention, rerouting, or recovery.

5. Empirical findings and recognized failure modes

Empirical studies generally report that explicit uncertainty modeling can improve both predictive performance and decision quality, although the gains depend strongly on representation, bias, and task structure.

In dermoscopic lesion analysis, adding confidence pooling to EfficientNet-b4 on ISIC 2018 increases F1 from $k$4 to $k$5, ACC from $k$6 to $k$7, BACC from $k$8 to $k$9, and mean AUC from $s \in [0,1]$0 to $s \in [0,1]$1; on ISIC 2019, EfficientNet-b4+CP improves F1 from $s \in [0,1]$2 to $s \in [0,1]$3, ACC from $s \in [0,1]$4 to $s \in [0,1]$5, BACC from $s \in [0,1]$6 to $s \in [0,1]$7, and mean AUC from $s \in [0,1]$8 to $s \in [0,1]$9. Rejecting the bottom $\Pr(\text{correct}\mid x)$0 lowest-confidence inputs on ISIC 2019 raises F1 from $\Pr(\text{correct}\mid x)$1 to $\Pr(\text{correct}\mid x)$2 and BACC from $\Pr(\text{correct}\mid x)$3 to $\Pr(\text{correct}\mid x)$4 (Lee et al., 2021).

In confidence-first LLM training, CoCA reports for the 7B model at pass@1 and temperature $\Pr(\text{correct}\mid x)$5 that math average ECE decreases from $\Pr(\text{correct}\mid x)$6 to $\Pr(\text{correct}\mid x)$7, factual QA ECE decreases from $\Pr(\text{correct}\mid x)$8 to $\Pr(\text{correct}\mid x)$9, AUROC increases by approximately $r \in [0,1]$00–$r \in [0,1]$01 points across categories, Brier scores drop by $r \in [0,1]$02–$r \in [0,1]$03, and confidence is produced after approximately $r \in [0,1]$04 tokens rather than hundreds or thousands, reducing token cost by more than $r \in [0,1]$05 while preserving answer quality within $r \in [0,1]$06–$r \in [0,1]$07 points of answer-only RL or the base model (Li et al., 6 Mar 2026).

Bias studies show that confidence quality is highly state-dependent. In VQA, mitigating text biases raises AUROC from approximately $r \in [0,1]$08 to approximately $r \in [0,1]$09 for GPT-4o on VL_Checklist. Across both GPT-4o and Qwen2-VL, lower bias-free confidence correlates with larger absolute changes in measured epistemic and aleatoric entropy under all considered bias types, with amplification approximately $r \in [0,1]$10–$r \in [0,1]$11 stronger for epistemic uncertainty than for aleatoric uncertainty. Lower bias-free confidence also leads to greater underestimation of epistemic uncertainty, i.e. overconfidence, due to bias (Liu et al., 20 Jun 2025).

Embodied agents exhibit a related pattern. In Minecraft tasks, structured elicitation improves calibration relative to vanilla self-assessment: for GPT-4V, ECE decreases from $r \in [0,1]$12 to $r \in [0,1]$13 under Chain-of-Thought and to $r \in [0,1]$14 under Plan-and-Solve, while Top-K reasoning remains weaker at $r \in [0,1]$15. The paper reports that CoT and PS yield the largest calibration gains and highest AUROCs, but abductive settings remain persistently difficult (Yu et al., 13 Mar 2025).

The literature also identifies failure modes that are not reducible to a bad scalar score. Agent systems can exhibit interface collapse when distinct upstream epistemic states $r \in [0,1]$16 satisfy $r \in [0,1]$17, so that a fragile and a confident state are passed downstream as the same artifact. This is termed confidence laundering: fragile upstream states are repackaged as procedurally valid artifacts that downstream agents over-trust (Shi et al., 9 Jun 2026). In moral-decision experiments, introducing inference-time dropout increases total entropy and especially mutual information, while conditional entropy is largely unchanged; the resulting change in mutual information correlates significantly with improvements in human–LLM moral alignment, with Pearson $r \in [0,1]$18 at dropout rate $r \in [0,1]$19 and $r \in [0,1]$20 at $r \in [0,1]$21 (Kwon et al., 17 Nov 2025).

These results indicate that uncertainty quality depends not only on calibration loss but also on prompt or interface design, bias mitigation, and whether the representation of uncertainty is structurally usable by the next component.

6. Quantum-mechanical confidence uncertainty

In quantum mechanics, “confidence uncertainty” has a specialized meaning unrelated to model calibration. For a pure state $r \in [0,1]$22 with position-wavefunction $r \in [0,1]$23 and confidence level $r \in [0,1]$24, the confidence uncertainty in position is defined as the minimal Lebesgue measure of any measurable set that carries probability at least $r \in [0,1]$25:

$r \in [0,1]$26

Restricting the support to a single interval gives the interval confidence uncertainty

$r \in [0,1]$27

with analogous definitions in momentum space for $r \in [0,1]$28 and $r \in [0,1]$29 (Lin et al., 6 May 2026).

The paper proves a phase transition at the antidiagonal $r \in [0,1]$30. If $r \in [0,1]$31, then for every $r \in [0,1]$32 there exists a state such that both $r \in [0,1]$33 and $r \in [0,1]$34 simultaneously; in particular, position and momentum can be jointly localized with probability at least $r \in [0,1]$35, so no nonzero product lower bound exists. If $r \in [0,1]$36, a nontrivial lower bound appears:

$r \in [0,1]$37

For interval supports, the sharp bound is

$r \in [0,1]$38

where $r \in [0,1]$39 is the largest prolate-spheroidal eigenvalue (Lin et al., 6 May 2026).

This formulation shifts attention from variance or entropy of the full distribution to guaranteed probability mass in small regions. Within the broader encyclopedia of confidence uncertainty, it is a mathematically distinct but conceptually related usage: confidence is a required probability threshold, and uncertainty is the smallest region compatible with that threshold.