---
title: Overconfidence Phenomenon in AI
url: https://www.emergentmind.com/topics/overconfidence-phenomenon
type: topic
---

# Overconfidence Phenomenon in AI

Overconfidence denotes a systematic excess of subjective certainty over warranted accuracy, correctness, completeness, or evidential support. Across the literature summarized here, the term does not refer to a single invariant construct: it appears as self-deception about one’s own capability in evolutionary games, as miscalibration between stated confidence and realized success in human and machine judgment, as excessive verbal certainty in large language models, as too-narrow confidence intervals in numerical estimation, as structurally induced certainty in out-of-distribution and open-set recognition, and as an apparent but sometimes physically induced posterior collapse in Bayesian scientific machine learning [1602.05782] [2410.09724] [2510.26995] [2509.23784]. A unifying characterization is that some internal or external estimate of reliability is sharper, larger, or more complete-seeming than the underlying evidence justifies.

## 1. Conceptual scope and major forms

The overconfidence phenomenon is operationalized differently across research areas.

| Domain | Operationalization | Representative source |
|---|---|---|
| RLHF-aligned LLMs | Verbalized confidence exceeds response quality or correctness | [2410.09724] |
| Numerical uncertainty | Nominal confidence intervals under-cover true answers | [2510.26995] |
| Agentic systems | Predicted task-success probability exceeds realized success | [2602.06948] |
| Scholarly assistance | Assertive, complete-seeming output under incomplete or unverifiable knowledge | [2606.08036] |
| Multi-solution reasoning | Undue certainty in an incomplete solution set | [2512.01725] |
| Evolutionary behavior | Self-assessed capability exceeds real capability | [1602.05782] |

In calibration-centered work, overconfidence is a misalignment between confidence and empirical correctness or quality. The RLHF study defines calibration as alignment between model confidence and actual performance, then distinguishes token-level sharpening from a second phenomenon—**verbalized overconfidence**—in which a model explicitly states excessive certainty in natural language [2410.09724]. Fermi-style numerical estimation casts the same problem in interval form: a model that claims \(99\%\) confidence but covers the truth far less often is overconfident in the classical probabilistic sense [2510.26995]. Agentic work generalizes the target from single predictions to entire trajectories through \(P(\mathrm{IS})\), the probability that an agent built on the same model succeeds on a task given the current information state [2602.06948].

Other papers widen the concept beyond explicit calibration. GIScholarBench defines overconfidence behaviorally as the production of confident, assertive, well-formatted scholarly outputs even when the required knowledge is incomplete or unverifiable, rather than through elicited confidence scores [2606.08036]. MuSoBench relocates the phenomenon from answer correctness to **solution completeness**: a model may be locally correct yet overconfident about having exhausted the solution space [2512.01725]. In evolutionary game theory, overconfidence is not verbal certainty at all, but a state variable \(\alpha_i\) added to real capability \(\gamma_i\), producing self-perceived capability \(k_i=\gamma_i+\alpha_i\) [1602.05782].

A further complication is that “overconfidence” can be partly structural. In Bayesian PINNs, narrow posterior bands may reflect either genuine restriction by physical constraints or an artifact of posterior collapse. The paper therefore argues that standard overconfidence language is too coarse unless one distinguishes confidence justified by the physics prior from confidence unsupported by information density [2509.23784]. This suggests that overconfidence is best treated as a family of phenomena linked by excess certainty, but differentiated by the object of certainty: correctness, interval coverage, task success, completeness, or epistemic support.

## 2. Formalization and measurement

The most common formalization is bin-based calibration error. For RLHF-calibrated LLMs, Expected Calibration Error and AUC are primary metrics; in the paper’s wording, ECE bins predictions by confidence and compares accuracy within bins, while AUC measures the ability of confidence to separate correct from incorrect responses [2410.09724]. MuSoBench adapts the same logic to completeness by computing confidence-relative calibration against **Recall** rather than only Precision. Its definitions include
\[
\text{Recall}(x)=\frac{|\mathcal{Y}_i \cap \hat{\mathcal{Y}}_i|}{|\hat{\mathcal{Y}}_i|}
\]
and
\[
\text{ECE(r)}=\sum_{m=1}^{M}\frac{|B_m|}{N}\,|\text{recall}(B_m)-\text{conf}(B_m)|,
\]
thereby measuring whether confidence tracks recovery of the full solution set [2512.01725].

For interval-valued uncertainty, FermiEval distinguishes nominal confidence \(1-\alpha\) from observed coverage. The empirical estimator is
\[
\widehat{\mathrm{coverage}}=\frac{1}{N}\sum_{i=1}^N \mathbf{1}[y_i\in [L_i,U_i]],
\]
and a calibrated model should satisfy \(\widehat{\mathrm{coverage}}\approx 1-\alpha\). To combine coverage with sharpness, the benchmark uses the Winkler interval score,
\[
\mathrm{WinklerScore}=\frac{1}{N}\sum_{i=1}^{N}\Big[(U_i-L_i)+\frac{2}{\alpha}\big|y_i-\operatorname{proj}_{[L_i,U_i]}(y_i)\big|\Big],
\]
so that narrow but badly under-covering intervals are penalized [2510.26995].

Several papers use simpler bias measures. In reasoning tasks with known ground truth, overconfidence is
\[
\text{Bias}=\mathbb{E}[\widetilde p]-\mathbb{E}[Y],
\]
where \(\widetilde p\) is stated probability of being correct and \(Y\in\{0,1\}\) is actual correctness [2505.02151]. Agentic overconfidence uses the same directional logic at task level: mean predicted success probability minus the true base success rate for that condition [2602.06948]. Human confidence learning work also treats miscalibration as the gap between ex ante subjective confidence and ex post success frequency, though its main emphasis is on dynamics rather than a single summary index [1707.02748].

Judge systems motivate a deployment-oriented metric. The LLM-as-a-Judge paper introduces **TH-Score**,
\[
\text{TH-Score}=(e^{(\text{accuracy}-0.5)}-1)\times \text{percentage},
\]
computed over specified high- or low-confidence intervals to emphasize the parts of the confidence distribution that matter for automatic acceptance or review routing [2508.06225]. This differs from ECE, ACE, Brier score, MCE, and NLL by explicitly combining interval accuracy with interval coverage for risk-aware selective automation.

Not all work accepts calibration metrics as sufficient. Bayesian PINNs argue that reliability diagrams, prediction-interval coverage, PIT histograms, and related tools conflate data-driven and physics-driven posterior contraction. Their alternative diagnostics are **information density**
\[
I(x)\equiv \sum_i \left\|\nabla_{\hat u}\chi^i(\hat u_\theta(x))\right\|^2
\]
and global **physics-constraint coupling**
\[
\mathrm{PCC}_\Omega=\frac{\int_\Omega I(x)\,dx}{\int_\Omega \sigma^2(x)\,dx},
\]
which are intended to distinguish justified narrowing caused by strong physical constraints from artifactual overconfidence [2509.23784].

## 3. Generative mechanisms

The surveyed work identifies multiple, partly incompatible mechanisms.

One class is **biased evidence aggregation**. In human confidence learning, uncertainty about one’s own ability induces contrarian illusory signals generated by doubt. For likely success, doubt makes failure salient and depresses confidence; for likely failure, doubt makes success salient and inflates confidence. The model writes these as
\[
q_H=\mu E p_H,\qquad
q_L=\mu E p_L+(1-\mu)\cdot 1,
\]
and uses this mechanism to account jointly for the hard–easy effect, Dunning–Kruger-type ability gradients, conservative learning, and overprecision [1707.02748].

A second class is **objective-induced optimization bias**. In RLHF, reward models used during PPO are shown to prefer higher-confidence versions of the same response, including rejected ones, so confidence acts as a spurious reward feature. This makes the policy optimize toward confident-sounding language even when quality does not warrant it [2410.09724]. In sequential recommendation, negative sampling similarly distorts the effective class prior, yielding the BCE equilibrium
\[
\sigma(s_i)=\frac{P(i)}{\alpha-\alpha P(i)+P(i)},
\]
which pushes predicted probabilities for top items toward one and produces overconfidence in SASRec-style training [2308.07192].

A third class is **search or inference collapse**. MuSoBench proposes the **cognitive-rigidity hypothesis**, according to which overconfidence arises when the reasoning process prematurely converges on a narrow set of thought paths, producing high certainty in a partial solution set [2512.01725]. FermiEval proposes a related but distributional account, the **perception-tunnel theory**: on each query, the model behaves as if it reasons over a truncated portion \(F_{I,I+\beta}\) of its inferred uncertainty distribution rather than the full \(F\), leading to tail neglect and overly narrow intervals [2510.26995].

A fourth class is **post hoc confirmation and anchoring**. Agentic overconfidence in coding agents is attributed partly to confirmation bias in post-execution review: once a coherent-looking patch exists, reviewers mistake plausibility for correctness. This helps explain the paper’s counterintuitive result that pre-execution assessment can discriminate success from failure better than vanilla post-execution review [2602.06948].

A fifth class is **architectural geometry**. In OOD detection, prior theory for piecewise-linear ReLU networks implies that as certain inputs are scaled away from the training distribution, the softmax can converge toward a one-hot vector. The CEA paper treats overconfidence here as intrinsic to extrapolation geometry and proves that asymptotic softmax overconfidence with a linear final layer implies divergence in at least one penultimate activation coordinate [2405.12658]. Open-set recognition work traces a related phenomenon to feature-space overlap between unknown classes and known-class manifolds, producing unjustifiably high confidence on semantically similar unknowns [2511.13775].

Finally, some work argues that apparent overconfidence may be **adaptive or induced by constraints** rather than purely erroneous. In Bayesian PINNs, hard or soft physics constraints can legitimately collapse posterior variance. In organizational theory, overconfidence can be selected because it amplifies responsiveness to private information when conflicts of interest vary across states; a well-calibrated employee is optimal iff \(\mathbb{E}_f[c(\tilde\theta)\mid s]\) is constant [2601.05206].

## 4. Empirical manifestations in contemporary AI

Large language models exhibit the phenomenon across multiple levels of abstraction. After RLHF, confidence distributions shift upward, and models verbally express excessive certainty even when wrong. On Llama3-8B, GSM8K, CoT prompting, vanilla PPO achieves ECE \(0.2566\), while PPO-M reduces it to \(0.1909\) and raises accuracy from \(0.7392\) to \(0.7703\); the paper frames this as a 6.44-point ECE reduction and a 2.73-point accuracy gain relative to standard PPO [2410.09724]. Mechanistic work later localized inflated verbalized confidence to a compact set of middle-to-late-layer MLPs and attention heads, with targeted steering cutting PopQA ECE from \(0.492\) to \(0.111\) on Qwen2.5-3B and from \(0.570\) to as low as \(0.018\) on Llama-3.2-3B, depending on the intervention [2604.01457].

Numerical uncertainty remains sharply miscalibrated. FermiEval reports that across several modern models, nominal \(99\%\) intervals cover the truth only about \(65\%\) of the time on average, with conformal adjustment restoring accurate \(99\%\) observed coverage and decreasing the Winkler score by \(54\%\) [2510.26995]. On reasoning problems with known ground truths, all five studied LLMs overestimate the probability that their answer is correct by between 20% and 60%; GPT-4o, for example, has fraction correct \(0.63\), mean confidence \(0.94\), and bias \(0.30\), whereas humans on the same paradigm are only about 4 percentage points overconfident on average [2505.02151].

Task-level and interactive settings intensify the problem. In coding-agent evaluation, some agents that succeed only 22% of the time predict 77% success, and post-execution review can be worse calibrated than pre-execution assessment. GPT-5.2-Codex post-execution reviewers, for instance, predict 73% success against a true rate of 35% [2602.06948]. In multi-turn debate, the zero-sum structure makes overconfidence logically transparent: initial confidence averages \(72.9\%\) against a rational \(50\%\) baseline, rises to \(83\%\) by the final round, and in 61.7% of cross-model debates both sides simultaneously claim at least \(75\%\) probability of victory [2505.19184].

Broader AI evaluation pipelines show analogous failures. GIScholarBench finds consistent overconfidence across metadata retrieval, literature linking, and research-direction generation: models continue producing definitive titles and DOIs when wrong, extend citation lists beyond reliable retrieval capacity, and generate future research agendas with lower topic coverage and higher novel miss rates than real future-citing papers [2606.08036]. In LLM-as-a-Judge systems, the concern is not only wrong verdicts but inflated confidence on those verdicts, motivating a shift from accuracy-centric to confidence-driven, risk-aware judge design [2508.06225].

Beyond LLMs, the same pattern appears in uncertainty-sensitive ML. In OOD detection, AUC can degrade as inputs are pushed farther from the in-distribution; for eICU with ResNet plus MSP, AUC falls from 47.9 at scaling factor 10 to 13.2 at 1000, a reversal of the intended novelty ranking [2405.12658]. Open-set recognition similarly documents high-confidence known-class assignments on semantically similar unknowns, while active learning shows that biased small-batch updates induce overconfident false predictions that in turn corrupt uncertainty-based selection [2511.13775] [2308.10571].

## 5. Human, social, evolutionary, and organizational dimensions

In human cognition, overconfidence is dynamic rather than static. The double-or-quits experiment shows that participants are underconfident on an easy training task but overconfident on harder future tasks, and that they learn to be overconfident faster than they learn their true ability [1707.02748]. In a distinct perceptual setting, microgravity increases subjective confidence without corresponding performance gains, especially under greater stimulus uncertainty; the paper interprets this as a confidence–performance dissociation consistent with confidence inflation rather than a full demonstration of metacognitive miscalibration [2304.12133].

Evolutionary models treat overconfidence as socially embedded self-deception. In the resource competition game, each player has real capability \(\gamma_i\), overconfidence intensity \(\alpha_i\), and bluffing intensity \(\beta_i\), with self-perceived capability \(k_i=\gamma_i+\alpha_i\) and displayed capability \(m_i=\gamma_i+\beta_i\). Across structured populations, bluffing generally evolves to higher levels than overconfidence, but both persist because they can increase access to contested resources, especially when true ability is imperfectly observed [1602.05782]. Network topology matters: heterogeneity facilitates bluffing while improving the efficiency of adequate punishment against overconfident behavior [1602.05782].

A related evolutionary game on spatial prisoner’s dilemma dynamics shifts the object of bias from capability to payoff comparison. Overconfident players overestimate their own payoff as \(\Pi_x'=\Pi_x(1+\alpha)\), making them less likely to imitate neighbors. This suppresses both cooperative and defective invasions, but defective spread is hindered more strongly, reinforcing spatial reciprocity and helping maintain a “reciprocity-based cooperative phalanx” [1809.01626]. Here overconfidence is not epistemically beneficial, but it has a collective side effect favorable to cooperation under nonlinear imitation dynamics.

Organizational theory introduces a different normative angle. In delegated decision-making under private information and conflict of interest, the organization may prefer a miscalibrated employee because confidence shapes how strongly signals are acted on. The paper’s central result is that an employee is optimally well-calibrated iff the expected conflict \(\mathbb{E}_f[c(\tilde\theta)\mid s]\) is constant across signals; when stronger responsiveness is desired, overconfidence becomes optimal [2601.05206]. This does not reclassify overconfidence as accurate. Rather, it treats it as a potentially useful distortion in environments where action sensitivity matters more than unbiased belief formation.

## 6. Mitigation strategies and unresolved questions

The most direct interventions modify the confidence-generating process itself. In RLHF, **PPO-M** calibrates the reward model so that high confidence is rewarded on chosen responses and low confidence on rejected responses, while **PPO-C** adjusts the online reward by comparing reward-free quality to a running threshold. Both reduce ECE while preserving performance, and the same logic extends to DPO through **CDPO** [2410.09724]. Mechanistic work achieves inference-time recalibration by ablating or steering a small set of confidence-writing components rather than retraining the full model [2604.01457].

Post hoc statistical calibration is also effective. FermiEval finds split conformal prediction to be the strongest interval-repair method, restoring near-target coverage and sharply improving Winkler score, while direct log-probability elicitation provides a weaker but lightweight baseline when calibration data are unavailable [2510.26995]. In agentic systems, reframing evaluation as bug-finding rather than correctness confirmation reduces ECE and overconfidence across all studied coding-agent families, and conservative ensembling between pre- and post-execution estimates further improves calibration in some cases [2602.06948]. For judges, the proposed remedy is **LLM-as-a-Fuser**, coupled with TH-Score and a confidence-driven pipeline, to make automated evaluation adaptive rather than purely accuracy maximizing [2508.06225].

Outside LLMs, several methods attack structural sources of overconfidence. **CEA** augments any OOD novelty score with a penalty derived from extreme penultimate activations, often converting catastrophically poor AUC into strong detection performance in ReLU-based models [2405.12658]. In sequential recommendation, **gBCE** and **gSASRec** correct negative-sampling-induced probability inflation while improving top-rank recommendation quality [2308.07192]. In active learning, **CMaM** expands the effective training distribution through dual-space mixing, while **RankedMS** alters acquisition to avoid selection driven by distorted confidence scores [2308.10571]. Open-set recognition uses perturbation-based uncertainty estimation plus a two-stage detector to mitigate high-confidence misclassification of semantically similar unknowns [2511.13775]. For multi-solution reasoning, Long-CoT, reflection, explicit exploration prompts, and voting-based self-consistency all improve recall and reduce reasoning overconfidence, whereas higher decoding temperature alone does not [2512.01725].

Several open problems recur. One is the relation between **verbalized** confidence and deeper latent uncertainty: RLHF, mechanistic LLM work, and GIS-style behavioral benchmarks all indicate that polished certainty can decouple from correctness, but the exact internal interfaces remain only partly understood [2410.09724] [2604.01457] [2606.08036]. Another is whether apparent overconfidence is always a bug: Bayesian PINNs argue that some narrow uncertainty is physically warranted, not pathological [2509.23784]. A third is coverage beyond scalar correctness. Multi-solution tasks and scholarly ideation show that future work must evaluate not only whether answers are right, but whether search is complete, citation neighborhoods are responsibly bounded, and ideation adequately covers the relevant semantic space [2512.01725] [2606.08036].

Taken together, these results support a broad but technically precise conclusion: overconfidence is best understood as an excess-certainty failure mode whose observable form depends on what the system is claiming—correctness, interval coverage, task success, out-of-distribution familiarity, citation support, or solution completeness. Its causes range from doubt-induced belief distortion to optimization bias, architectural geometry, confirmation bias, premature search convergence, and strategic or stylistic answer formation. Its mitigation therefore requires interventions at multiple levels: objectives, reward models, uncertainty estimators, search procedures, mechanistic circuits, and evaluation protocols.

Source: https://www.emergentmind.com/topics/overconfidence-phenomenon