Self-Verbalized Confidence in LLMs

• Self-verbalized confidence is a method where LLMs articulate their own estimated likelihood of correctness using numerical, categorical, or free-form expressions.
• Empirical studies reveal that such confidence signals often saturate at high values, show poor correlation with factual accuracy, and reflect superficial cues.
• Research focuses on improving calibration via contrastive fine-tuning, proper scoring rules, and reasoning scaffolds to align verbalized confidence with true model performance.

Self-verbalized confidence is the process by which a LLM or a multimodal large model articulates, in natural language or a structured output, its own estimation of the probability—or subjective certainty—that its response is correct. This construct is increasingly deployed across tasks such as open- and closed-domain question answering, chain-of-thought reasoning, translation, long-form generation, tabular QA, web agentic scenarios, and even vision-language inference. Despite intuitive appeal for transparency and trust calibration, research shows that self-verbalized confidence is often poorly correlated with factual correctness, saturates at high values regardless of true uncertainty, and may reflect superficial or answer-independent cues. Contemporary work seeks to understand the sources, measurement, fine-tuning strategies, limitations, and pathways to more reliable elicitation of verbalized confidence.

1. Formal Definitions, Prompting, and Measurement

Self-verbalized confidence refers to a model’s explicit statement or rating, typically in response to a prompt such as “How confident are you that your answer is correct?” or by appending a probability, percentage, categorical label, or natural language phrase to its output. Prompts may elicit:

• Numerical confidence: e.g., “state a probability from 0 to 1,” or “give your confidence as a percentage.”
• Categorical or Likert-style confidence: pre-defined bins such as “very uncertain,” “fairly certain,” etc., mapped to ordinal or numeric values.
• Free-form/natural language hedges: e.g., “I am mostly certain,” “I think it is likely that…”

A generic schema utilizes a two-stage prompt: (1) generate the answer, (2) explicitly elicit a confidence estimate (e.g., % correct). Other settings decouple answer and self-assessment per token, component, or statement in structured, translation, or long-form tasks (Marashian et al., 15 Jun 2026).

In multimodal and agentic contexts, models may be prompted to provide confidence per step or per candidate, possibly via structured JSON or tagged spans (Ou et al., 27 Oct 2025, Dang et al., 19 Apr 2026).

The core functional is to define a mapping

$$f^{\text{VC}}: \mathcal{Q},\,\mathcal{A} \mapsto [0,1]$$

where $\mathcal{Q}$ is a query (possibly with context), $\mathcal{A}$ the model’s generated answer, and the output is a confidence score or text.

2. Empirical Characterization: Overconfidence, Saturation, and Source Analysis

Multiple studies establish that naïvely elicited self-verbalized confidence is dominated by pathological behaviors:

• Ceiling saturation: Under minimal numeric or off-the-shelf categorical prompts, >90% of responses are at or above 95% confidence in 3–9B LLMs, collapsing the entire signal space (Cacioli, 24 Apr 2026).
• Answer-independence: The model’s stated confidence is nearly invariant to its generated answer; i.e., $P(C|q,a) \approx P(C|q)$ with very low Jensen-Shannon Divergence, so it fails to ground confidence in its own proposition (Seo et al., 13 Oct 2025).
• Superficial cues: Lexical influence-tracing reveals that especially in larger models, verbalized confidence arises from generic, confidence-phrasing training data rather than content-relevant support. The “content-over-confidence ratio” (ccr) metric quantifies the extent to which confidence is grounded in relevant versus generic antecedents, with values $\ll 1$ indicating a pattern-mimicking, ungrounded signal (Xia et al., 15 Jan 2026).
• Reasoning contamination effect: In instruction-tuned and distilled models, confidence output can be a function of chain-of-thought length (longer = less confident), with the relationship driven more by surface features than epistemic uncertainty (Cacioli, 24 Apr 2026).

Overconfidence is most acute when a model is epistemically uncertain: presenting a fact or claim in the prompt boosts the model’s output confidence even when its own support is weak (suggestibility bias) (Wang et al., 29 Sep 2025). In LVLMs, overconfidence persists and is magnified by converse effects: the addition of vision lowers accuracy but may, paradoxically, dampen confidence and improve calibration relative to pure-text LLMs (Ding et al., 26 Aug 2025).

3. Calibration Metrics and Psychometric Validity

Assessment of self-verbalized confidence leverages multiple metrics:

• Expected Calibration Error (ECE): Measures absolute calibration by binning predictions by confidence and reporting the average (or smoothed) absolute difference between empirical accuracy and mean confidence per bin (Seo et al., 13 Oct 2025, Li et al., 26 Aug 2025, Yoon et al., 20 May 2025).
• AUROC/Type-2 AUROC: Probability that a correct response is given a higher confidence than an incorrect one (discriminative calibration) (Xia et al., 15 Jan 2026, Cacioli, 27 Apr 2026).
• Brier Score: Quadratic penalty for miscalibrated confidence (Li et al., 26 Aug 2025, Cacioli, 27 Apr 2026).
• Resolution, net calibration error, and others: Decomposition to separate informativeness from bias.
• Psychometric validity index: For deterministic, minimal prompts, signals must escape a degeneracy pre-check: possessing three or more distinct values, not saturating a bin, not assigning high confidence to >95% of incorrect responses, etc. Failure leads to classification as “Invalid” for selective prediction or abstention purposes (Cacioli, 24 Apr 2026).
• Within-question discrimination: For multiple sampled answer paths, the model’s confidence should distinguish correct from incorrect alternatives within a single question, not just across rows (Taubenfeld et al., 10 Feb 2025).

Calibration failures are pervasive: ECEs of 0.35–0.64 are common for tabular QA (compared to 0.10–0.15 on textual QA); in small LLMs, verbalized confidence is invalid by all item-level screens (Voss, 14 Apr 2026, Cacioli, 24 Apr 2026).

4. Techniques for Calibration and Improvement

Multiple methodological innovations seek to improve the alignment of verbalized confidence with correctness:

• Contrastive and answer-dependent fine-tuning (ADVICE): Explicitly penalizes answer-independence by maximizing the Jensen–Shannon Divergence between confidence scores for correct and incorrect answers, together with a margin loss to ensure higher confidence is assigned to the correct output (Seo et al., 13 Oct 2025).
• Calibration with proper scoring rules (ConfTuner): Implements tokenized Brier loss as a discrete proper scoring rule, matching softmax over confidence tokens to actual correctness probability (Li et al., 26 Aug 2025).
• Critique-Calibration (CritiCal): Supervises LLMs with natural language critiques—externally generated or teacher-provided—explaining whether given confidences are appropriate, too high, or too low for specific reasoning steps or answers (Zong et al., 28 Oct 2025).
• Self-consistency and weighted voting (CISC): Rather than using a single pass, samples multiple chain-of-thought completions and aggregates answers using model-provided confidence scores as weights, increasing discrimination and reducing sample size (Taubenfeld et al., 10 Feb 2025).
• Distractor-normalized coherence (DiNCo): Normalizes confidence in a candidate answer by dividing by the total confidence assigned across self-generated mutually exclusive distractors, accounting for suggestibility (Wang et al., 29 Sep 2025).
• Reinforcement learning (LoVeC): Trains LLMs via reward or preference optimization to produce numerical confidence scores per generated statement that agree with oracle fact-checker signals (Zhang et al., 29 May 2025).
• Self-verification supervision: Fine-tuning only on scalar confidence labels (not explicit reasoning signals) can induce alignment between output confidence and the generation of longer, more thorough, or self-verifying reasoning paths for low-confidence items (Jang et al., 4 Jun 2025).
• Chain-of-thought (CoT) and slow thinking: Structuring prompts to require token-by-token reasoning, backtracking, and explicit “confidence reasoning” yields a monotonic improvement in calibration throughout the trace; ablations confirm that exploration and verification steps drive the benefit (Yoon et al., 20 May 2025, Podolak et al., 28 May 2025).
• Calibration in multimodal and agent settings: Instinct vs. reflection fusion mechanisms combine token-level (instinct) and verbalized (reflection) confidence, with monotonic-parameterized logistic fusion for improved reliability; cross-channel consistency is formalized and corrected by order-preserving mean alignment (Dang et al., 19 Apr 2026).

A recurring negative result is that merely prompting (e.g., “state your confidence from 0–100%”)—without calibration-aware loss, answer-grounding, or reasoning scaffolding—does not rescue reliability and may produce signals that are actively misleading (Cacioli, 24 Apr 2026, Voss, 14 Apr 2026).

5. Domains, Modalities, and Task-Specific Findings

Self-verbalized confidence is deployed in diverse tasks with modality-specific considerations:

• Tabular QA: All LLMs tested exhibit strong overconfidence and poor discrimination; ensemble and perturbation-based methods dominate (Voss, 14 Apr 2026).
• Machine translation: Per-token or per-span confidence can be elicited via numeric, Likert, or span-list prompts, rivaling internal entropy or probability-based metrics in error detection and calibration; continuous confidence (numeric) is less reliable than ordinal (Likert) scales (Marashian et al., 15 Jun 2026).
• Multi-modal/LVLMs: Verbalized confidence remains the least reliable among probabilistic, consistency, and verbal signals. Reasoning-based scaffolding (“analyze step by step,” explicit explanations, or image-by-text CoT) and post-hoc numeric scoring (Prob-Thr) partially remedy calibration but generally lag token-level and consistency-based baselines (Ding et al., 26 Aug 2025, Dang et al., 19 Apr 2026).
• Long-form factual generation: On-the-fly statement-level self-verbalized confidence tags can be RL-fine-tuned (LoVeC), boosting alignment with oracle fact-checkers across both free-form and iterative tagging modes (Zhang et al., 29 May 2025).
• Web/browsing agents: Confidence is used as an explicit stopping criterion for the agent’s answer (test-time scaling), where confidence ≥ τ indicates sufficiency. High verbalized confidence is predictive of accuracy, enabling dynamic allocation of computational budget (Ou et al., 27 Oct 2025).
• Self-assessment beyond confidence: Multidimensional strategies drawing from cognitive appraisal theory (effort, ability, self-esteem, pleasantness, etc.) reveal that model-estimated effort and ability often outperform confidence as predictors of success, yielding more robust, non-overoptimistic self-assessment across diverse domains and tasks (Bhattacharyya et al., 8 May 2026).

6. Open Challenges and Theoretical Insights

Despite recent improvements, self-verbalized confidence continues to face several structural limitations and open research questions:

• Grounding and attribution: Overconfidence is primarily a result of models grounding their outputs in generic confidence signals or superficial verbal patterns rather than content-specific evidence; scaling and fine-tuning alone do not reliably shift this tendency (Xia et al., 15 Jan 2026, Seo et al., 13 Oct 2025).
• Label entropy and training balance: Effective supervised training for verbalized confidence requires a non-degenerate target distribution with sufficient entropy; filtering for high-confidence or correct-modal only training items destroys the training signal (Cacioli, 27 Apr 2026).
• Cross-modal misalignment: In multimodal MLLMs, instinctive (token-level) and reflective (verbalized) confidence channels can diverge substantially, requiring explicit fusion and calibration (Dang et al., 19 Apr 2026).
• Task and prompt-dependence: The most informative self-assessment dimension is task-dependent; e.g., confidence serves better for retrieval or multiple-choice tasks, effort and ability for inductive/reasoning-heavy challenges (Bhattacharyya et al., 8 May 2026, Zong et al., 28 Oct 2025).
• Interpretability: Well-trained models can be prompted to produce confidence-aligned, concise, and consistent output formats, and changes in confidence labels can signal when to trigger rethinking or additional computational resource allocation (Jang et al., 4 Jun 2025, Ou et al., 27 Oct 2025).

7. Future Directions and Recommendations

Emerging research suggests several foundational and applied directions for improving the reliability of self-verbalized confidence:

• Answer-grounded supervision: Explicitly force confidence to condition on the answer using contrastive objectives and attention-based or attribution-based grounding (Seo et al., 13 Oct 2025).
• Fusion of uncertainty channels: Combine verbalized confidence with internal token-level certainty or generation geometry in monotonic, order-preserving fusion frameworks for enhanced calibration and selective-prediction performance (Dang et al., 19 Apr 2026, Martell et al., 7 May 2026).
• Richer calibration objectives: Employ proper scoring rules (tokenized Brier, focal loss) with discrete or continuous confidence tokens, and incorporate auxiliary losses for output consistency and reasoning trace quality (Li et al., 26 Aug 2025).
• Reasoning scaffolds and critique-based interventions: Couple answer and confidence elicitation with structured chain-of-thought, external or natural language critiques, and feedback loops to boost calibration and discrimination (Yoon et al., 20 May 2025, Zong et al., 28 Oct 2025).
• Selective prediction and abstention: Use confidence as a practical signal for dynamic resource allocation, answer validation, or abstention in web agents and cascaded model systems (Ou et al., 27 Oct 2025).
• Domain-specific adaptation: Track calibration and discriminative power across tasks, sizes, and domains; adopt multidimensional self-assessment for performance prediction where verbalized confidence is insufficient (Bhattacharyya et al., 8 May 2026).

Practitioners and model developers are encouraged to test psychometric validity and apply calibration-aware methods before relying on self-verbalized confidence for any downstream risk-sensitive, selective-prediction, or user-facing task. Accurate and trustworthy confidence reporting remains a critical—but unresolved—challenge at the core of LLM reliability research.