Verbal Uncertainty Estimation

• Verbal uncertainty estimation is a framework that quantifies model confidence using natural language cues and numerical indicators, vital for trustworthy AI.
• It employs prompt-based elicitation, multi-sample extraction, and mechanistic probing to derive calibrated confidence signals from language models.
• Metrics like Expected Calibration Error, AUROC, and Brier Score validate these techniques, guiding improvements in high-stakes AI deployments.

Verbal uncertainty estimation is the practice of quantifying and communicating confidence in an output—such as an answer, prediction, or explanation—through natural language, numerical statements, or linguistic hedges, rather than direct accesses to internal model probabilities or logits. This paradigm is central to applications of LLMs and vision-LLMs (VLMs) in high-stakes domains, where user trust, transparency, and the ability to detect error or ambiguity are paramount. The field encompasses methods for eliciting, calibrating, evaluating, and intervening on these verbalized signals, spanning black-box APIs, open-source LLMs, and human–machine comparative studies.

1. Core Definitions and Taxonomies

Verbal uncertainty estimation encompasses several closely related forms:

• Numerical Verbal Confidence: Direct self-reports such as “Confidence: 85%” or “90% certain.” These are typically elicitated via prompt engineering and normalized to produce a probability-like scalar $c \in [0,1]$ (Lin et al., 2022, Del et al., 19 Mar 2026).
• Linguistic Verbal Uncertainty (LVU): Use of natural language hedges and modifiers (“I think,” “probably,” “not sure”) embedded in generated responses. LVU can be quantified post-hoc by a dedicated judge model or mapped to a confidence interval via calibrated scoring (Tao et al., 29 May 2025).
• Words of Estimative Probability (WEPs): Categorical phrases aligned to probability intervals, such as “unlikely,” “possibly,” “very likely.” Empirical mapping of WEPs to numerical probabilities in both humans and LLMs reveals non-trivial calibration gaps on non-extreme terms (Tang et al., 2024).
• Imprecise and Higher-Order Probabilities: Interval or set-valued confidence statements reflecting uncertainty both about the answer and about the confidence itself (first- vs. second-order uncertainty) (Yang et al., 11 Mar 2026).

A key distinction exists between lexical (string-level) uncertainty and semantic (meaning-/answer-level) uncertainty: lexical uncertainty concerns the probability of generating exactly the observed output, while semantic uncertainty aggregates probabilities over all semantically equivalent answers (Hager et al., 18 Mar 2025).

2. Methods for Elicitation and Extraction

2.1 Prompt-Based Elicitation

Most frameworks prompt the model either for a direct confidence scalar or a categorical statement:

• Numeric Format: “How confident are you? Give a number from 0 to 100.” Result is parsed and normalized (Lin et al., 2022, Kumaran et al., 18 Mar 2026).
• Verbal Hedging: Prompts encourage free-form answers (“Please explain and indicate your certainty.”), with a downstream judge model mapping hedges to $[0,1]$ (Tao et al., 29 May 2025).
• Imprecise Probabilities: Direct requests for intervals—“What is the lowest and highest probability you assign to each answer?”—yield a credal or set-valued uncertainty (Yang et al., 11 Mar 2026).

2.2 Multi-Sample and Perturbation-Based Extraction

• Multiple Rephrasings: Submit several semantically equivalent queries to a black-box LLM, aggregate answer variability to estimate uncertainty. Consensus frequency among outputs is interpreted as the model’s confidence in the “most frequent” answer (Yang et al., 2024).
• Parallel Sampling in Reasoning: Sample multiple chain-of-thought completions, extract both self-reported confidence and self-consistency (fraction agreeing with the modal prediction). A hybrid estimator leveraging both signals achieves near-optimal AUROC with few samples (Del et al., 19 Mar 2026).
• Probing Uncertainty in Explanations: Quantifies agreement across perturbed explanations (e.g., paraphrasing, temperature-sampling), with higher instability indicating higher uncertainty in faithfulness (Tanneru et al., 2023).

2.3 Mechanistic and Representation-Level Approaches

• Activation Probing and Patching: Mechanistic dissection reveals a dedicated “confidence cache” in LLM hidden states at key post-answer positions (“PANL”), which is later verbalized at a “confidence colon” token. Linear probing at this locus explains significantly more verbal confidence variance than log-probabilities alone (Kumaran et al., 18 Mar 2026).
• Verbal Uncertainty Feature (VUF): A single linear direction in intermediate activations controls degree of verbal hedging; causal intervention on this feature modulates the language of certainty, reducing overconfident hallucinations without fine-tuning (Ji et al., 18 Mar 2025).

3. Calibration and Evaluation Metrics

The fidelity of verbal uncertainty is assessed by metrics quantifying both its calibration and effectiveness as an error signal:

• Expected Calibration Error (ECE): $$ECE=\sum_{m=1}^M \frac{|B_m|}{n}\left|\text{acc}(B_m)-\overline{conf}(B_m)\right|$$, partitioning predictions into bins by predicted confidence. Lower values indicate better calibration (Tao et al., 29 May 2025, Lin et al., 2022).
• Brier Score: $\text{Brier} = \frac{1}{n} \sum_i (p_i - y_i)^2$, integrating calibration and accuracy.
• AUROC (Area Under Receiver-Operator Characteristic Curve): Probability a randomly chosen correct answer receives higher confidence than a random incorrect one; optimal is 1.0, random is 0.5.
• Net Calibration Error (NCE): Signed analog of ECE isolating whether models are systematically over- or under-confident (Groot et al., 2024).
• Reliability Diagrams: Plots of predicted confidence versus actual accuracy (Hager et al., 18 Mar 2025).

Human alignment is probed with KL divergence or Mann–Whitney comparisons between LLM-derived and survey-based WEP–probability mappings (Tang et al., 2024).

4. Practical Findings, Calibration Results, and Limitations

4.1 Empirical Calibration and Discrimination

• Linguistic Verbal Uncertainty (LVU) consistently surpasses both numerical verbal confidence and token-prob based methods in calibration (ECE) and discrimination (AUROC), especially in reasoning-intensive settings. LVU achieves AUROC ≈ 0.75 vs. ≈ 0.68 for raw numerical prompts (Tao et al., 29 May 2025).
• Overconfidence is endemic: LLMs and VLMs systematically report high confidence, even on difficult or ambiguous queries—a phenomenon robust to prompt format and model scale (Groot et al., 2024, Podolak et al., 28 May 2025). Reported coverage for confidence intervals in VLMs is far below nominal (e.g., <25% coverage for “95% CI” intervals) (Borszukovszki et al., 4 Apr 2025).
• Reasoning-encouraged outputs: Forcing extended chain-of-thought prompts surfaces more faithful confidence estimates, allowing verbalized scores to approach the calibration effectiveness of sampling-based semantic entropy (Podolak et al., 28 May 2025).
• Estimation under Distribution Shift: Explicitly-trained or few-shot models retain moderate calibration on unseen arithmetic tasks, though the mapping degrades for multi-answer or open-ended setups (Lin et al., 2022, Hager et al., 18 Mar 2025).
• Explanation Uncertainty: LLMs self-report maximal confidence in all explanations; only perturbation-based “probing uncertainty” reliably indicates explanation faithfulness (Tanneru et al., 2023).

4.2 Decision-Theoretic Gaps

Although models can calibrate confidence to error rates, they fail to couple confidence to risk-sensitive decision-making such as abstention under high penalties—always answering, even when optimal policy is to abstain. Model policies remain invariant to error-punishment even when prompted to defer (Wang et al., 12 Jan 2026). Practical implication: verbalized uncertainty alone is insufficient for trustworthy agent behavior in high-stakes settings.

5. Modelling Verbal Categories and Human/LLM Alignment

• Possibility Theory and Fuzzy Categories: Categorical labels (e.g., “likely,” “unlikely”) are modelled as fuzzy sets over $[0,1]$, with each subject’s usage calibrated by sequential testing. Fuzzy quantifiers provide “elastic” confidence intervals (Zimmer, 2013).
• WEP–probability distributions: Both LLMs and humans associate WEPs with broad, overlapping intervals; LLMs achieve near-human alignment on extreme WEPs, but diverge by 10–15% median on mid-range terms, especially in gendered or multilingual contexts; monotonicity is sometimes achieved by output collapse (same WEP for all probabilities) (Tang et al., 2024).

6. Applications, Mitigation Techniques, and Deployment Considerations

• Reducing Hallucinations: Mechanistic interventions on the VUF (verbal uncertainty feature) can increase hedging where semantic uncertainty is high, lowering the overconfident hallucination rate by ~30% (Ji et al., 18 Mar 2025).
• Closed-Source and Black-Box LLMs: Multi-rephrasing protocols yield improved calibration (ECE reduced by 65–80%) without access to model internals, at the cost of multiple API calls per task (Yang et al., 2024).
• VLMs and Multimodal Uncertainty: Verbal confidence outputs degrade substantially under image corruption, revealing vulnerability to distribution shift. Failure to calibrate intervals or mean/SD reflects overconfidence and lack of quantification under ambiguity (Borszukovszki et al., 4 Apr 2025, Groot et al., 2024).
• Prosody in Speech: Verbal uncertainty in spoken dialogue is reflected in prosodic features—e.g. increased silence, lower loudness, pitch variation, and slowed speaking rate—corresponding both to self-reported and perceived uncertainty. Phrase-level feature extraction boosts detection accuracy in dialogue systems (Pon-Barry et al., 2011).

7. Theoretical Advances and Future Directions

• Imprecise Probabilities as a Framework: Elicitation of confidence intervals or credal sets extends verbal uncertainty to higher-order uncertainty (“uncertainty about one’s uncertainty”). Prompting for both lower and upper probabilities yields more faithful, ambiguity-sensitive estimates. Prompt design (De Finetti’s coherent betting, interval, credal, and possibility-style prompts) plays a critical role; the resulting intervals can be mapped to verbal statements for user transparency (Yang et al., 11 Mar 2026).
• Semantic Uncertainty Distillation: Fine-tuning LLMs to align their verbal confidence with true semantic probability (i.e., the probability that a semantically equivalent answer is correct) via sampling, calibration (e.g., isotonic regression), bucketing, and verbal annotation yields confidence outputs that track observed error rates better than token-based or raw prompting approaches (Hager et al., 18 Mar 2025).
• Guidelines for Deployment: For high-stakes systems, verbal uncertainty should be elicited in combination with abstention policies, calibration via post-processing or fine-tuning, and auditing of output compliance. Attention must be paid to failure under distribution shift, adversarial queries, and overly terse answers.

Verbal uncertainty estimation thus stands as a foundational component of trustworthy AI involving language or multimodal generation. Its efficacy and reliability are strongly dependent on prompt format, calibration technique, presence of reasoning traces, and post-hoc interpretive infrastructure. Ongoing challenges include closing the human–machine alignment gap on nuanced linguistic hedges, integrating uncertainty with strategic decision policies, and robustifying uncertainty estimates under out-of-distribution and adversarial conditions (Lin et al., 2022, Tao et al., 29 May 2025, Yang et al., 11 Mar 2026, Borszukovszki et al., 4 Apr 2025, Ji et al., 18 Mar 2025, Tang et al., 2024, Groot et al., 2024, Yang et al., 2024, Tanneru et al., 2023, Pon-Barry et al., 2011).