---
title: Verbalized Uncertainty in Language Models
url: https://www.emergentmind.com/topics/verbalized-uncertainty
type: topic
---

# Verbalized Uncertainty in Language Models

Verbalized uncertainty is the explicit expression of uncertainty in language rather than a quantity inferred only from hidden states, token logits, or repeated sampling. In contemporary large language model research, it appears as verbal probability phrases, percentage confidence scores, ordinal labels, interval-valued reports, structured belief annotations, and explicit reasoning-time markers; in older and adjacent literatures, it also denotes hedges, categorical judgments of probability, prosodic signals of certainty in speech, and uncertainty spans in scientific writing [2205.14334; 2603.10396; 2605.11436; 2503.11376]. Across these settings, the central problem is not merely whether a system can state confidence, but whether the stated uncertainty is calibrated, behaviorally meaningful, and matched to the decision or communicative interface for which it is used [2412.14737; 2604.05306].

## 1. Conceptual scope and core distinctions

A persistent distinction in the literature is between uncertainty as an internal or latent state and uncertainty as an outward linguistic act. One line of work separates **semantic uncertainty (SU)**, defined as intrinsic uncertainty about the semantic content or truth of an answer, from **verbal uncertainty (VU)**, defined as uncertainty conveyed by the phrasing of the answer through hedges, refusals, or expressions such as “I’m not sure” and “maybe” [2503.14477]. Another distinguishes **perceived certainty**, or how certain a speaker sounds to listeners, from **self-reported certainty**, used as a proxy for internal certainty; in a speech corpus of 600 utterances from 20 adult native English speakers, mean self-reported certainty was **2.6 ± 1.4**, mean perceived certainty was **3.5 ± 1.1**, and the correlation between the two was **0.42**, indicating that linguistic or prosodic certainty can diverge from internal state [1103.1898].

A second conceptual split concerns the order of uncertainty. Under classical probabilistic elicitation, first-order uncertainty is uncertainty over outcomes or responses and may be represented by a point-valued distribution $p(y \mid x)$ or its entropy $H(p)$. The imprecise-probability framework argues that this is often insufficient for LLMs, and introduces second-order uncertainty as uncertainty about the probability model itself, represented by intervals such as $[\underline{p}(y), \overline{p}(y)]$ or by credal sets [2603.10396]. This distinction is especially important in ambiguous question answering, in-context learning, and self-reflection, where a single confidence score may conflate ignorance with indifference [2603.10396].

A third distinction is historical and cognitive rather than computational. In work on subjective probabilities, verbal categories such as *unlikely* and *very likely* were modeled with possibility theory and fuzzy set theory as fuzzy numbers on the unit interval, with “elastic constraints” calibrated per subject via a modified Robbins-Monroe sequential testing procedure [1304.3443]. That study argued that processing uncertainty categorically by means of verbal expressions imposes less mental workload than immediate numerical processing, and reported that verbal judgments were more precise than numerical ones in some tasks [1304.3443]. This suggests that verbalized uncertainty has long been treated not only as an interface choice but also as a representational choice.

## 2. Representational forms and uncertainty interfaces

The simplest representation is an answer paired with a confidence statement. In one formulation, a model receives a prompt $X$, generates a response $Y=\mathrm{LLM}(X)$, and verbalizes confidence $C=\mathrm{UQ}(X,Y)$, with calibration defined by $\Pr(Y\text{ is correct} \mid C=c)=c$ [2412.14737]. Early work on “verbalized probability” trained GPT-3 to generate both an answer and a confidence statement such as “90% confidence” or a confidence word category, showing that these verbalized outputs could be mapped to reasonably calibrated probabilities under some distribution shift [2205.14334].

More structured interfaces have since appeared. ORCE defines confidence as a function of a fixed question–answer pair, $C(x,y)$, and explicitly separates answer generation from confidence generation so that confidence optimization does not perturb the answer model [2605.12446]. A more control-oriented formulation distinguishes a **global interface**, in which the model states a scalar confidence score for its final answer, from a **local interface**, in which it emits an explicit `<uncertain>` marker during reasoning when it reaches a high-risk state [2604.05306]. The former summarizes outcome-level reliability; the latter exposes process-level fragility during generation [2604.05306].

In long-horizon partially observable agents, verbalized uncertainty has also been used as a belief-state representation. Agent-BRACE approximates the belief state $b_t=P(s_t\mid \mathcal{H}_t)$ not with a free-form summary but with a set of atomic natural-language claims, each annotated with a **Words of Estimative Probability (WEP)** label on the ordinal scale **confirmed $\succ$ almost certain $\succ$ probable $\succ$ possible $\succ$ unlikely $\succ$ doubtful $\succ$ unknown** [2605.11436]. The claim is that summary-based beliefs collapse a posterior into a single point estimate, whereas atomic claims with certainty labels preserve epistemic state in a compact, interpretable form [2605.11436].

At the highest level of expressivity, recent work argues that LLMs should verbalize not just point confidence but higher-order uncertainty. The proposed alternatives include lower and upper probabilities, credal sets, and possibility functions, with second-order uncertainty read from interval width or from **maximum mean imprecision (MMI)** [2603.10396]. This enlarges the representational range of verbalized uncertainty from “How confident are you?” to “How precisely can you even specify your confidence?” [2603.10396].

## 3. Calibration, evaluation, and empirical behavior

Calibration remains the dominant evaluative lens. Across papers, verbalized uncertainty is measured with **Expected Calibration Error (ECE)**, **Brier score**, **mean squared error (MSE)** or **MAD**-style calibration error, **AUROC**, **Spearman correlation**, **AURC / E-AURC**, and, in some studies, **Net Calibration Error (NCE)** for directional miscalibration [2205.14334; 2412.14737; 2405.02917; 2605.12446]. The standard ideal is that if a model says “70% confidence,” then among outputs assigned 70% confidence, about 70% should be correct [2412.14737; 2205.14334].

Empirical findings are mixed. A broad benchmark over **10 datasets, 11 LLMs, and 17 prompt methods** found that the reliability of verbalized confidence strongly depends on how the model is asked, and that for large models the `combo` prompt gave an average deviation from empirical accuracy of about **7%** [2412.14737]. By contrast, a study of explanation uncertainty found that verbalized confidence for natural language explanations was almost always extremely high, with average verbalized confidence reported as **94.46%**, and concluded that verbalized uncertainty was **not a reliable estimate of explanation confidence** because it was non-discriminative and clustered near 100% [2311.03533]. In evaluation of LLMs and VLMs on sentiment, GSM8K, NER, and the Japanese Uncertain Scenes benchmark, high calibration error and overconfidence were common; the same work introduced **NCE** to capture the direction of miscalibration and reported that overconfidence dominated on math, NER, and image recognition, while binary sentiment often showed underconfidence [2405.02917].

The literature also contains positive results under targeted training. ORCE, which is explicitly decoupled and order-aware, reported improved calibration and failure prediction while largely preserving answer accuracy; on **MMLU**, ECE dropped from **0.170** to **0.025** for **Llama-3 8B** and from **0.212** to **0.034** for **Qwen3 8B** [2605.12446]. In confidence-supervised fine-tuning for chain-of-thought reasoning, **LLaMA3.2-3B-Instruct** on **GSM8K** improved from **AUROC 50.57 → 81.25**, **ECE 0.2065 → 0.0568**, and **BS 0.2549 → 0.1450**, while also increasing accuracy from **68.68 → 71.34** [2506.03723]. Agent-BRACE evaluates calibration by mapping WEP labels to nominal probabilities and computing Brier score against judge-verified ground truth; over the course of an episode, Brier score drops from **0.40 to 0.28** and confirmed claims grow from **21% to 52%**, which the paper interprets as improved calibration as evidence accumulates [2605.11436].

A recurrent empirical theme is that calibration and usefulness are not identical. One benchmark therefore supplements ECE with **informativeness**, measured by the number of distinct confidence values and the variance of the confidence distribution, and **meaningfulness**, measured by KL divergence between per-dataset and global confidence distributions [2412.14737]. Another contrasts local ranking rewards with a global Spearman-correlation reward and argues that correct ordering of confidence across responses is more robust than matching absolute numbers [2605.12446]. This suggests that verbalized uncertainty is often evaluated simultaneously as calibration, ranking, failure prediction, and control signal.

## 4. Alignment and training methodologies

Current methods span prompting, supervised fine-tuning, reinforcement or preference optimization, mechanistic intervention, and rule-based detection. Prompt-only elicitation remains common because it is cheap and does not require access to logits; extensive prompt variation has been explored over score ranges, score formulations, few-shot examples, “best guess” framing, chain-of-thought, and top-$k$-style outputs [2412.14737]. Yet prompt dependence is itself a central result: simple prompt changes help small models most, while more elaborate prompts improve large models more substantially [2412.14737].

Supervised approaches differ in the targets they use. “Teaching Models to Express Their Uncertainty in Words” constructs a confidence target from the model’s empirical sub-task accuracy $\hat p_T$ and trains GPT-3 to output either integer percentages or coarse word categories; calibration is then evaluated on the fixed zero-shot answers rather than on improved answer accuracy [2205.14334]. Confidence-supervised fine-tuning for reasoning uses self-consistency over sampled generations to derive scalar labels in $\{0,10,\dots,100\}$ and trains only the confidence span, not the reasoning trace, yet reports emergent self-verification behavior without explicit reasoning supervision or reinforcement-learning rewards [2506.03723].

Preference- and RL-based alignment methods increasingly decouple answer content from uncertainty content. ORCE uses a two-stage pipeline in which an answer LLM first generates $y$, after which a confidence LLM estimates verbalized confidence conditioned on the fixed pair $(x,y)$; multiple sampled confidence statements are ranked by a Spearman-correlation-based reward, and **Direct Preference Optimization (DPO)** increases the likelihood of the preferred confidence output relative to the rejected one [2605.12446]. SAGE and **Group-Uncertainty Preference Optimization (GUPO)** frame verbal uncertainty alignment as a distributional calibration problem: uncertainty targets should be estimated from repeated model outputs rather than from an isolated response, and training should supervise the uncertainty channel rather than the full response [2606.11512]. Their proposed **Semantic-Answer Guided Entropy** target constructs an answer-conditioned uncertainty geometry over sampled responses, aiming to preserve categorical, numeric, and symbolic answer distinctions while maintaining a smooth and scale-preserving calibration signal [2606.11512].

Other lines of work manipulate the model’s internal or architectural relation to uncertainty. The mechanistic study of VU identifies a **Verbal Uncertainty Feature (VUF)** as a single linear direction in hidden-state space, estimates it by a difference-in-means direction over high-VU and low-VU examples, and uses inference-time activation steering to increase or decrease verbal uncertainty [2503.14477]. The proposed **Mechanistic Uncertainty Calibration (MUC)** scales that intervention by the mismatch between semantic and verbal uncertainty and reports an average **relative 31.9%** reduction in confident hallucinations on short-form QA [2503.14477]. At the agent level, verbalized uncertainty can be trained jointly with policy. Agent-BRACE uses **joint PPO training** for both a belief-state model and a policy model, with a composite belief reward that includes state tracking, state correctness, diversity over the WEP vocabulary, format gating, and discounted success [2605.11436].

## 5. Domain-specific instantiations

In text-based question answering and reasoning, verbalized confidence is typically attached directly to answers and evaluated against correctness. This is the dominant setting in work on calibration, failure prediction, and selective prediction [2412.14737; 2605.12446; 2205.14334]. In natural language explanations, however, the target of confidence is not the answer but the explanation itself. There, verbalized uncertainty has been defined as the model’s own stated confidence in token-importance explanations or chain-of-thought steps, but empirical analysis found it poorly calibrated relative to explanation faithfulness, whereas perturbation-based probing uncertainty correlated better with faithfulness [2311.03533].

In long-horizon agents, verbalized uncertainty is part of state estimation rather than answer presentation. Agent-BRACE decouples a belief-state model $f_\phi$ from a policy model $\pi_\theta$ and has the policy condition on a compact belief rather than the full interaction history, yielding average absolute improvements of **+14.5%** for **Qwen2.5-3B-Instruct** and **+5.3%** for **Qwen3-4B-Instruct** over the strongest RL baseline on TextWorld tasks, while maintaining a near-constant context window independent of episode length [2605.11436]. This use of verbalized uncertainty is atypical because the uncertainty-bearing output is not primarily user-facing; it is an internal control representation encoded in natural language [2605.11436].

In speech, verbalized or perceivable uncertainty is expressed prosodically rather than textually. Phrase-centered prosodic features improve recognition of perceived certainty relative to utterance-level features alone, and the same models can help identify which phrase is the source of uncertainty [1103.1898]. In scientific writing, uncertainty becomes a span-level annotation problem. **UnScientify** defines a sentence as expressing uncertainty if it explicitly expresses absence of knowledge, insufficient knowledge, or lack of precision, and uses a weakly supervised, rule-based pipeline in **spaCy** with up to **89 patterns** across **12 Scientific Uncertainty Pattern groups**; on a **975-sentence** corpus from **312 articles**, it achieved **0.808** accuracy and outperformed the tested fine-tuned, few-shot, and zero-shot baselines overall [2503.11376].

Multimodal settings expose additional failure modes. On corrupted image data, **GPT-4V**, **Gemini Pro Vision**, and **Claude 3 Opus** remained overconfident across easy VQA, hard VQA, and counting, with increasing corruption severity worsening calibration because accuracy fell while confidence did not fall enough [2504.03440]. A broader study of VLMs found pervasive miscalibration across image, video, factuality, and mathematical reasoning tasks, but reported that vision-centric reasoning models such as **o3** and **o4-mini** were consistently better calibrated than instruction-tuned or text-centric models; it proposed **Visual Confidence-Aware Prompting (VCAP)**, a two-stage method that first elicits a visual description with confidence and then solves the task using that description, improving both accuracy and ECE on IsoBench [2505.20236]. In medical VQA, where verbalized confidence often tracks language priors rather than genuine visual evidence, a training-based framework using a **$2\times2$ factorial perturbation design** over image presence and text integrity reported calibration-error reductions of **60% or more** and discrimination improvements of **26% or more** while preserving predictive accuracy across three benchmarks and two architectures [2606.27023].

## 6. Debates, biases, and future directions

A major debate concerns whether verbalized uncertainty should be numeric, verbal, or both. A laboratory study with **200 UK participants** found that uncertain options with medium to high likelihoods were valued significantly lower when uncertainty was communicated verbally rather than numerically, and that this effect persisted even when verbal phrases were translated correctly into their associated numerical uncertainties [2502.06241]. The same paper interprets the residual effect in terms of ambiguity and framing, not merely mistranslation [2502.06241]. By contrast, cognitive work on subjective probabilities argued that verbal categories can reduce mental workload and may access a broader knowledge base than numerical formats [1304.3443]. Taken together, these findings suggest that “words or numbers” is not a superficial formatting choice.

Another debate concerns whether current systems are missing essential human pragmatics. The proposal of **anthropomimetic uncertainty** argues that most NLP work still relies disproportionately on numeric confidence outputs: in a survey of **40 relevant publications since 2022**, **61%** used numerical registers, only **16%** used fluent verbalization, and **73%** relied on prompting [2507.10587]. The same work reports that uncertainty style changes substantially across conversational roles and subject domains, indicating that verbalized uncertainty is socially and contextually sensitive in ways that are not necessarily aligned with correctness [2507.10587]. This line of argument does not reject calibration, but it broadens the target from probabilistic correctness alone to linguistic authenticity, personalization, and user-centered trust [2507.10587].

A related technical controversy is whether verbalized confidence should be trusted as a self-report at all. Several studies document overconfidence, anti-informative confidence, or misalignment between token-level and verbalized signals. In multimodal large models, this has been described as an **instinct–reflection misalignment** or **verbal-internal disconnect**, where implicit token-level support diverges from explicit verbal self-assessment; the proposed solution is a monotone confidence fusion framework that combines token confidence, verbal confidence, and cross-channel consistency, followed by order-preserving mean alignment [2604.17274]. In control-oriented LLMs, global verbalized confidence and local `<uncertain>` signaling were shown to behave differently internally: verbal confidence mainly refines how existing uncertainty is decoded, whereas reasoning-time signaling induces a broader late-layer reorganization [2604.05306]. This suggests that verbalized uncertainty is not a single mechanism but a family of interfaces with different computational roles.

Current research directions therefore emphasize task-matched communication. For final-answer trust, abstention, or selective retrieval, the literature increasingly favors calibrated global confidence [2604.05306]. For intervention during reasoning, retrieval triggering, or surfacing silent failures, local uncertainty signaling may be more appropriate [2604.05306]. For ambiguous or open-ended settings, higher-order verbalization via imprecise probabilities is proposed as a more faithful alternative to point confidence [2603.10396]. For multimodal reasoning, modality alignment and evidence-aware training have become central themes [2505.20236; 2606.27023]. Across these strands, the common claim is not that verbalized uncertainty is inherently reliable or inherently unreliable, but that its usefulness depends on representation, calibration target, training signal, and the decision interface it is meant to support [2412.14737; 2606.11512].

Source: https://www.emergentmind.com/topics/verbalized-uncertainty