Verbalized Uncertainty in Language Models

• Verbalized uncertainty is the explicit expression of uncertainty via language, using hedges, numerical percentages, and structured markers.
• It distinguishes between internal latent uncertainty and its outward linguistic representation, which affects calibration and decision-making.
• It applies across diverse domains—from LLM reasoning to speech and scientific writing—using varied evaluation metrics like ECE and Brier score.

Verbalized uncertainty is the explicit expression of uncertainty in language rather than a quantity inferred only from hidden states, token logits, or repeated sampling. In contemporary LLM research, it appears as verbal probability phrases, percentage confidence scores, ordinal labels, interval-valued reports, structured belief annotations, and explicit reasoning-time markers; in older and adjacent literatures, it also denotes hedges, categorical judgments of probability, prosodic signals of certainty in speech, and uncertainty spans in scientific writing (Lin et al., 2022, Yang et al., 11 Mar 2026, Singh et al., 12 May 2026, Ningrum et al., 14 Mar 2025). Across these settings, the central problem is not merely whether a system can state confidence, but whether the stated uncertainty is calibrated, behaviorally meaningful, and matched to the decision or communicative interface for which it is used (Yang et al., 2024, Guo et al., 7 Apr 2026).

1. Conceptual scope and core distinctions

A persistent distinction in the literature is between uncertainty as an internal or latent state and uncertainty as an outward linguistic act. One line of work separates semantic uncertainty (SU), defined as intrinsic uncertainty about the semantic content or truth of an answer, from verbal uncertainty (VU), defined as uncertainty conveyed by the phrasing of the answer through hedges, refusals, or expressions such as “I’m not sure” and “maybe” (Ji et al., 18 Mar 2025). Another distinguishes perceived certainty, or how certain a speaker sounds to listeners, from self-reported certainty, used as a proxy for internal certainty; in a speech corpus of 600 utterances from 20 adult native English speakers, mean self-reported certainty was 2.6 ± 1.4, mean perceived certainty was 3.5 ± 1.1, and the correlation between the two was 0.42, indicating that linguistic or prosodic certainty can diverge from internal state (Pon-Barry et al., 2011).

A second conceptual split concerns the order of uncertainty. Under classical probabilistic elicitation, first-order uncertainty is uncertainty over outcomes or responses and may be represented by a point-valued distribution $p(y \mid x)$ or its entropy $H(p)$. The imprecise-probability framework argues that this is often insufficient for LLMs, and introduces second-order uncertainty as uncertainty about the probability model itself, represented by intervals such as $[\underline{p}(y), \overline{p}(y)]$ or by credal sets (Yang et al., 11 Mar 2026). This distinction is especially important in ambiguous question answering, in-context learning, and self-reflection, where a single confidence score may conflate ignorance with indifference (Yang et al., 11 Mar 2026).

A third distinction is historical and cognitive rather than computational. In work on subjective probabilities, verbal categories such as unlikely and very likely were modeled with possibility theory and fuzzy set theory as fuzzy numbers on the unit interval, with “elastic constraints” calibrated per subject via a modified Robbins-Monroe sequential testing procedure (Zimmer, 2013). That study argued that processing uncertainty categorically by means of verbal expressions imposes less mental workload than immediate numerical processing, and reported that verbal judgments were more precise than numerical ones in some tasks (Zimmer, 2013). This suggests that verbalized uncertainty has long been treated not only as an interface choice but also as a representational choice.

2. Representational forms and uncertainty interfaces

The simplest representation is an answer paired with a confidence statement. In one formulation, a model receives a prompt $X$, generates a response $Y=\mathrm{LLM}(X)$, and verbalizes confidence $C=\mathrm{UQ}(X,Y)$, with calibration defined by $\Pr(Y\text{ is correct} \mid C=c)=c$ (Yang et al., 2024). Early work on “verbalized probability” trained GPT-3 to generate both an answer and a confidence statement such as “90% confidence” or a confidence word category, showing that these verbalized outputs could be mapped to reasonably calibrated probabilities under some distribution shift (Lin et al., 2022).

More structured interfaces have since appeared. ORCE defines confidence as a function of a fixed question–answer pair, $C(x,y)$, and explicitly separates answer generation from confidence generation so that confidence optimization does not perturb the answer model (Li et al., 12 May 2026). A more control-oriented formulation distinguishes a global interface, in which the model states a scalar confidence score for its final answer, from a local interface, in which it emits an explicit <uncertain> marker during reasoning when it reaches a high-risk state (Guo et al., 7 Apr 2026). The former summarizes outcome-level reliability; the latter exposes process-level fragility during generation (Guo et al., 7 Apr 2026).

In long-horizon partially observable agents, verbalized uncertainty has also been used as a belief-state representation. Agent-BRACE approximates the belief state $b_t=P(s_t\mid \mathcal{H}_t)$ not with a free-form summary but with a set of atomic natural-language claims, each annotated with a Words of Estimative Probability (WEP) label on the ordinal scale confirmed $\succ$ almost certain $H(p)$0 probable $H(p)$1 possible $H(p)$2 unlikely $H(p)$3 doubtful $H(p)$4 unknown (Singh et al., 12 May 2026). The claim is that summary-based beliefs collapse a posterior into a single point estimate, whereas atomic claims with certainty labels preserve epistemic state in a compact, interpretable form (Singh et al., 12 May 2026).

At the highest level of expressivity, recent work argues that LLMs should verbalize not just point confidence but higher-order uncertainty. The proposed alternatives include lower and upper probabilities, credal sets, and possibility functions, with second-order uncertainty read from interval width or from maximum mean imprecision (MMI) (Yang et al., 11 Mar 2026). This enlarges the representational range of verbalized uncertainty from “How confident are you?” to “How precisely can you even specify your confidence?” (Yang et al., 11 Mar 2026).

3. Calibration, evaluation, and empirical behavior

Calibration remains the dominant evaluative lens. Across papers, verbalized uncertainty is measured with Expected Calibration Error (ECE), Brier score, mean squared error (MSE) or MAD-style calibration error, AUROC, Spearman correlation, AURC / E-AURC, and, in some studies, Net Calibration Error (NCE) for directional miscalibration (Lin et al., 2022, Yang et al., 2024, Groot et al., 2024, Li et al., 12 May 2026). The standard ideal is that if a model says “70% confidence,” then among outputs assigned 70% confidence, about 70% should be correct (Yang et al., 2024, Lin et al., 2022).

Empirical findings are mixed. A broad benchmark over 10 datasets, 11 LLMs, and 17 prompt methods found that the reliability of verbalized confidence strongly depends on how the model is asked, and that for large models the combo prompt gave an average deviation from empirical accuracy of about 7% (Yang et al., 2024). By contrast, a study of explanation uncertainty found that verbalized confidence for natural language explanations was almost always extremely high, with average verbalized confidence reported as 94.46%, and concluded that verbalized uncertainty was not a reliable estimate of explanation confidence because it was non-discriminative and clustered near 100% (Tanneru et al., 2023). In evaluation of LLMs and VLMs on sentiment, GSM8K, NER, and the Japanese Uncertain Scenes benchmark, high calibration error and overconfidence were common; the same work introduced NCE to capture the direction of miscalibration and reported that overconfidence dominated on math, NER, and image recognition, while binary sentiment often showed underconfidence (Groot et al., 2024).

The literature also contains positive results under targeted training. ORCE, which is explicitly decoupled and order-aware, reported improved calibration and failure prediction while largely preserving answer accuracy; on MMLU, ECE dropped from 0.170 to 0.025 for Llama-3 8B and from 0.212 to 0.034 for Qwen3 8B (Li et al., 12 May 2026). In confidence-supervised fine-tuning for chain-of-thought reasoning, LLaMA3.2-3B-Instruct on GSM8K improved from AUROC 50.57 → 81.25, ECE 0.2065 → 0.0568, and BS 0.2549 → 0.1450, while also increasing accuracy from 68.68 → 71.34 (Jang et al., 4 Jun 2025). Agent-BRACE evaluates calibration by mapping WEP labels to nominal probabilities and computing Brier score against judge-verified ground truth; over the course of an episode, Brier score drops from 0.40 to 0.28 and confirmed claims grow from 21% to 52%, which the paper interprets as improved calibration as evidence accumulates (Singh et al., 12 May 2026).

A recurrent empirical theme is that calibration and usefulness are not identical. One benchmark therefore supplements ECE with informativeness, measured by the number of distinct confidence values and the variance of the confidence distribution, and meaningfulness, measured by KL divergence between per-dataset and global confidence distributions (Yang et al., 2024). Another contrasts local ranking rewards with a global Spearman-correlation reward and argues that correct ordering of confidence across responses is more robust than matching absolute numbers (Li et al., 12 May 2026). This suggests that verbalized uncertainty is often evaluated simultaneously as calibration, ranking, failure prediction, and control signal.

4. Alignment and training methodologies

Current methods span prompting, supervised fine-tuning, reinforcement or preference optimization, mechanistic intervention, and rule-based detection. Prompt-only elicitation remains common because it is cheap and does not require access to logits; extensive prompt variation has been explored over score ranges, score formulations, few-shot examples, “best guess” framing, chain-of-thought, and top-$H(p)$5-style outputs (Yang et al., 2024). Yet prompt dependence is itself a central result: simple prompt changes help small models most, while more elaborate prompts improve large models more substantially (Yang et al., 2024).

Supervised approaches differ in the targets they use. “Teaching Models to Express Their Uncertainty in Words” constructs a confidence target from the model’s empirical sub-task accuracy $H(p)$6 and trains GPT-3 to output either integer percentages or coarse word categories; calibration is then evaluated on the fixed zero-shot answers rather than on improved answer accuracy (Lin et al., 2022). Confidence-supervised fine-tuning for reasoning uses self-consistency over sampled generations to derive scalar labels in $H(p)$7 and trains only the confidence span, not the reasoning trace, yet reports emergent self-verification behavior without explicit reasoning supervision or reinforcement-learning rewards (Jang et al., 4 Jun 2025).

Preference- and RL-based alignment methods increasingly decouple answer content from uncertainty content. ORCE uses a two-stage pipeline in which an answer LLM first generates $H(p)$8, after which a confidence LLM estimates verbalized confidence conditioned on the fixed pair $H(p)$9; multiple sampled confidence statements are ranked by a Spearman-correlation-based reward, and Direct Preference Optimization (DPO) increases the likelihood of the preferred confidence output relative to the rejected one (Li et al., 12 May 2026). SAGE and Group-Uncertainty Preference Optimization (GUPO) frame verbal uncertainty alignment as a distributional calibration problem: uncertainty targets should be estimated from repeated model outputs rather than from an isolated response, and training should supervise the uncertainty channel rather than the full response (Shi et al., 9 Jun 2026). Their proposed Semantic-Answer Guided Entropy target constructs an answer-conditioned uncertainty geometry over sampled responses, aiming to preserve categorical, numeric, and symbolic answer distinctions while maintaining a smooth and scale-preserving calibration signal (Shi et al., 9 Jun 2026).

Other lines of work manipulate the model’s internal or architectural relation to uncertainty. The mechanistic study of VU identifies a Verbal Uncertainty Feature (VUF) as a single linear direction in hidden-state space, estimates it by a difference-in-means direction over high-VU and low-VU examples, and uses inference-time activation steering to increase or decrease verbal uncertainty (Ji et al., 18 Mar 2025). The proposed Mechanistic Uncertainty Calibration (MUC) scales that intervention by the mismatch between semantic and verbal uncertainty and reports an average relative 31.9% reduction in confident hallucinations on short-form QA (Ji et al., 18 Mar 2025). At the agent level, verbalized uncertainty can be trained jointly with policy. Agent-BRACE uses joint PPO training for both a belief-state model and a policy model, with a composite belief reward that includes state tracking, state correctness, diversity over the WEP vocabulary, format gating, and discounted success (Singh et al., 12 May 2026).

5. Domain-specific instantiations

In text-based question answering and reasoning, verbalized confidence is typically attached directly to answers and evaluated against correctness. This is the dominant setting in work on calibration, failure prediction, and selective prediction (Yang et al., 2024, Li et al., 12 May 2026, Lin et al., 2022). In natural language explanations, however, the target of confidence is not the answer but the explanation itself. There, verbalized uncertainty has been defined as the model’s own stated confidence in token-importance explanations or chain-of-thought steps, but empirical analysis found it poorly calibrated relative to explanation faithfulness, whereas perturbation-based probing uncertainty correlated better with faithfulness (Tanneru et al., 2023).

In long-horizon agents, verbalized uncertainty is part of state estimation rather than answer presentation. Agent-BRACE decouples a belief-state model $[\underline{p}(y), \overline{p}(y)]$0 from a policy model $[\underline{p}(y), \overline{p}(y)]$1 and has the policy condition on a compact belief rather than the full interaction history, yielding average absolute improvements of +14.5% for Qwen2.5-3B-Instruct and +5.3% for Qwen3-4B-Instruct over the strongest RL baseline on TextWorld tasks, while maintaining a near-constant context window independent of episode length (Singh et al., 12 May 2026). This use of verbalized uncertainty is atypical because the uncertainty-bearing output is not primarily user-facing; it is an internal control representation encoded in natural language (Singh et al., 12 May 2026).

In speech, verbalized or perceivable uncertainty is expressed prosodically rather than textually. Phrase-centered prosodic features improve recognition of perceived certainty relative to utterance-level features alone, and the same models can help identify which phrase is the source of uncertainty (Pon-Barry et al., 2011). In scientific writing, uncertainty becomes a span-level annotation problem. UnScientify defines a sentence as expressing uncertainty if it explicitly expresses absence of knowledge, insufficient knowledge, or lack of precision, and uses a weakly supervised, rule-based pipeline in spaCy with up to 89 patterns across 12 Scientific Uncertainty Pattern groups; on a 975-sentence corpus from 312 articles, it achieved 0.808 accuracy and outperformed the tested fine-tuned, few-shot, and zero-shot baselines overall (Ningrum et al., 14 Mar 2025).

Multimodal settings expose additional failure modes. On corrupted image data, GPT-4V, Gemini Pro Vision, and Claude 3 Opus remained overconfident across easy VQA, hard VQA, and counting, with increasing corruption severity worsening calibration because accuracy fell while confidence did not fall enough (Borszukovszki et al., 4 Apr 2025). A broader study of VLMs found pervasive miscalibration across image, video, factuality, and mathematical reasoning tasks, but reported that vision-centric reasoning models such as o3 and o4-mini were consistently better calibrated than instruction-tuned or text-centric models; it proposed Visual Confidence-Aware Prompting (VCAP), a two-stage method that first elicits a visual description with confidence and then solves the task using that description, improving both accuracy and ECE on IsoBench (Xuan et al., 26 May 2025). In medical VQA, where verbalized confidence often tracks language priors rather than genuine visual evidence, a training-based framework using a $[\underline{p}(y), \overline{p}(y)]$2 factorial perturbation design over image presence and text integrity reported calibration-error reductions of 60% or more and discrimination improvements of 26% or more while preserving predictive accuracy across three benchmarks and two architectures (Senoglu et al., 25 Jun 2026).

6. Debates, biases, and future directions

A major debate concerns whether verbalized uncertainty should be numeric, verbal, or both. A laboratory study with 200 UK participants found that uncertain options with medium to high likelihoods were valued significantly lower when uncertainty was communicated verbally rather than numerically, and that this effect persisted even when verbal phrases were translated correctly into their associated numerical uncertainties (Bodenberger et al., 10 Feb 2025). The same paper interprets the residual effect in terms of ambiguity and framing, not merely mistranslation (Bodenberger et al., 10 Feb 2025). By contrast, cognitive work on subjective probabilities argued that verbal categories can reduce mental workload and may access a broader knowledge base than numerical formats (Zimmer, 2013). Taken together, these findings suggest that “words or numbers” is not a superficial formatting choice.

Another debate concerns whether current systems are missing essential human pragmatics. The proposal of anthropomimetic uncertainty argues that most NLP work still relies disproportionately on numeric confidence outputs: in a survey of 40 relevant publications since 2022, 61% used numerical registers, only 16% used fluent verbalization, and 73% relied on prompting (Ulmer et al., 11 Jul 2025). The same work reports that uncertainty style changes substantially across conversational roles and subject domains, indicating that verbalized uncertainty is socially and contextually sensitive in ways that are not necessarily aligned with correctness (Ulmer et al., 11 Jul 2025). This line of argument does not reject calibration, but it broadens the target from probabilistic correctness alone to linguistic authenticity, personalization, and user-centered trust (Ulmer et al., 11 Jul 2025).

A related technical controversy is whether verbalized confidence should be trusted as a self-report at all. Several studies document overconfidence, anti-informative confidence, or misalignment between token-level and verbalized signals. In multimodal large models, this has been described as an instinct–reflection misalignment or verbal-internal disconnect, where implicit token-level support diverges from explicit verbal self-assessment; the proposed solution is a monotone confidence fusion framework that combines token confidence, verbal confidence, and cross-channel consistency, followed by order-preserving mean alignment (Dang et al., 19 Apr 2026). In control-oriented LLMs, global verbalized confidence and local <uncertain> signaling were shown to behave differently internally: verbal confidence mainly refines how existing uncertainty is decoded, whereas reasoning-time signaling induces a broader late-layer reorganization (Guo et al., 7 Apr 2026). This suggests that verbalized uncertainty is not a single mechanism but a family of interfaces with different computational roles.

Current research directions therefore emphasize task-matched communication. For final-answer trust, abstention, or selective retrieval, the literature increasingly favors calibrated global confidence (Guo et al., 7 Apr 2026). For intervention during reasoning, retrieval triggering, or surfacing silent failures, local uncertainty signaling may be more appropriate (Guo et al., 7 Apr 2026). For ambiguous or open-ended settings, higher-order verbalization via imprecise probabilities is proposed as a more faithful alternative to point confidence (Yang et al., 11 Mar 2026). For multimodal reasoning, modality alignment and evidence-aware training have become central themes (Xuan et al., 26 May 2025, Senoglu et al., 25 Jun 2026). Across these strands, the common claim is not that verbalized uncertainty is inherently reliable or inherently unreliable, but that its usefulness depends on representation, calibration target, training signal, and the decision interface it is meant to support (Yang et al., 2024, Shi et al., 9 Jun 2026).