Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?

Abstract: LLMs' linguistically expressed confidence should faithfully reflect their intrinsic uncertainty. While recent work shows LLMs struggle to use epistemic markers (e.g., "it is likely...") in a human-aligned fashion, it remains unclear whether models can apply their own linguistic confidence framework to associate markers with specific confidence levels in a stable and generalizable way, and how contextual features impact this ability. We conduct the first systematic study of this question, formalizing marker internal confidence (MIC) as the estimated intrinsic confidence a model associates with a specific epistemic marker in a given task domain. We present 7 metrics to evaluate the stability of MICs within and across distributions. Applying our analysis framework to diverse models and tasks, we find that LLMs remain faithfully miscalibrated even under model-centric interpretation of marker meanings, struggling to differentiate markers by internal confidence across distributions despite preserving a somewhat consistent ranking order across tasks. This supplies critical, complementary evidence to existing work toward a holistic understanding of faithful calibration in LLMs, emphasizing the need for more aligned and stable marker use to improve trustworthiness and reliability.

• The paper reveals that LLMs miscalibrate epistemic markers, with MIC values failing to consistently reflect intrinsic confidence.
• It applies seven quantitative metrics (e.g., ID-MAE and CD-MAE) to assess marker stability, showing weak discriminability and context sensitivity.
• Findings indicate that while larger models slightly improve MIC stability, LLMs still struggle to align hedging with actual uncertainty, affecting trust.

Epistemic Markers and Internal Confidence: Evaluating the Faithfulness of LLMs’ Linguistic Uncertainty

Problem Formulation and Methodology

This work provides the first systematic study of whether LLMs can reliably map their own linguistic expressions of uncertainty—specifically, epistemic markers such as "I think" or "it is likely"—to their intrinsic, model-internal confidence. Rather than focusing on external assessments of factual correctness, the study operationalizes marker internal confidence (MIC) as the average internal confidence a model associates with a particular epistemic marker in a specific task domain.

The intrinsic confidence for each sentence using a given marker is estimated via sampling consistency: the model responds to a query multiple times, and a secondary LLM judge determines whether each sampled response contradicts the sentence. MIC for a marker thus quantifies the mean intrinsic confidence when that marker is used.

A comprehensive suite of seven metrics is developed to evaluate the stability, consistency, and discriminability of MICs. These metrics encompass in-domain (ID-MAE) and cross-domain (CD-MAE) error, per-marker and per-dataset consistency (M-AvgCV, D-AvgCV), marker ranking (MRC), and the correlation between MIC and performance or calibration (MAC, MCC).

Core Findings: Inconsistency and Limited Discriminability

A primary contribution is the demonstration that LLMs are faithfully miscalibrated even by model-centric standards—i.e., even when using the model’s own assignment of meaning to markers, rather than aligning with human linguistic expectations, the mapping between epistemic markers and internal confidence is unstable and context-dependent. Markers do not consistently represent specific confidence levels across task domains or shifts in data distribution.

Figure 1: An example of the framework used to calculate the internal confidence a model associates with the epistemic marker "I think."

Key numerical results:

• In-domain alignment (ID-MAE) values are low (e.g., 0.08–0.16), indicating somewhat stable marker–confidence associations within a task.
• Cross-domain alignment (CD-MAE) is substantively worse (e.g., 0.12–0.29), evidencing limited generalization of MICs between tasks.
• Within-dataset marker discriminability (D-AvgCV) is generally low (e.g., 0.08–0.29): most models' use of markers does not meaningfully separate uncertainty levels, even though intrinsic confidence distributions for responses often span the full 0–1 range.
• Marker rankings are somewhat preserved across tasks (MRC ≈ 0.7–0.9), but this is largely attributable to whether a response contains any hedge or none (<no_hedge>), rather than nuanced marker semantics.

Figure 2: Representative violin plots of models' MIC densities across datasets, stratified by response correctness.

Figure 3: Representative violin plots of models' MIC densities across datasets, stratified by faithful calibration level.

Figure 5: Heatmaps of MIC values for selected markers across datasets, highlighting weak discriminability and substantial context sensitivity.

These results demonstrate that the internal semantics models assign to linguistic hedges are compressed and lack robustness under distribution shift, indicating that current LLMs do not develop a stable, model-internal lexicon for expressing uncertainty.

Marker Semantics, Model Scale, and Calibration

Despite low discriminability, larger models can encode more distinct, albeit coarse, gradations of uncertainty across markers. KDE analyses show multiple, albeit weak, density peaks for large models, suggesting some limited ability to represent several levels of uncertainty (see Figure 6 for Llama3.1-8B-Instruct and Figure 7 for Qwen3-32B).

Figure 8: KDE plots of MIC values for Llama3.1-8B-Instruct; generally narrow confidence ranges, but with weak secondary peaks.

Figure 9: KDE plots of MIC values for Qwen3-32B; stronger multimodal structure, indicating slightly more robust uncertainty encoding with larger model size.

Model size correlates with improved MIC stability but does not yield clear improvements in per-marker discriminability: larger models show lower ID-MAE, CD-MAE, and M-AvgCV values, but not discernibly higher D-AvgCV.

A notable empirical finding is that MIC tracks accuracy (MAC often > 0.6) rather than faithful calibration (MCC tends negative with larger models). In other words, markers’ internal confidence levels correspond to task difficulty or correctness, not to the alignment between model-internal confidence and human-perceived linguistic assertiveness. This implies that LLMs could at best leverage epistemic markers to indicate dataset-level task uncertainty rather than encode nuanced, sentence-level gradations of their own uncertainty in a linguistically interpretable way.

Figure 10: Plots correlating MIC values for markers with human–model decisiveness divergence (MF), showing positive trends—high-MIC markers are a primary driver of unfaithful calibration.

Implications for Trustworthy AI and Future Research

These findings carry substantial ramifications for AI trustworthiness and model-user alignment. Since LLMs' use of epistemic markers fails to consistently reflect their own internal confidence—and cannot be reliably mapped to specific uncertainty bands—users may be misled by hedge phrases, increasing the risk of over- or under-reliance on system output.

Moreover, the observed alignment between MIC and accuracy, but not with faithful calibration, demonstrates that current LLMs are incapable of flexibly decoupling surface-level linguistic hedging from raw performance. As such, attempts to train or prompt models to express uncertainty linguistically should not presume that marker choices already correspond to the model's internal epistemic state.

Practically, this suggests that model-centric approaches alone are insufficient for robust, transparent uncertainty communication. It will be necessary to develop either (a) training protocols that enforce explicit alignment between marker use and internal uncertainty signals, or (b) auxiliary interpretation layers that translate between intrinsic confidence and linguistic expressions in a stable, calibrated manner.

Theoretically, the work highlights a structural misalignment between the features LLMs learn for uncertainty and the kinds of metacognitive communication desirable for human-AI interaction. Addressing this may require architectural innovations or refined calibration objectives that target the link between internal state and communication interface.

Conclusion

This paper delivers a rigorous, quantitative analysis demonstrating that LLMs do not consistently or reliably associate epistemic markers with their own internal confidence, even on the model’s own terms and independent of human interpretation. Marker semantics are not stable under domain shift, and their discriminability is weak even within a domain, despite some improvement with model scale. These findings underscore an ongoing challenge for AI safety, reliability, and interpretability, and suggest the need for new alignment objectives and calibration protocols to enable more trustworthy and transparent uncertainty communication in future generations of LLMs.