Linguistic Confidence in AI Systems

• Linguistic Confidence (LC) is the expression, elicitation, or estimation of uncertainty through language using verbal probability phrases, epistemic markers, and decisiveness in reasoning.
• LC research employs methods such as isotonic regression, hedge mapping, and distributional calibration to align linguistic expressions with empirical correctness using metrics like ECE and Brier scores.
• The approach has practical implications in multilingual settings and downstream tasks, improving trust, user reliance, and decision-making by ensuring calibrated verbal expressions of uncertainty.

Searching arXiv for papers on linguistic confidence, linguistic calibration, and epistemic markers in LLMs. Linguistic Confidence (LC) denotes the expression, elicitation, or estimation of uncertainty through language rather than solely through scalar probabilities. In the contemporary LLM literature, LC is not a single formalism but a family of related constructs: verbalized probability phrases, epistemic markers such as “probably” or “I think,” categorical confidence styles such as high- versus low-confidence phrasing, multilingual confidence elicitation, and step-level decisiveness in long reasoning traces. Across these formulations, the common objective is alignment between what a model says about its certainty and either empirical correctness, human-interpretable probability, or the model’s own internal uncertainty state (Chaudhry et al., 2024, Mielke et al., 2020, Xue et al., 2024).

1. Conceptual scope and definitions

A central feature of the LC literature is definitional pluralism. Some papers define LC as a mapping from calibrated probabilities to natural-language phrases; others define it as the observed reliability of a marker once that marker appears; still others distinguish linguistic confidence from intrinsic confidence and study the gap between them.

Formulation	Core object	Source
Verbalized probability bins	Mapping $LC:[0,1]\to E$ from calibrated scores to phrases such as “certain,” “likely,” or “impossible”	(Chaudhry et al., 2024)
Three-way response framing	Labels $L\in\{\mathrm{DK},\mathrm{LO},\mathrm{HI}\}$, evaluated by $P(\mathrm{correct}\mid L)$	(Mielke et al., 2020)
Marker confidence	Observed accuracy when a marker $W_i$ appears	(Liu et al., 30 May 2025)
Marker internal confidence	Average intrinsic confidence associated with marker $E$ in a dataset	(Liu et al., 27 May 2026)
Step-level decisiveness	Linguistic decisiveness $D(s_i)$ aligned with internal confidence $C(s_i)$	(Gani et al., 2 Jun 2026)
Distributional LC	Response mapped to a distribution on $[0,1]$, typically $\mathrm{Beta}(\alpha,\beta)$	(Yeh et al., 19 May 2026)

In the supervised fine-tuning framework of Chaudhry et al., a linguistic expression of uncertainty is well calibrated when, for every expression $e$, the empirical correctness of answers carrying $L\in\{\mathrm{DK},\mathrm{LO},\mathrm{HI}\}$0 matches the target midpoint probability $L\in\{\mathrm{DK},\mathrm{LO},\mathrm{HI}\}$1 associated with that phrase (Chaudhry et al., 2024). In the dialogue-calibration work of 2020, LC is the model’s verbalized stance toward its answer—explicit ignorance, hedged uncertainty, or strong confidence—and calibration is the empirical ordering $L\in\{\mathrm{DK},\mathrm{LO},\mathrm{HI}\}$2 (Mielke et al., 2020).

Later work broadened the concept. “Marker confidence” is defined as the observed accuracy of a model on all questions for which a particular epistemic marker appears (Liu et al., 30 May 2025). “Marker internal confidence” (MIC) is weaker and more model-centric: it asks whether a model uses markers consistently relative to its own intrinsic confidence, even if those markers are not human-aligned (Liu et al., 27 May 2026). In long reasoning traces, the relevant distinction becomes one between intrinsic confidence and linguistic decisiveness, with faithful calibration defined as their alignment at the level of individual reasoning steps (Gani et al., 2 Jun 2026).

2. Formal targets and evaluation criteria

The dominant calibration metrics remain Expected Calibration Error (ECE) and Brier score. In the phrase-conditioned calibration setting, ECE is computed by binning examples according to confidence values and comparing binwise empirical accuracy to average confidence,

$L\in\{\mathrm{DK},\mathrm{LO},\mathrm{HI}\}$3

while the Brier score is

$L\in\{\mathrm{DK},\mathrm{LO},\mathrm{HI}\}$4

These metrics are used both for self-evaluated probabilities and for probabilities recovered from generated phrases (Chaudhry et al., 2024).

Several papers depart from bin-based calibration and evaluate LC through conditional probabilities tied to specific linguistic phenomena. In multilingual overconfidence work, overconfidence is operationalized as

$L\in\{\mathrm{DK},\mathrm{LO},\mathrm{HI}\}$5

and overreliance risk is

$L\in\{\mathrm{DK},\mathrm{LO},\mathrm{HI}\}$6

This directly links strong certainty markers to both model error and human behavior (Rathi et al., 8 Jul 2025).

Marker-based work defines confidence at the level of an epistemic expression rather than an utterance-level scalar. For a marker $L\in\{\mathrm{DK},\mathrm{LO},\mathrm{HI}\}$7, dataset $L\in\{\mathrm{DK},\mathrm{LO},\mathrm{HI}\}$8, and model $L\in\{\mathrm{DK},\mathrm{LO},\mathrm{HI}\}$9,

$P(\mathrm{correct}\mid L)$0

so the “confidence” of the marker is simply its observed correctness frequency (Liu et al., 30 May 2025). This formulation makes it possible to study stability across domains using I-AvgECE, C-AvgECE, coefficients of variation, Marker Accuracy Correlation (MAC), and Marker Ranking Correlation (MRC) (Liu et al., 30 May 2025).

Faithfulness-oriented work replaces correctness calibration with alignment between external expression and internal state. In long-form reasoning, faithful calibration at step $P(\mathrm{correct}\mid L)$1 is

$P(\mathrm{correct}\mid L)$2

where $P(\mathrm{correct}\mid L)$3 is linguistic decisiveness and $P(\mathrm{correct}\mid L)$4 is an intrinsic-confidence estimator (Gani et al., 2 Jun 2026). In the distributional framework of Retrieval-Augmented Linguistic Calibration, LC is represented as a Beta distribution over perceived correctness, and Faithfulness Divergence is defined as

$P(\mathrm{correct}\mid L)$5

capturing both mean misalignment and the strength with which that misaligned belief is held (Yeh et al., 19 May 2026).

A recurrent methodological point is that the literature is not uniform about whether LC should be scalar, categorical, marker-conditioned, or distributional. This suggests that “confidence expressed in language” is being operationalized at multiple levels simultaneously: phrase selection, answer style, marker semantics, and reasoning-trace pragmatics.

3. Elicitation, mapping, and training strategies

One major line of work induces LC by calibrating a model’s own self-evaluated probabilities and then fine-tuning it to emit matching verbal phrases. In this approach, a QA corpus is split into few-shot, calibration, and training subsets; for each generated answer, the model is re-prompted with a True/False self-evaluation, and the normalized probability of the “True” token is taken as confidence. An isotonic regression map is then fit on calibration data, and recalibrated scores are converted into phrases using eight bins ranging from $P(\mathrm{correct}\mid L)$6 “certain” to $P(\mathrm{correct}\mid L)$7 “impossible.” Supervised fine-tuning on answer-plus-phrase pairs yields post-finetuning models with ECE $P(\mathrm{correct}\mid L)$8 and Brier $P(\mathrm{correct}\mid L)$9 on held-out TriviaQA and AmbigQA, and a simple postfix format outperforms prefix or interleaved placements; for Gemini Medium on TriviaQA, postfix gives $W_i$0 versus $W_i$1 for prefix and $W_i$2 for interleaved (Chaudhry et al., 2024).

A second line of work treats hedges themselves as the primary signal and learns a lightweight mapper from hedge text to probability. One study constructed a large-scale hedging dataset by collecting 40,000 candidate expressions from four LLMs, sampling 10,000 for annotation, and filtering 50,000 raw annotations down to 12,762 valid annotations and 1,622 final expressions. The final inventory contains 64 high-confidence, 988 moderate, 428 low, 131 lowest, and 11 complete-uncertainty expressions. A DistilRoBERTa encoder with a linear head and sigmoid output is trained under mean-squared error to match human mean scores; on the test set, the mapper achieves MSE $W_i$3 when trained on human annotations, compared with $W_i$4–$W_i$5 for LLM-based prompting baselines, at near-zero monetary cost and sub-second latency (Tao et al., 29 Sep 2025).

Prompting remains a strong baseline. Under a vanilla one-sentence QA prompt, LC on SimpleQA for gpt-5-mini yields $W_i$6 and $W_i$7; adding the instruction “If you are uncertain, linguistically hedge” produces LC+, reducing ECE to $W_i$8 and increasing AUROC to $W_i$9. The same study reports that enabling chain-of-thought reasoning reduces ECE by $E$0 for both LC and LC+, and that a small LoRA fine-tuning pipeline improves Qwen3-8B on NQ-Open from $E$1, $E$2 to $E$3, $E$4 (Tao et al., 29 Sep 2025).

A third line of work treats LC as a post-hoc rewriting problem rather than a generation-time skill. Retrieval-Augmented Linguistic Calibration models each response as a distribution over perceived correctness, calibrates the mean with Platt scaling while preserving concentration, retrieves hedging phrases whose Beta profiles match the calibrated target, and rewrites the original response accordingly. Across three QA benchmarks and five model families, this pipeline improves in-domain faithfulness and calibration by up to 66% and 58%, respectively, without any additional fine-tuning (Yeh et al., 19 May 2026).

Earlier dialogue work used an explicitly controlled generation architecture. A calibrator predicts whether a baseline answer is correct from encoder and decoder hidden states; the predicted probability is then mapped to a control token such as LO or HI, and a controlled generator rewrites the answer with linguistically calibrated confidence. This establishes an early template for LC as a controllable text-generation attribute rather than merely a passive diagnostic (Mielke et al., 2020).

4. Multilingual LC and language dominance

Multilingual confidence estimation extends LC from English-centric settings to cross-linguistic and language-specific contexts. MlingConf frames the problem around both language-agnostic (LA) and language-specific (LS) tasks, using four human-evaluated multilingual datasets for LA tasks and one LS dataset tailored to specific social, cultural, and geographical contexts of a language. Its core findings are asymmetric: on LA tasks, English exhibits notable linguistic dominance in confidence estimations over other languages, whereas on LS tasks, using question-related language to prompt LLMs demonstrates better linguistic dominance. These findings motivate a native-tone prompting strategy that employs language-specific prompts for LS tasks and improves reliability and accuracy in LS scenarios (Xue et al., 2024).

A later multilingual study examines five languages—English, French, German, Japanese, and Mandarin—through the distribution of epistemic markers and the behavior they induce. The generated marker distributions differ sharply across languages: English 4% weak / 46% moderate / 50% strong, French 9% / 45% / 46%, German 17% / 30% / 53%, Japanese 59% / 32% / 9%, and Mandarin 2% / 18% / 80%. The paper interprets these as consistent with documented linguistic variation, with Japanese exhibiting the most uncertainty markers and German and Mandarin the most certainty markers (Rathi et al., 8 Jul 2025).

That cross-linguistic variation does not imply safety. The same study reports that GPT-4o’s overconfidence rate $E$5 differs by language: 11.26% in English, 15.97% in French, 15.42% in German, 14.85% in Japanese, and 18% in Mandarin. Human reliance on strong markers is also high across languages, with 66% in English and 79% in Japanese, yielding overreliance risk that is nearly 12% in Japanese versus 7.5% in English. A particularly important result is that users rely significantly more on expressions of uncertainty in Japanese than in English, so more hedging does not necessarily protect against overreliance (Rathi et al., 8 Jul 2025).

The multilingual literature therefore distinguishes at least three issues: the model’s capacity to estimate confidence in different languages, the distributional semantics of confidence markers across languages, and language-specific user interpretation. This suggests that LC cannot be reduced to direct translation of English uncertainty behavior.

5. Human reliance and downstream decision-making

LC matters because it mediates trust and action. In multilingual reliance experiments, participants saw difficult geography questions paired with the start of an LLM answer and chose whether to “Rely on AI” or “Look up themselves.” The study used 45 bilingual speakers per language for English, French, German, and Japanese, with self-reported proficiency of at least C1; stimuli comprised 5 strengtheners, 5 weakeners, 15 moderates, and 5 plain expressions per language; language order and item order were fully randomized; and reliance rates were reported with bootstrapped 95% confidence intervals. The headline result is not merely that users overrely on confidence markers, but that reliance behaviors differ cross-linguistically in ways that undermine naive assumptions about hedging as a universal safeguard (Rathi et al., 8 Jul 2025).

Long-form calibration work formalizes this human-facing perspective more directly. There, an LM is linguistically calibrated if its generations enable downstream users to make calibrated probabilistic forecasts. Training proceeds in two stages: supervised fine-tuning on “summary distilled” responses that report approximate claim frequencies, followed by reinforcement learning with a proper scoring-rule reward based on user forecasts. On in-distribution TriviaQA and out-of-distribution Jeopardy, the resulting LC RL model achieves accuracy of approximately 64.7% and ECE of approximately 0.108, compared with approximately 63.3% and 0.367 for a strong factuality-trained RL baseline. On a held-out biography generation task, claim-level accuracy is approximately 46.8% versus 39.9%, and claim-level ECE is approximately 0.266 versus 0.601 (Band et al., 2024).

The earlier dialogue-agent study provides a complementary, low-accuracy but instructive case. A vanilla BlenderBot-2.7B system was strongly overconfident: about 27.8% of responses were labeled HI, but only 13.75% of those were correct. The calibrator-controlled chatbot reduced HI frequency to 1.8% while increasing correctness among HI responses to 38.89%, with overall accuracy changing only from 4.8% to 5.1%. The calibrator itself achieved ECE 0.018, MCE 0.292, and ANLL 0.165 (Mielke et al., 2020).

A common misconception is that LC is merely rhetorical decoration on top of answer generation. The decision-theoretic and human-reliance results show otherwise: LC changes whether users defer, verify, or act, and it can be evaluated by the quality of downstream forecasts rather than by surface fluency alone.

6. Faithfulness, robustness, and unresolved problems

A major controversy in the field concerns whether epistemic markers can be treated as stable verbal probabilities. Out of distribution, current evidence is unfavorable. Marker confidence is relatively well behaved within the same dataset, where I-AvgECE is lower than numeric-confidence NumECE for all but one model and MAC is above 0.7 on average, but it becomes unstable across datasets: C-AvgECE exceeds NumECE for 6 of 7 models, C-AvgCV averages 23.4%, and MRC averages only about 0.21. The same marker can therefore correspond to widely different accuracies across tasks (Liu et al., 30 May 2025).

Model-centric evaluation does not eliminate the problem. MIC asks whether a model uses a marker consistently relative to its own intrinsic confidence rather than to human interpretation, yet the resulting picture is still one of “faithfully miscalibrated” behavior. Across 13 LLMs and 11 tasks, ID-MAE is roughly 0.08–0.16 and CD-MAE 0.12–0.29; marker-level and dataset-level coefficients of variation are low to moderate; MRC is relatively high at roughly 0.68–0.90 only when the special <no_hedge> marker is included; and MCC is mixed or negative for larger models. Larger models improve MAE and stability somewhat, but not fine-grained discrimination among markers (Liu et al., 27 May 2026).

Large reasoning models introduce a further complication: even if they exhibit strong answer accuracy or elaborate reasoning traces, their expressed decisiveness need not track internal uncertainty. A systematic framework for LRMs finds that models are highly decisive, with average decisiveness around 0.717 for DeepSeek-R1-8B across five datasets, while cMFG* faithfulness scores range only from about 0.64 to 0.79 depending on the intrinsic-confidence estimator. Different estimators disagree substantially: DeepConf aligns better with expressed certainty than RCC or sampled-response consistency, and prompt interventions such as “perception” or “MetSens+Hedge” change mean cMFG* by at most about $E$6, despite sometimes improving accuracy by up to approximately 15 percentage points (Gani et al., 2 Jun 2026).

Interventions on the linguistic surface can help, but not uniformly. Regression analysis over chain-of-thought n-grams identifies phrases associated with higher or lower confidence and shows that suppressing a small set of high-positive-coefficient expressions can reduce ECE and ACE without major accuracy loss for many models; for example, R1 Qwen improves from ECE 0.334 to 0.318 and from Brier 0.713 to 0.700 while accuracy moves from 0.627 to 0.630. But the same strategy can fail badly: for R1 Llama, ECE drops from 0.245 to 0.079 while accuracy collapses from 0.620 to 0.251 and Brier worsens from 0.615 to 0.769 (Ozaki et al., 7 Apr 2026).

At the same time, metacognitive-control work indicates that internal confidence is behaviorally real. In a four-phase abstention paradigm, calibrated choice confidence predicts abstention far better than difficulty, retrieval accessibility, or sentence-embedding features; for GPT-4o, the standardized effect of confidence in the full abstention model is $E$7, compared with about $E$8 for RAG and 0.11 for difficulty and the top embedding component. Activation steering on Gemma 3 27B shifts abstention from 66.5% at $E$9 to 7.0% at $D(s_i)$0, with mediation analysis attributing 67.1% of the total effect to confidence redistribution (Kumaran et al., 23 Mar 2026). This does not solve LC, but it shows that the underlying internal signal exists and can causally drive behavior.

Taken together, these results suggest that LC is best understood as an alignment problem with at least three layers: calibration to correctness, faithfulness to internal uncertainty, and interpretability to human audiences. Current systems can often produce useful uncertainty language, and in some settings that language can be calibrated or improved post hoc. But marker meanings are unstable across domains, multilingual interpretation is culturally contingent, long reasoning traces remain only moderately faithful, and surface-level hedging is not a reliable proxy for either truth or safety.