Trained on Tokens, Calibrated on Concepts: The Emergence of Semantic Calibration in LLMs (2511.04869v1)

Abstract: LLMs often lack meaningful confidence estimates for their outputs. While base LLMs are known to exhibit next-token calibration, it remains unclear whether they can assess confidence in the actual meaning of their responses beyond the token level. We find that, when using a certain sampling-based notion of semantic calibration, base LLMs are remarkably well-calibrated: they can meaningfully assess confidence in open-domain question-answering tasks, despite not being explicitly trained to do so. Our main theoretical contribution establishes a mechanism for why semantic calibration emerges as a byproduct of next-token prediction, leveraging a recent connection between calibration and local loss optimality. The theory relies on a general definition of "B-calibration," which is a notion of calibration parameterized by a choice of equivalence classes (semantic or otherwise). This theoretical mechanism leads to a testable prediction: base LLMs will be semantically calibrated when they can easily predict their own distribution over semantic answer classes before generating a response. We state three implications of this prediction, which we validate through experiments: (1) Base LLMs are semantically calibrated across question-answering tasks, (2) RL instruction-tuning systematically breaks this calibration, and (3) chain-of-thought reasoning breaks calibration. To our knowledge, our work provides the first principled explanation of when and why semantic calibration emerges in LLMs.

• The paper demonstrates that base LLMs trained on proper loss exhibit calibrated semantic confidence for direct responses.
• It establishes an equivalence between semantic calibration and local loss optimality, connecting calibration errors to training dynamics.
• The work evidences that RLHF and chain-of-thought reasoning disrupt calibration, leading to significant confidence misalignments.

Emergence of Semantic Calibration in LLMs: Theory and Experiment

Motivation and Context

This paper systematically investigates the phenomenon of semantic calibration in LLMs, i.e., whether models trained purely for next-token prediction can produce semantically meaningful uncertainties regarding their long-form answers. While traditional calibration literature has demonstrated that next-token probabilities in LLMs are generally well-calibrated for token-level or multiple-choice outputs, it has been unclear whether sampling-based notions of confidence—derived from the frequency of semantically equivalent answers across generations—are themselves calibrated absent explicit training objectives.

Semantic Calibration: Definitions and Mechanisms

The authors formalize semantic calibration by introducing a collapsing function $B$, which maps model outputs to semantic equivalence classes. For any prompt $x$, the LLM induces a categorical distribution over $K$ semantic classes by sampling responses, applying $B$, and computing the empirical frequencies. Calibration then measures whether, for all questions with semantic confidence $c$, the true semantic accuracy matches $c$ in expectation.

Figure 1: Illustration of semantic calibration via a collapsing function $B$ mapping generations to semantic classes; semantic confidence is computed from the plurality class among sampled outputs.

The paper’s central theoretical contribution is an equivalence between semantic calibration (via $B$) and local loss optimality: an LLM is $B$-calibrated if it is loss-optimal with respect to a perturbation class that can rescale the probability mass assigned to semantic classes. This reduction allows the authors to connect calibration to the optimization dynamics of pretraining, under the hypothesis that standard maximum likelihood objectives should discourage persistent calibration errors that can be “easily fixed” by simple perturbations—provided the model internally “knows” its own semantic class distribution prior to generation.

Experimental and Theoretical Predictions

The theory gives rise to three empirically testable predictions:

1. Base LLMs exhibit semantic calibration on in-distribution data. When responses are direct (i.e., not chain-of-thought or multi-step reasoning), even purely pre-trained models are well-calibrated with respect to semantic confidence.
2. Calibration breaks with non-proper loss post-training. Models subject to RLHF, DPO, or other post-training procedures diverge from calibration, often becoming overconfident; this is explained by the non-proper loss landscapes involved in RL alignment.
3. Chain-of-thought (CoT) reasoning typically disrupts calibration. For reasoning tasks requiring generation of intermediate steps (e.g., math word problems), the model cannot “know” its final semantic class a priori, breaking the conditions for calibration.

The authors validate these predictions on extensive benchmarks covering GSM8K, OpenMathInstruct, TriviaQA, and SimpleQA, analyzing a wide spectrum of models (Qwen, Gemini, Mistral, Llama) across direct, sentence-form, and CoT prompts.

Figure 2: Reliability diagrams for semantic calibration across model families, prompt types, and datasets; base models with direct response types are consistently well-calibrated.

Numerical Results and Scaling Behavior

Reliability diagrams demonstrate consistently low calibration errors for base models across sizes from 0.5B to 70B parameters when using direct or sentence responses (SmoothECE below $0.05$ in most cases). Calibration error drastically increases for instruct models (RLHF/DPO) and for chain-of-thought settings, often yielding large overconfidence or underconfidence regimes.

Figure 3: Semantic confidence calibration curves for multiple base models and datasets, showing strong alignment between confidence and empirical accuracy.

Model scaling exhibits counterintuitive effects: semantic calibration is not significantly impacted by model size; small base models remain well-calibrated even when their accuracy is low. In the miscalibrated settings, increased accuracy only marginally affects calibration error except near perfect accuracy. This is in sharp contrast to next-token calibration literature, where scaling typically improves calibration.

Figure 4: Scatter plot of calibration error versus semantic accuracy; in direct response settings, calibration error is decoupled from model capability/scale, while RLHF-CoT configurations show characteristic over- or underconfidence.

Comparisons across variants of the same model architecture (e.g., Mistral-7B, Zephyr SFT, DPO) show that only models trained with sequence-level proper loss (SFT or standard pretraining) maintain semantic calibration, but reward-model-trained variants are egregiously miscalibrated.

Figure 5: Calibration error in Mistral-7B derivatives; DPO post-training introduces significant miscalibration relative to SFT and base models in sentence response mode.

Theoretical Implications and Implementation Guidance

The results underscore that sampling-based semantic confidence is a theoretically justified—and, in base models, empirically reliable—approach to uncertainty quantification for open-ended generation. For practical deployment, semantic confidence requires multiple samples per prompt, post-processed via a collapsing function $B$ (e.g., a strong LLM classifier or extraction pipeline).

Despite strengths, the computational cost of sampling $M \gg 1$ generations per example is a limitation for production pipelines. For CoT domains or instruct/RLHF models, empirical confidence does not match true answer correctness; further research into either explicit calibration regularization or architecture-probing is required. The authors propose a LoRA-based diagnostic: sequence-level semantic calibration is present if a small adapter can accurately predict the model’s own semantic class distribution before generation begins.

Relation to Prior Work and Generalization

The semantic calibration formalism generalizes earlier sampling-based methods (self-consistency, semantic entropy) and provides a unifying theory by parameterizing calibration with respect to arbitrary post-processing functions $B$. These connections extend to conformal prediction and weighted calibration frameworks. The paper’s results suggest that models trained with log-loss objectives naturally propagate semantic calibration if the “semantic class” is an easy-to-learn function of the prompt; full calibration for arbitrary $B$ is provably infeasible for large $K$.

Limitations

Semantic calibration as explored here covers only sampling-based confidence on open-form QA pads; the framework does not address verbalized confidence, nor the computational cost of sampling in high-throughput scenarios. The requirement for “in-distribution” prompts (i.e., the test data matching the training domain) restricts generalization to adversarial or out-of-distribution settings, such as TruthfulQA or common misconception datasets.

Conclusion

This work provides a principled and well-supported theory for the emergence of semantic calibration in base LLMs. Empirically, semantic confidence—defined by sampling and semantic collapse—is predictively useful when models are trained on proper losses and the semantic class is computable directly from the prompt. These findings have direct implications for deploying LLMs in reliability-critical settings, suggesting that uncertainty quantification via semantic sampling is justified for base models but not for reward-aligned or chain-of-thought-optimized systems. The abstract theoretical mechanism introduced can likely be extended to other generative modeling frameworks, emphasizing loss optimality with respect to “easy-to-learn” functions as a driver for reliable empirical confidence.