CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

Abstract: Reasoning LLMs are increasingly asked not only to answer difficult questions, but also to estimate their likelihood of success. Existing methods typically elicit confidence only once: either before thinking or after answering. We argue that confidence in reasoning models is state-dependent: before thinking, confidence should estimate the chance of the model correctly solving the prompt, while after thinking it should predict whether the realized answer is likely to be correct. This distinction determines the appropriate supervision target: prompt-level success should supervise confidence estimates made after seeing the prompt, while individual answer-level correctness should supervise confidence estimates made after answering. We introduce CALIBER (Calibration Before and After Reasoning), which elicits both estimates and supervises each with the target matched to its information state. Under this unified protocol, CALIBER reduces Expected Calibration Error (ECE) by 52.5% over the strongest single-confidence baseline on BigMathDigits for the 7B model, while achieving the best Brier score and AUROC, and remains within 2.1 points of the best accuracy. Further, on a larger 30B model, CALIBER achieves the best ECE on BigMathDigits while remaining competitive in Brier score and AUROC. Out of distribution, it achieves the best ECE and Brier score on GPQA and TriviaQA, and remains competitive on SimpleQA. Ablations further show that this position-target alignment is most beneficial under distribution shift where it consistently reduces calibration error across all out-of-distribution benchmarks.

• The paper presents a dual-confidence framework that estimates pre-thinking and post-thinking confidence to improve uncertainty calibration in LLMs.
• It utilizes a two-stage training process, leveraging prompt-level success and instance-level correctness to optimize metrics like ECE, Brier score, and AUROC.
• Empirical results demonstrate significant calibration error reduction and robust performance across both in-distribution and out-of-distribution benchmarks.

State-Dependent Calibration of Reasoning LLMs: A Technical Summary of CALIBER

Motivation and Problem Formulation

The challenge of calibrating LLMs' confidence estimates is exacerbated when models are tasked with complex reasoning. Traditionally, calibration methods have elicited confidence either before the model begins reasoning or after an answer is produced, but not both. These approaches fail to account for the dynamic evolution of a model's information state during reasoning, leading to suboptimal calibration under both in-distribution and distribution shift scenarios.

CALIBER proposes to treat confidence estimation as a state-dependent process:

• Pre-thinking confidence estimates the probability of solving the prompt before seeing any reasoning trace or answer, conditioned only on the prompt.
• Post-thinking confidence estimates the probability the realized answer is correct, conditioning on both the prompt and the full reasoning trace.

The paper demonstrates that the alignment between the location of confidence elicitation and the supervision target is critical. Supervising pre-thinking confidence with prompt-level success, and post-thinking confidence with instance-level correctness, avoids mismatches that degrade calibration, especially in OOD settings.

Methodology

CALIBER introduces a dual-verbalized confidence protocol, wherein the model emits both pre- and post-confidence values as part of a structured response:

<pre_confidence>p_pre</pre_confidence> <thinking> ... </thinking> <answer> ... </answer> <post_confidence>p_post</post_confidence>

Training Scheme

• Pre-confidence is supervised using the group-wise empirical accuracy of all rollouts for the prompt, approximating the policy's expected success rate.
• Post-confidence is supervised with the binary correctness of the specific sampled answer.
• Both estimates receive format rewards to enforce response structure, and a standard correctness reward ensures answer quality is not sacrificed for calibration metrics.
• Training proceeds in two stages: initial warmup with only pre-confidence calibration (to diversify estimates), followed by joint optimization of both calibration objectives and correctness.

Evaluation Metrics

Calibration is measured using:

• ECE (Expected Calibration Error): Quantifies the deviation between predicted confidence and empirical accuracy across bins.
• Brier Score: Captures squared error between predicted probabilities and actual correctness.
• AUROC: Measures the discriminative ability of confidence to distinguish correct from incorrect answers.
• Accuracy: Ensures calibration improvements do not degrade task performance.

Empirical Results

In-Distribution (BigMathDigits)

• 7B Model: CALIBER reduced ECE by 52.5% (0.059→0.028) compared to the best single-confidence baseline and achieved the best Brier score and AUROC. Task accuracy remained within 2.1 points of the best-performing model.
• 30B Model: CALIBER achieved the lowest ECE (0.022), competitive Brier score (0.115), and high AUROC (0.797), maintaining task accuracy parity.

Crucially, CALIBER updates post-confidence relative to pre-confidence in accordance with correctness, increasing confidence for correct answers and decreasing it for failures. This shows the model meaningfully incorporates reasoning trace evidence in final confidence estimates.

Out-of-Distribution (GPQA, TriviaQA, SimpleQA)

• CALIBER consistently yielded the lowest ECE and Brier score across OOD benchmarks, with strong AUROC reflecting robust discrimination.
• On SimpleQA, a difficult distribution shift with low accuracy, CALIBER maintained the best calibration and competitive discrimination, demonstrating resilience to domain shifts.
• The ablation experiments with swapped targets revealed substantial degradation in calibration when supervision does not match elicitation position—especially pronounced under distribution shift.

Technical Implications

CALIBER reframes confidence calibration in reasoning LLMs as a dynamic, state-dependent problem. The separation of pre- and post-confidence, with position-target alignment, enables specialization:

• Pre-confidence (prompt-level): Facilitates accurate estimation of overall task solvability, capturing domain and format cues.
• Post-confidence (instance-level): Conditions on full reasoning, refining uncertainty estimates based on explicit solution evidence.

This dual-confidence framework mitigates the calibration-discrimination tradeoff, as evidenced by favorable ECE-AUROC Pareto frontier placement. CALIBER's architecture and training protocol are agnostic to the underlying LLM, making the approach theoretically extensible to models with multi-step reasoning, tool use, or interaction settings.

Future Directions

• Fine-grained Confidence Trajectories: Extending CALIBER to track confidence evolution across intermediate reasoning steps, with targeted supervision at each stage.
• Integration with Segmented Credit Assignment: Leveraging token-level calibration for richer reward structures.
• Testing across Model Families and Domains: Assessing generalization and robustness in settings with diverse reasoning requirements.
• Adaptive Reward Weighting: Optimizing the balance among pre/post calibration and correctness rewards for better model efficacy.

Conclusion

CALIBER establishes that state-dependent confidence calibration, with explicit position-target alignment, is critical for reliable uncertainty estimation in reasoning LLMs. The framework achieves substantial calibration error reductions, maintains competitive accuracy and discrimination, and demonstrates strong robustness to distribution shifts. The evidence supports treating confidence as a dynamic property in reasoning workflows, and sets the stage for future research into multi-point, curriculum-based calibration in advanced LLM systems.

(2606.24281)