Refining certainty metrics beyond token-level probabilities

Construct and assess alternative certainty metrics for Certainty-Guided Reasoning (CGR) beyond token-level min-of-max probabilities, such as entropy-based measures, variance across sampled reasoning trajectories, or agreement with external verifiers, and determine whether they yield improved stopping decisions and predictive reliability.

Background

CGR’s certainty is currently computed from token-level probabilities, taking the minimum across the answer tokens. While simple and effective, this measure may not capture broader uncertainty in the reasoning process.

Alternative metrics could better reflect confidence, including entropy of distributions, variability across sampled trajectories, or external verification signals. The open question is whether these metrics improve calibration and stop decisions.

References

Several promising directions remain open for exploration. Third, the certainty metric could be refined beyond token-level probabilities—for example, incorporating entropy, variance across sampled trajectories, or alignment with external verifiers.

— Certainty-Guided Reasoning in Large Language Models: A Dynamic Thinking Budget Approach  (2509.07820 - Nogueira et al., 9 Sep 2025) in Conclusions and Future Work

Running the m sweep the registration already pre-specifies is recommended before the tolerance result is cited as evidence of robustness.

— Chain-of-Thought Entropy as a Reliability Signal: A Preregistered Reproduction  (2609.19606 - Cochran, 17 Sep 2026) in Appendix, Section 13, Analyses Not Performed; Appendix, Section 16, The Monotonicity Tolerance

Finally, our declarative-statement protocol is complementary to standard likelihood-based MCQA, where answer options are scored in the original multiple-choice context. We include this formulation as a control, but leave fuller comparisons with calibrated likelihood scores, option-fluency corrections, open-ended factual probes, and truth-evaluation protocols such as $P(\mathrm{True}\mid s)$ to future work.

— Likelihood Ranking doesn't Scale Like Prompting in LLMs  (2609.29390 - Bondielli et al., 24 Sep 2026) in Limitations, final paragraph of Section 6

We conjecture that the limited performance of all investigated UQ extensions arises from the somewhat imprecise description of potential reasoning branches inherent in the turn-level reasoning process.

— GRUET: Quantifying Uncertainty of Agentic Reasoning-and-Acting Processes  (2609.24831 - Liang et al., 21 Sep 2026) in Section 1, Introduction