Manifestation of Confidence in LLM Reasoning

Determine how confidence is manifested within the reasoning process of large language models by identifying and characterizing the internal signals present during token generation that reflect confidence in intermediate steps and final answers.

Background

The paper discusses limitations of Test-Time Reinforcement Learning approaches that rely on majority voting as a proxy for confidence, noting the risk of reinforcing incorrect pseudo-labels. It motivates exploring other forms of prior knowledge within LLMs, particularly the token-level probability distribution, which may encode uncertainty, confidence, and decisiveness signals relevant to reasoning quality.

This open question frames the need to understand confidence at a process level, not just outcome level, as a foundation for designing intrinsic reward mechanisms that can stably guide reinforcement learning on unlabeled data.

References

While using confidence as a correctness proxy aligns with cognitive principles, a critical question remains: how is confidence actually manifested in the reasoning process of LLMs?

— Rewarding the Journey, Not Just the Destination: A Composite Path and Answer Self-Scoring Reward Mechanism for Test-Time Reinforcement Learning  (2510.17923 - Tang et al., 20 Oct 2025) in Section 1 (Introduction)

To the best of our knowledge, despite a substantial body of work on output diversity, model confidence, and calibration, the relationship between lexical or token-level diversity and verbalized confidence or calibration has not yet been investigated.

— Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversity  (2608.13430 - Proskurina et al., 13 Aug 2026) in Section 2, Related Work