Large Language Models Decide Early and Explain Later

Abstract: LLMs often achieve strong performance by generating long intermediate chain-of-thought reasoning. However, it remains unclear when a model's final answer is actually determined during generation. If the answer is already fixed at an intermediate stage, subsequent reasoning tokens may constitute post-decision explanation, increasing inference cost and latency without improving correctness. We study the evolution of predicted answers over reasoning steps using forced answer completion, which elicits the model's intermediate predictions at partial reasoning prefixes. Focusing on Qwen3-4B and averaging results across all datasets considered, we find that predicted answers change in only 32% of queries. Moreover, once the final answer switch occurs, the model generates an average of 760 additional reasoning tokens per query, accounting for a substantial fraction of the total reasoning budget. Motivated by these findings, we investigate early stopping strategies that halt generation once the answer has stabilized. We show that simple heuristics, including probe-based stopping, can reduce reasoning token usage by 500 tokens per query while incurring only a 2% drop in accuracy. Together, our results indicate that a large portion of chain-of-thought generation is redundant and can be reduced with minimal impact on performance.

• The paper demonstrates that LLMs often settle on their final answer early in the reasoning process, with only 32% of cases showing subsequent answer changes.
• The analysis reveals that 35–67% of generated reasoning tokens are redundant post-decision, underscoring inefficiencies in chain-of-thought protocols.
• Probe-based early stopping reduces token usage by up to 1000 tokens per query with minimal accuracy drop, offering practical benefits for resource-constrained settings.

Early Answer Commitment and Redundant Reasoning in LLMs

Introduction

The paper "LLMs Decide Early and Explain Later" (2604.22266) provides an empirical analysis of the temporal dynamics of answer formation in LLMs during chain-of-thought (CoT) reasoning. Contrary to the assumption that each reasoning step is necessary for decision refinement, the authors demonstrate that model answers often stabilize early in the reasoning trace, and a substantial fraction of subsequent tokens serve as redundant, post-decision explanations. This work analyzes answer evolution across diverse tasks and models, quantifies token efficiency, and proposes early stopping strategies with minimal impact on downstream performance.

Methodology

The core contribution is the introduction of forced answer completion to probe intermediate decision points: after each reasoning step, the model is queried for its current answer by terminating the CoT generation and prompting for an immediate prediction. This produces a fine-grained answer trajectory $T = (A_0, A_1, ..., A_n)$ corresponding to sequential reasoning units $R = (R_1, ..., R_n)$. Answer switches, transient answer flips, and final answer stabilization points are systematically identified. Additionally, the authors implement a denoising technique ("hold-for-k" smoothing) to distinguish substantive answer changes from ephemeral flips and evaluate the number of reasoning tokens produced after the final substantive answer switch.

Empirical investigation covers a spectrum of tasks:

• Multiple-choice QA (MCQ)
• Numeric-answer tasks
• Search-query generation with tool interfaces
• Tool-selection in agentic environments

A variety of Qwen3 and GPT-based model architectures and scales are analyzed.

Key Findings

Answer Formation Dynamics

The analysis reveals that, averaged across tasks and models, the predicted answer changes in only 32% of cases, indicating early decision commitment. For nearly half of MCQ queries and the vast majority in tool-selection, the answer is fixed from the start of reasoning. Even when answer switches occur, they predominantly happen within the initial segments of the reasoning trace.

Transient answer flips (short-lived, locally inconsistent answer changes) are frequent but almost never persist. Denoising removes over 70% of switch events in numeric tasks, refining the estimate of genuine answer switches.

Redundant Reasoning Token Generation

Highly redundant CoT generation is pervasive. On average, 760 tokens per query (often comprising 35–67% of the total reasoning trace) are generated after the final substantive answer switch. This pattern holds across MCQ, numeric, and open-ended search tasks and persists even on complex benchmarks (Humanity's Last Exam, GPQA-Diamond, AIME 2026), equivocating the notion that harder reasoning tasks necessitate prolonged answer evolution.

In tool-selection scenarios, answer stability is even more pronounced, with almost no substantive answer switches, underscoring that lengthy rationales often do not affect the selected tool or final output.

Early Stopping Strategies

The paper evaluates online early-stopping mechanisms, notably:

• Random gating (baseline)
• Probe-based gating: A linear probe observes intermediate model states and predicts whether answer stabilization is achieved

Both task-specific and generic probes (the latter trained across all task domains) achieve a strong accuracy-token trade-off. For Qwen3-4B, probe-based stopping reduces reasoning token usage by 500 tokens per query with only a 2% decrease in accuracy. In numeric tasks, 1,000 token reductions come with <4% accuracy drop. Cosine similarity for search-query arguments remains >0.95 with substantial token reductions.

A universal probe—without task-specific adaptation—performs on par with the best bespoke probes, indicating a shared underlying structure in reasoning state representations that signals decision commitment.

Implications

Theoretical Implications

These findings challenge the canonical view of CoT reasoning as a necessity for accurate prediction. Once the answer is fixed internally, remaining generation is primarily a post hoc justification rather than an essential computational process, aligning with prior critiques of chain-of-thought faithfulness. This decoupling of decision-making and explanation exposes limitations in contemporary explanations as epistemic windows into LLM internal cognition.

The presence of robust, generic signals for answer stabilization hints at the existence of emergent representations related to certainty or "belief closure" in LLMs, which could be leveraged for introspection, verifiability, and adaptive computation.

Practical Implications

Inference cost, latency, and energy usage are heavily affected by unnecessarily verbose reasoning. Probe-based early stopping, being a decoding-time intervention, offers a practical solution for efficient deployment, especially in high-throughput and agentic LLM systems. Resource-constrained environments and tool-integrated workflows benefit from decreased output verbosity and increased responsiveness without material loss in accuracy or output quality.

Future Directions

Extensions to larger-scale models and multimodal architectures may elucidate whether early commitment is a universal property or task/modality-bound. Refinement of probe architectures could enable finer-grained introspective control, interactive explanation, and even adaptive reasoning depth based on user or downstream task feedback. Understanding the interplay between instruction tuning, CoT prompting, and internal decision boundary formation remains a promising avenue.

Conclusion

This work rigorously demonstrates that LLMs typically commit to their final answer early in chain-of-thought generation, and that a significant portion of generated reasoning tokens are redundant from an accuracy perspective. Strategically truncating reasoning sequences using learned or even generic gating mechanisms can yield substantial efficiency gains with marginal performance degradation. These results inform both the scientific understanding of LLM cognition and the design of more efficient, transparent inference protocols.