How Language Models Fail: Token-Level Signatures of Committed and Persistent Reasoning Failures

Abstract: Failures in LLM reasoning emerge through distinct processes that leave identifiable signatures in the reasoning trace. We characterize these failures using token-level uncertainty signals, finding they arise through two empirically distinguishable processes. The first is committed failure, in which a model locks onto an incorrect reasoning path early in its trace. A central diagnostic signature is the commitment point, beyond which considering additional tokens hurt rather than help failure detection. In the second, persistent uncertainty, uncertainty instead accumulates throughout, and the full trace is needed to best distinguish failing from successful completions. These signatures reproduce across 23 model-dataset configurations, with the framework's falsifiable predictions holding in 20 of 23 cases, well above chance across both failure modes. Finally, we demonstrate our failure mode framework has direct implications for self-consistency, identifying when uncertainty signals complement it and when it can be selectively skipped. These results offer a foundation for understanding when LLM reasoning failures become detectable and for adapting detection strategies accordingly.

• The paper demonstrates that LLM failures emerge as either committed failures, where early token decisions lock in errors, or persistent uncertainty, where errors diffuse over the entire trace.
• It employs token-level metrics (entropy, margin, negative log-likelihood, nucleus size, near-tie fraction) and PR-AUC analysis across 23 model configurations to quantify failure detection power.
• Findings suggest that adaptive triage strategies—such as selective self-consistency—can efficiently mitigate failures by tailoring interventions to each distinct failure mode.

Token-Level Diagnostics of LLM Reasoning Failures: Committed and Persistent Uncertainty

Motivation and Framework Overview

This work investigates the observable trajectories of token-level uncertainty in LLM chain-of-thought (CoT) completions and provides a systematic characterization of how reasoning failures manifest in such traces. The central claim is that LLM failures are not a monolithic phenomenon; instead, they emerge via two empirically distinguishable modes, each with concrete implications for detection and mitigation. These are committed failure, where the model locks into an incorrect reasoning pathway early in generation, and persistent uncertainty, in which the model remains undecided and diffuses uncertainty throughout the trace.

The framework further exploits only externally observable signals—token log-probabilities acquired from a single CoT completion—making the diagnostic methodology deployable for both open-weight and closed-API models. A suite of token-level uncertainty metrics (entropy, margin, negative log-likelihood, nucleus size, and near-tie fraction) is computed for early prefixes of CoT traces. Failure detection power is quantified by PR-AUC (Precision-Recall Area-Under-Curve) computed at progressively larger prefix windows.

Failure Mode Taxonomy and Commitment Point Identification

Under this paradigm, a model's failure mode is classified based on the shape of the PR-AUC performance curve as a function of prefix size:

• Committed Failure: The predictive power for failure peaks at an early prefix (the commitment point, $T^*$), after which inspecting more of the trace reduces discriminative power (inverted-U PR-AUC curve). This reflects early “lock-in” to an incorrect pathway, beyond which additional tokens mostly add entropy without corrective information.
• Persistent Uncertainty: Here, PR-AUC increases monotonically with window size. The full trace always confers the highest detection capability, as the model’s uncertainty persists throughout generation.

  Figure 1: Strong committed failure: Gemma4-31 on LiveCodeBench, with the delta confidence interval on PR-AUC strictly above zero, indicating the early window is maximally predictive.

A commitment point is statistically substantiated when the improvement in PR-AUC over the full-trace baseline, $\Delta(T^*)$, is positive and its 95% bootstrap CI excludes zero ("strong commitment"); "weak commitment" denotes cases where the CI spans zero but mean delta is positive.

Empirical Validation Across Models and Tasks

The framework is validated across 23 model-dataset configurations, spanning Qwen3.5, Llama3.1, GPT-OSS, Gemma, GPT-4o, Gemini-2.5Pro families, and benchmarks covering mathematics (GSM8K, MATH-500), scientific reasoning (GPQA), and code generation (LiveCodeBench).

Results demonstrate clear evidence for both failure regimes:

• Committed failure is most commonly observed (in 14/23 cases), with models like Gemma4-31B on LiveCodeBench and Gemini-2.5Pro on MATH-500 exhibiting strong inverted-U PR-AUC curves.
• Persistent uncertainty is prominent in nine configurations, e.g., Llama3.1-8B on MATH-500, where early prefixes steadily accumulate predictive power and no early window is more effective than the full trace.

No architecture or data domain is exclusively associated with either failure mode. Notably, PR-AUC gain at the commitment point ($\hat\Delta$) for committed regimes clusters strictly above zero across all configurations (sign test: 14/14, $p = 6.1 \times 10^{-5}$), with a meta-analytic pooled effect size of $+0.013$, 95% CI $[+0.005, +0.020]$.

Figure 2: Persistent uncertainty: Llama3.1-8B on MATH-500, where no early window surpasses the full trace in PR-AUC.

A comprehensive forest plot consolidates these findings, showing the positive delta for committed configurations and near-zero or negative deltas for persistent regimes.

Figure 3: Forest plot of $\Delta$PR-AUC at $T^*$ across all 23 configurations, with committed cases (blue) showing $\Delta$ in $[+0.005, +0.135]$ and persistent cases (red) at or below zero.

The analysis controls for the confound of trace length by stripping post-answer tokens ("pre-final" analysis), preventing a degenerate classification based on output length rather than genuine uncertainty evolution.

Implications for Failure Detection Algorithms

The concrete regime classification induces powerful implications for resource-allocation and triage systems in LLM deployment:

• Self-Consistency Triage: For committed failure cases, models that are highly confident early in the trace can be exempted from expensive self-consistency ensembling (majority vote over multiple completions) without sacrificing recall—empirically, up to 30% of top-confidence cases can be skipped with zero recall loss for self-consistency failures.

  Figure 4: Selective self-consistency triage shows that pre-final uncertainty features enable high-recall detection of self-consistency failures with low skip rates, especially in the committed regime.

For persistent regimes, uncertainty-based triage is less sharp, as self-consistency agreement tracks model correctness more closely.

• Complementarity with Self-Consistency: While self-consistency agreement (across sampled completions) is a robust failure signal in persistent regimes, single-trace uncertainty features remain complementary, offering additional detection power in both settings. Integrating both signals yields a consistent lift in PR-AUC (e.g., $\Delta(T^*)$0 for Gemma4-31B/GPQA).

Figure 5: Comparison of self-consistency agreement, uncertainty-based signals, and their aggregation on GPQA shows that combination improves detection across regimes and models.

Theoretical and Practical Implications

These findings refine the theoretical understanding of LLMs’ reasoning dynamics, emphasizing that failure is a process observable from output statistics alone, not solely via internal representations. This strengthens the link between mechanistic interpretability (activation-probe–based studies indicating early commitment) and externally observable signals. The deployment implications are substantial—failure mitigation should be mode-adaptive: e.g., apply resource-intensive agreement aggregation only in the persistent uncertainty regime, and prioritize architectural or training interventions for the committed regime, where uncertainty collapses prematurely.

Future work should refine the localization of the token-level commitment point and extend closed-API model analyses as log-probability access becomes more granular.

Conclusion

This token-level analysis reveals that LLM reasoning failures are not homogeneous but emerge through committed or persistent uncertainty trajectories, each with distinct detection and mitigation affordances. By exploiting only external token log-probabilities, this framework is robust to model access constraints and directly actionable for failure detection systems. Adapting detection strategy to the characterized failure regime rather than applying uniform methods is shown to improve both efficiency and predictive power. These results delineate a principled foundation for future research on interpretable, adaptive, and reliable LLM deployment.

---

Reference:

"How LLMs Fail: Token-Level Signatures of Committed and Persistent Reasoning Failures" (2606.06635)