Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization

Abstract: Recent work, using the Biasing Features metric, labels a CoT as unfaithful if it omits a prompt-injected hint that affected the prediction. We argue this metric confuses unfaithfulness with incompleteness, the lossy compression needed to turn distributed transformer computation into a linear natural language narrative. On multi-hop reasoning tasks with Llama-3 and Gemma-3, many CoTs flagged as unfaithful by Biasing Features are judged faithful by other metrics, exceeding 50% in some models. With a new faithful@k metric, we show that larger inference-time token budgets greatly increase hint verbalization (up to 90% in some settings), suggesting much apparent unfaithfulness is due to tight token limits. Using Causal Mediation Analysis, we further show that even non-verbalized hints can causally mediate prediction changes through the CoT. We therefore caution against relying solely on hint-based evaluations and advocate a broader interpretability toolkit, including causal mediation and corruption-based metrics.

• The paper demonstrates that CoT explanations can be causally faithful even when hints are not explicitly verbalized, challenging established evaluation metrics.
• It employs diverse methodologies—including FUR, filler tokens, and logit lens analysis—to uncover multiple causal pathways in LLM reasoning.
• Results reveal that traditional bias metrics often conflate explanation incompleteness with unfaithfulness, urging the development of broader interpretability benchmarks.

Chain-of-Thought Explanations: Faithfulness Beyond Hint Verbalization

Introduction

The work "Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization" (2512.23032) provides a systematic analysis of the conditions under which Chain-of-Thought (CoT) explanations by LLMs are deemed faithful with respect to the model's internal computations. Contradicting prior research that heavily relies on the Biasing Features metric for evaluating CoT faithfulness, this paper demonstrates that the absence of explicit hint verbalization does not necessarily imply a lack of causal faithfulness. Instead, much measured “unfaithfulness” is frequently attributable to inherent incompleteness in compressed natural language explanations or the limitations of the metric rather than genuine misalignment. The study integrates multiple experimental axes, including alternative faithfulness metrics, inference-time sampling, and causal mediation analysis, and informs methodological progress for model interpretability in NLP.

Figure 1: Overview of approach, summarizing the critique of Biasing Features, evaluation with multiple metrics, and causal analysis methodologies.

Theoretical Framework and Methodology

The central critique targets the interpretation of the Biasing Features metric, a common hint-based evaluation where faithfulness is contingently defined by explicit CoT mention of a provided cue (hint) that flips the model’s prediction. The paper adopts multi-hop question answering datasets such as OpenbookQA, StrategyQA, and ARC-Easy, employing Llama-3-8B-Instruct, Llama-3.2-3B-Instruct, and gemma-3-4b-it as underlying models. Besides the Professor hint (natural language cue), experiments encompass XML Metadata and Black Squares (visual markers) as additional biasing signals. Faithfulness assessment is extended to the Filler Tokens corruption (contextual faithfulness) and Faithfulness through Unlearning Reasoning steps (FUR, parametric faithfulness via NPO-based step erasure).

This framework is supplemented by inference-time scaling (faithful@k) to test the effect of output variability and explanation length, as well as Logit Lens analysis and mediation-based probability decomposition (NDE/NIE) to quantify causal information flows not reflected in explicit verbalization.

Empirical Results and Metric Contradictions

Unfaithfulness via Biasing Features

Replicating prior results, Biasing Features consistently labels over 80% of instances as unfaithful across models and datasets, flagging nearly all cases lacking explicit mention of the hint even after the model’s prediction changes accordingly.

Figure 2: Unfaithfulness rates (>80%) via Biasing Features across tasks, models, and hint types; 95% bootstrap confidence intervals.

Divergence with Alternative Metrics

A major empirical claim is the systematic divergence between Biasing Features and other faithfulness metrics. For CoTs flagged as unfaithful by Biasing Features, the Filler Tokens (contextual) and FUR (parametric) metrics respectively identify 20–40% (sometimes exceeding 50%) of these CoTs as faithful, particularly for Llama-3.2-3B-Instruct under the Professor hint. With FUR, which intervenes directly on model parameters, at least half of evaluated CoTs contain at least one causally faithful step for several model/setting pairs.

Figure 3: Proportion of contextually faithful CoTs (Filler Tokens) among those classified as unfaithful by Biasing Features.

Figure 4: Percentage of parametrically faithful CoTs (FUR metric) among those deemed unfaithful by Biasing Features.

These findings demonstrate that the overwhelming majority of “unfaithfulness” alleged by prior work arises from overly restrictive definitions relying on explicit verbalization.

Incompleteness vs. Unfaithfulness

To distinguish between incompleteness (lossy explanation compression) and genuine unfaithfulness, the paper adapts the pass@k approach (here, faithful@k over $k$ samples). Results indicate that for certain hints and models, especially with naturalistic hints, the faithful@k rate increases substantially with $k$—reaching nearly 0.9 for gemma-3-4b-it at $k=16$—implying that with a large enough output budget, nearly all runs yield at least one faithful explanation. For Metadata and Black Squares hints, faithful@k remains flat, indicating structural difficulties in verbalizing certain compressed or non-linguistic cues.

Figure 5: faithful@k rates—probability of at least one hint-verbalizing CoT—increase strongly with sample budget in several settings.

Logit Lens and Causal Mediation Analysis

The analysis is extended to direct measurement of information flow and influence. Using Logit Lens, the paper deciphers activation of hint-related tokens across model layers, revealing that even when hints are not explicitly verbalized, representation space activations align with hint content, particularly at answer/prediction prompts and contrastive steps.

Figure 6: Activation patterns of hint-related tokens in Llama-3.2-3B-Instruct, grouped by function and centered on CoT structure.

Causal Mediation Analysis (CMA) quantifies direct and indirect effects of hint injection on the probability of selecting the hinted answer and on the redistribution of probability among alternatives. For all tasks and models, the Natural Indirect Effect (NIE) is consistently positive and nonzero for non-verbalizing CoTs, sometimes matching or exceeding the Natural Direct Effect (NDE). This evidences that CoTs mediate part of the hint’s causal effect on predictions, independent of explicit cue mention.

Figure 7: Direct and indirect effects (NDE/NIE) of the Professor hint on predicted probability for the hinted answer.

Figure 8: NDE/NIE on sum of non-hinted option probabilities; indirect suppression of alternatives is frequently large.

A notable finding is that in several cases, the indirect effect (mediation through CoT) is larger for decreasing alternative probabilities than for boosting the hinted answer, underscoring the multiplicity of causal pathways not captured by naive verbalization checks.

Implications and Future Directions

The findings robustly challenge the dominant narrative that CoT is not explainability due to its failure to verbalize known interventions. The analysis establishes that standard “hint-verbalization” metrics systematically conflate incompleteness with unfaithfulness and thereby underestimate the explanatory value of CoTs. The practical implication is that LLM users and safety auditors should avoid optimizing solely for verbalized faithfulness and measure CoT explanations with a broader set of contextual, parametric, and causal-tracing tools. Theoretically, these results point to compressed natural language as an incomplete but potentially reliable proxy for causally-relevant computation, where omission of some factors is a feature of explanation “compression” rather than misalignment.

The work foregrounds the need for future research to (1) devise more robust, multi-faceted faithfulness benchmarks, (2) explore methods incentivizing the surfacing of latent, real-world decision factors rather than toy cues, and (3) leverage causal analysis techniques for comprehensive model monitoring.

Conclusion

This study demonstrates that the absence of hint verbalization in CoTs does not constitute evidence for explanation unfaithfulness in LLMs. Rather, the use of singular metrics like Biasing Features generates both false negatives and a misleading conception of model interpretability. CoTs often encode causally relevant reasoning steps, and their effectiveness as explanations depends not solely on overt cue mention but on a broader, outcome-relevant alignment. The integration of sampling, parametric intervention, and probabilistic causal analysis yields a more nuanced and actionable interpretability toolkit for LLM auditing and deployment.