How Likely Do LLMs with CoT Mimic Human Reasoning?

Abstract: Chain-of-thought emerges as a promising technique for eliciting reasoning capabilities from LLMs. However, it does not always improve task performance or accurately represent reasoning processes, leaving unresolved questions about its usage. In this paper, we diagnose the underlying mechanism by comparing the reasoning process of LLMs with humans, using causal analysis to understand the relationships between the problem instruction, reasoning, and the answer in LLMs. Our empirical study reveals that LLMs often deviate from the ideal causal chain, resulting in spurious correlations and potential consistency errors (inconsistent reasoning and answers). We also examine various factors influencing the causal structure, finding that in-context learning with examples strengthens it, while post-training techniques like supervised fine-tuning and reinforcement learning on human feedback weaken it. To our surprise, the causal structure cannot be strengthened by enlarging the model size only, urging research on new techniques. We hope that this preliminary study will shed light on understanding and improving the reasoning process in LLM.

• The paper demonstrates that CoT inconsistently influences LLM performance, impairing basic tasks while aiding complex reasoning.
• The study employs causal analysis and structural causal models to compare LLM reasoning with human logical processes.
• Findings highlight that training methods like RLHF can partially align model reasoning with causal logic, suggesting paths for future improvement.

How Likely Do LLMs with CoT Mimic Human Reasoning?

Introduction

The paper "How Likely Do LLMs with CoT Mimic Human Reasoning?" examines the efficacy of the Chain of Thought (CoT) approach in LLMs, particularly focusing on its reasoning capabilities. The authors investigate whether CoT reasoning accurately reflects human-like logical processes, emphasizing the effects of CoT on LLM performance across different tasks. The study aims to explore discrepancies in reasoning patterns between LLMs and humans potentially introduced by CoT methodologies.

Methodology

The authors leverage causal analysis to assess how CoT prompts influence reasoning in LLMs. They examine the Structural Causal Models (SCMs) implicated by these models during reasoning tasks, comparing them against typical human reasoning patterns. The significance of the relationship between CoT instructions and resulting answers was tested using interventions and causal inference methodologies. The impact of common training techniques, such as in-context learning (ICL), supervised fine-tuning (SFT), and reinforcement learning with human feedback (RLHF) on these causal structures was also thoroughly analyzed.

Figure 1: CoT and Answer do not fully align.

Findings and Analysis

Unstable Effectiveness of CoT: The CoT mechanism was shown to inconsistently enhance performance across different tasks. For basic arithmetic tasks, CoT impaired performance, whereas for complex reasoning problems it was beneficial, suggesting that simple tasks might not benefit from structured reasoning steps anticipated in CoT.

Incongruency Between CoT and Answers: A remarkable observation throughout the evaluations highlighted that incorrect CoT could still lead to correct answers, and vice versa, raising questions about the causal relations between steps of reasoning and outcomes provided by the models.

Impact of Model Size and Training Techniques: Larger LLMs generally demonstrated a structure more akin to effective causal reasoning, although this was not uniformly observed across all tasks. Moreover, while SFT was found to frequently infer extraneous features resulting in misalignment between CoT and answer accuracy, RLHF partially mitigated this issue by aligning answers more closely with the correct causal pathway.

Figure 2: LLMs with CoT exhibit inconsistent effects, where CoT shows inferior performance to Direct in some tasks, superior in others.

Causal Structures and Errors

The study revealed various SCM types based on the strength of relationships between Instructions, CoT, and Answers. Many LLMs exhibited SCM types that allowed for spurious dependencies influencing the relation between CoT and answers. Crucially, these spurious correlations were evident in producing logical inconsistencies, an issue less prevalent in ideal causal chains where reasoning strictly follows logical progression.

Figure 3: Potential structural causal models (SCMs) for chain-of-thought (CoT) in question-answering.

Practical and Theoretical Implications

The findings underscore significant implications for developing models that reason in a manner faithful to human-like logical processes. Alterations in training methods may help align CoT mechanisms with human reasoning more closely. Improved emphasis on causal consistency could enhance the reliability of models, especially when reasoning through complex, context-rich problems. Further research could investigate more refined causal structures or employ counterfactual reasoning integration to bridge existing gaps in LLM reasoning fidelity.

Conclusion

This research provides insights into the cognitive alignment challenges of LLMs when adopting CoT strategies. By uncovering inconsistencies between CoT reasoning and model decision-making processes, it suggests that current CoT implementations may not suffice for faithfully mirroring human logic. Future directions could explore more causally sound approaches for LLM training, or develop novel CoT paradigms that reinforce genuine causal dependencies akin to human reasoning mechanisms.