---
title: Chain of Thoughts (CoT) in LLMs
url: https://www.emergentmind.com/topics/chain-of-thoughts-cot
type: topic
---

# Chain of Thoughts (CoT) in LLMs

Chain of Thoughts (CoT) is a prompting and modeling paradigm for large language models (LLMs) that elicits explicit, step-by-step reasoning traces—or “chains of thought”—prior to outputting final answers. CoT has become central to efforts in scaling, understanding, and controlling the reasoning behaviors of LLMs across arithmetic, symbolic, commonsense, and open-domain tasks. It comprises both a set of practical prompt-based methods as well as a focal point for theoretical analysis of reasoning in neural networks.

## 1. Formal Definition and Fundamental Mechanisms

In the canonical formulation, a CoT-augmented LLM, given an input prompt $p$, generates a chain of intermediate reasoning steps $r=(r_1, \dots, r_n)$, followed by a final answer $a$. The joint generation process decomposes as:
$$
P(r, a \mid p) = P(r \mid p) \cdot P(a \mid p, r)
$$
with $P(r \mid p) = \prod_{i=1}^n P(r_i \mid p, r_{<i})$ [2310.04959]. Each $r_i$ is typically rendered as one or more English-language sentences, equations, or code lines expressing a partial rationale.

The motivating principle is that prompting for multi-step rationales enhances LLM accuracy on tasks needing compositional or abstract reasoning (arithmetic, symbolic manipulation, commonsense QA, code generation). Empirically, CoT is effective once model scale exceeds $\sim$10B parameters, with performance gains often showing “emergent” scaling [2310.04959].

Mechanistically, recent theoretical and empirical work converges on the interpretation that CoT is not a guarantee of abstract “reasoning” in the mechanistic sense; rather, it functions as a powerful structural constraint on sequence generation [2506.02878]. CoT prompts induce the LLM to allocate high probability mass to trajectories matching familiar reasoning formats (e.g., “Step 1: …, Step 2: …, Therefore, …”), which narrows down generative variability and increases answer accuracy—primarily by imitation and pattern activation rather than symbolic deduction.

## 2. Prompting Strategies and Design Variants

### Zero-Shot vs. Few-Shot CoT
- **Zero-Shot CoT**: Attaches a generic instruction (“Let’s think step by step”) to the query. Triggers stepwise rationales without exemplars; surprisingly effective at scale [2310.04959].
- **Few-Shot CoT**: Prepends several worked examples $(x_j, r_j, a_j)$ before the query. Yields more robust and faithful chains, especially on complex or structured tasks [2310.04959].

### Automatic and Dynamic CoT
- **Auto-CoT**: Selects or synthesizes exemplars (cluster center, diversity-promoting [CDW-CoT]) to maximize coverage.
- **Clustered/Adaptive CoT**: CDW-CoT clusters the training data via sentence embeddings and optimizes a prompt-distribution for each cluster. At inference, test questions receive a distance-weighted mixture of cluster-specific prompt selections, yielding up to +25.34% accuracy improvement over manual CoT [2501.12226].
- **Uncertainty-Aware CoT**: UnCert-CoT applies CoT only when uncertainty is high, prioritizing stepwise inference for difficult lines in code generation and bypassing it on “easy” sub-tasks to prevent overthinking [2503.15341].

### Extension Methods
- **Self-Consistency**: Samples $k$ independent CoT traces via temperature sampling and selects the answer with the modal support [2310.04959].
- **Decomposed Prompting (LtM, Self-Ask)**: Breaks complex tasks into explicit sub-questions, each answered in sequence [2310.04959].

### Structural and Modal Variants
- **Program-aided CoT**: Uses code interpreter (PAL, PoT) to execute chain steps, commonly for math/code tasks.
- **Compact CoT**: CAC-CoT restricts reasoning to a small set of “connector” phrases to induce shorter, more efficient chains (~1/3 the length of standard CoT) while preserving or increasing accuracy across System-1 and System-2 benchmarks [2508.18743].
- **Collaborative/Editable CoT**: Co-CoT exposes CoT steps as user-editable blocks, tracks edits, and supports preference learning and responsible AI [2504.17091].

## 3. Theoretical Insights, Empirical Analysis, and Limitations

### Mechanisms for Improvement
- **Sparse Attention and Sequential Dependence**: CoT scaffolds the learning problem into an arrangement where each intermediate token depends on a small, local subset of previous tokens (sparse dependency graph). This enables transformers to efficiently learn via attention pointing, reducing sample complexity from exponential (no CoT) to polynomial (with CoT) in settings such as parity learning [2410.05459].
- **Decoding-Space Pruning**: CoT acts as a decoding filter, concentrating the probability mass over answer templates and solution traces that conform to well-learned formats (template adherence correlates $r \approx 0.8$ with answer accuracy on GSM8K) [2507.20758].

### Mechanistic Debates
- **Imitation vs. Reasoning**: Recent theoretical critique holds that CoT does not induce genuine reasoning, but rather tightens the generative constraint to trajectories resembling stepwise, human-annotated explanations seen in pre-training; it does not abstract or derive novel inference patterns in zero-shot [2506.02878].
- **Internal State as Variables**: Empirical interventions show that CoT tokens are actively read and updated as mutable “variables” within LLM hidden states—akin to program variables—enabling direct manipulation and causal interventions [2505.04955].

### Known Limitations
- **Curse of CoT in Pattern-Based ICL**: In pattern-inference benchmarks, CoT can underperform direct answering by 5.1% absolute, degrading because long rationales introduce context distance, disrupt implicit pattern learning, and inject explicit inference noise [2504.05081].
- **Task-Specific Fragility**: On hard audio reasoning tasks, longer chains can propagate errors; for open-domain QA, CoT may hallucinate in absence of retrieval [2501.07246].

## 4. Scaling, Multimodality, and Applications

### Scaling and Performance Trends
- For models $<10$B, CoT can hurt performance; above this scaling threshold, it enables sudden jumps in arithmetic, symbolic, and commonsense reasoning [2310.04959].
- Parameter-efficient architectures (e.g., KAM-CoT at $<300$M params) can, with knowledge graph-enhanced CoT, surpass GPT-3.5/4 on science benchmarks, indicating that architectural and data design can partially substitute brute-force model size [2401.12863].

### Multimodal and Non-Textual CoT
- **Audio-CoT**: Extends CoT reasoning to audio-language models, using audio-anchored exemplars. Results establish positive correlation between chain length and task accuracy, but also highlight that for complex sound/music question, wrongly calibrated chains can outperform the baseline [2501.07246].
- **Knowledge-Augmented Multimodal CoT**: KAM-CoT fuses language, vision, and external KG streams via cross-attention and GNNs, using a two-stage (rationale, answer) decoder for interpretable, grounded chains [2401.12863].

### Code, NLU, and Program-Based CoT
- **Program CoT**: For math problem solving, executable CoT (particularly self-describing Python) provides superior accuracy and diversity compared to natural-language or “bare” code CoTs, especially when paired with reranking or self-consistency. Python is favored over Wolfram due to LLM pre-training biases [2309.11054].
- **Masked LM CoT**: CoTT adapts CoT for MLMs (e.g., BERT/Roberta) in NLU via slot-based decomposition and prompt tuning, yielding state-of-the-art results on hierarchical classification and relation extraction [2310.11721].

## 5. Analysis, Control, and Steering of Reasoning Strategies

- **Strategy Discovery and Control**: The CoT Encyclopedia framework automatically discovers and clusters types of reasoning strategies from LLM traces, enabling the analysis and steering of models towards strategies with higher empirical accuracy by explicit prompt control [2505.10185].

- **Dynamic Prompting**: Approaches like CDW-CoT construct dynamic, instance-adaptive prompt distributions via clustering and distance-weighted mixing, achieving substantial gains over global, fixed CoT strategies [2501.12226].

- **Error Mitigation and Reliability**: Deep hidden cognition analysis shows that specific transformer attention heads stably encode “truthfulness” of individual CoT steps; extracting and leveraging these activations for confidence prediction enables more reliable CoT decoding via confidence-calibrated beam search [2507.10007].

## 6. Distillation, Efficiency, and Optimization

- **Knowledge Distillation**: CoT rationales serve as white-box scaffolds for distilling large LLM reasoning capability into smaller models, producing up to +37.5% improvement in BBH task accuracy. The key gain comes from exposing the student to intermediate reasoning distributions, not just final answers [2511.05184].

- **Efficiency and Compression**: Compact CoT (CAC-CoT), Markov Chain-of-Thought (MCoT), and continuous CoT modeling (MARCoS) variants reduce inference cost by constraining reasoning trace length, compressing state between steps, and decoupling step-level generation from fine-grained token-level sampling—achieving large speedups and/or memory savings with limited accuracy tradeoff [2509.25020, 2410.17635, 2508.18743].

## 7. Open Challenges and Future Directions

- **Faithfulness and Causal Attribution**: Most CoT methods do not guarantee that the output reasoning chain causally yields the answer, or that intermediate steps are logically valid. Ongoing work targets probing and enforcing faithfulness at the chain or token variable level [2310.04959, 2505.04955].
- **Generalization and Theory**: The field lacks a unifying theory of CoT in transformers bridging empirical “induction heads” and pattern-matching with symbolic generalization. Mechanistic understandings of “emergent” reasoning and their dependence on training data, pretraining, and architecture remain open [2506.02878, 2507.20758, 2410.05459].
- **Adaptive/Hybrid Approaches**: Several limitations—such as CoT’s underperformance on pattern-based ICL—motivate hybrid prompting control, dynamic switching between explicit and implicit reasoning, and tighter integration of retrieval, tool use, and self-verification modules [2503.15341, 2504.05081].
- **Human-in-the-Loop Reasoning**: Collaborative CoT frameworks (Co-CoT) and visible intermediate representations (CoCT) reflect growing interest in transparent, editable, and modular reasoning pipelines, which are critical for responsible AI, safety, and alignment [2504.17091, 2510.18434].

CoT thus represents both a highly practical method and a fundamental research axis in neural reasoning—enabling sophisticated behaviors through prompt engineering, model design, and a deepening theoretical and mechanistic understanding of how LLMs mimic, alter, and, in some settings, exceed the reach of human reasoning patterns.

Source: https://www.emergentmind.com/topics/chain-of-thoughts-cot