Chain-of-Thought Model

• Chain-of-thought models are reasoning frameworks that generate intermediate natural-language steps before producing final answers.
• They employ diverse methodologies—from latent hierarchical processes to iterative refinement—to boost performance on complex tasks.
• Empirical studies reveal significant accuracy gains in large models while theoretical work highlights the importance of reducing hidden-context ambiguity.

Chain-of-thought (CoT) models are reasoning formulations for LLMs in which intermediate steps are generated, selected, or internalized before a final answer is produced. In the original prompting paradigm, the intermediate steps are natural-language rationales placed directly in the token stream; in later work, CoT is also formalized as a latent hierarchical process, a sequence of state transitions, a decomposition tree, or a test-time iterative refinement procedure. Empirically, CoT prompting changes the behavior of sufficiently large autoregressive models on arithmetic, symbolic, and commonsense tasks, with early benchmark results showing that PaLM 540B on GSM8K rises from 17.9% under standard prompting to 56.9% under CoT prompting, while GPT-3 175B rises from 15.6% to 46.9% (Wei et al., 2022). Subsequent theory and systems work has treated the “CoT model” not as a single algorithm but as a family of mechanisms for expanding effective computation, controlling search, and exposing or compressing intermediate state (Tutunov et al., 2023).

1. Origins and canonical prompting form

The basic CoT model in autoregressive language modeling is a prompt format in which each exemplar is a triple of question, rationale, and answer, and the model is induced to emit a “series of intermediate natural-language reasoning steps” before the conclusion (Wei et al., 2022). In this formulation, the model still defines a standard token factorization,

$P(x_{1:T})=\prod_{t=1}^T P(x_t\mid x_{<t}),$

but the generated sequence is partitioned into reasoning tokens followed by answer tokens. The prompt itself usually contains multiple such exemplars; for math tasks, eight exemplars can fit in roughly a 1K-token prompt window, and the original experiments used greedy decoding.

The earliest systematic empirical finding was that CoT gains are strongly scale-dependent. On GSM8K, CoT prompting improved performance for very large models, but smaller models often generated plausible yet illogical rationales and could be harmed by “think step by step” prompting. The same study reported that gains emerged only at large scale, roughly $100$B parameters or above, and documented several recurring error types among incorrect CoTs on GSM8K: approximately 8% “calculator errors only,” 16% symbol-mapping errors, 22% “one step missing,” and 54% semantic or coherence errors (Wei et al., 2022).

This large-model dependence later became an object of reinterpretation rather than a fixed law. Symbolic Chain-of-Thought Distillation showed that models in the 125M to 1.3B range can benefit from CoT after training on rationalizations sampled from a much larger teacher. In that framework, a GPT-3 code-davinci-002 teacher generates many rationale-answer pairs per input, and a compact OPT student is fine-tuned on the concatenated rationale-plus-answer sequence. A central result is that sampling many chains per instance is more important than selecting only the most likely rationale: as the number of sampled chains per input increases from $1$ to $30$, student accuracy rises substantially and then saturates near $N=30$ (Li et al., 2023).

2. Hierarchical generative model and geometric convergence

A major theoretical account models CoT as a two-level latent generative process. At the first level, a discrete context variable $c\in C$ is drawn from a prior $q(c)$, followed by latent intentions $\theta_0,\theta_1,\dots,\theta_M$. At the second level, each intention emits an observed message $x_i$, with the process terminating when $x_M$ is the special “END” token. The model imposes conditional-independence assumptions: given $100$0, $100$1 is independent of everything else, and given $100$2, $100$3 is independent of future intentions and messages (Tutunov et al., 2023).

Within this framework, the true distribution $100$4 and the learned autoregressive distribution $100$5 are separated conceptually. Under the infinite-data, infinite-capacity limit, $100$6 approximates the true marginal on any finite token sequence. CoT prompting is then interpreted as a conditioning operation: the prompt $100$7 is “warped” by prepending $100$8 example chains $100$9, all drawn from the same unknown context $1$0. The theoretical question becomes whether the model’s conditional distribution over candidate chains approaches the oracle distribution that knows $1$1.

The central theorem introduces an ambiguity quantity $1$2 through the posterior identity

$1$3

and proves a geometric convergence bound. For any candidate chain $1$4,

$1$5

with $1$6. If each example satisfies $1$7, then with $1$8,

$1$9

A supporting lemma further states that as an example chain is lengthened so that it reveals more distinguishing information about the hidden context, $30$0 (Tutunov et al., 2023).

The significance of this model is that it recasts CoT prompting as hidden-context inference rather than merely verbosity. The examples do not help only by adding tokens; they reduce uncertainty about the latent reasoning regime. This suggests a precise operational criterion for effective exemplars: they should be low-ambiguity traces that are distinctive of the relevant context, either because they are long enough or because their reasoning pattern is unusually revealing.

3. Conditions under which CoT helps, saturates, or hurts

Several later theories generalize the preceding intuition into broader criteria. A learning-theoretic treatment decomposes the CoT reasoning risk into two terms with opposite roles: oracle-trajectory risk (OTR), which captures the benefit of CoT, and trajectory-mismatch risk (TMR), which captures the cost of following a mismatched reasoning trajectory. Under stability of the loss, the answer map, and the chain rule, TMR admits a tight upper bound controlled by an amplification factor $30$1; the product $30$2 then yields three regimes: bounded error growth when $30$3, linear growth when $30$4, and exponential blow-up when $30$5. The same work proves a no-free-lunch statement: if any one of the loss, the hypothesis answer map, or the chain rule lacks stability, TMR can be arbitrarily large even when OTR is zero and the hypothesis is uniformly close to ground truth (Zhang et al., 20 May 2026).

A Markovian account reaches a related conclusion from a different angle. Here each reasoning step is a state in a finite Markov chain with transition kernels $30$6, and the end-to-end task is governed by the product kernel $30$7. When the intermediate transitions are homogeneous and aligned, CoT reduces inference-time sample complexity, with the leading term scaling like $30$8, effectively a $30$9 speedup relative to direct inference. When transitions are heterogeneous, this structural gain disappears; one recovers only a logarithmic overhead in $N=30$0, with no guaranteed advantage over direct prediction (Wang et al., 27 Feb 2026). This identifies transition alignment as a structural prerequisite for reliable CoT benefit.

A decomposition theory casts CoT as an $N=30$1-ary tree that factorizes a large $N=30$2-class decision into $N=30$3 smaller classification steps with $N=30$4. In that model, direct prediction has error $N=30$5, while the CoT decomposition obeys

$N=30$6

The resulting comparison yields a critical degree $N=30$7: below this threshold, “thinking” is detrimental because too many smaller decisions accumulate error; above it, there exists an optimal depth $N=30$8, beyond which additional depth cannot further reduce error (Nadgir et al., 10 Apr 2026). A plausible implication is that “longer reasoning” is not intrinsically better; improvement requires the subtask degree and chain depth to lie in a favorable region.

An asymptotic theory of CoT in in-context linear regression reaches a similar conclusion through random matrix analysis. It derives an exact finite-depth generalization formula and a phase transition controlled by pretraining diversity $N=30$9, context ratio $c\in C$0, and noise variance $c\in C$1. Four regimes emerge: exponential improvement, polynomial decay, saturation, and overthinking. In the overthinking phase, longer reasoning eventually amplifies error; the optimal finite depth diverges as the phase boundary is approached, but the best achievable error improves only polynomially in that depth (Takanami et al., 2 Jun 2026).

These theories also clarify a recurring controversy around “one-prompt-for-all” CoT. A prompt-space analysis argues that CoT success depends not only on answer-space search but on selecting the correct reasoning template. On ten structured tasks, unsupervised CoT performed well on several regular and context-free tasks yet remained weak on harder context-sensitive tasks such as multiplication and sorting, whereas task-specific supervision “nearly always” pushed accuracy above 90% and incorrect supervision often collapsed accuracy toward zero (Zhang et al., 2024). This directly challenges the misconception that a universal instruction such as “think step by step” is sufficient across task families.

4. Mechanistic interpretations inside transformer computation

Mechanistic work has attempted to locate CoT’s effects in decoding, representation projection, and neuron activation. One account models CoT as a “decoding space pruner” that steers generation toward an answer template formalized as

$c\in C$2

where input entities, operation tokens, intermediate entities, and final answer statements define a fixed structural skeleton. Template adherence is measured by an “Imitation Count,” and on GSM8K, across four prompt variants, the correlation between Imitation Count and accuracy is approximately $c\in C$3. Prompts aligned with the task domain induce both higher adherence and higher accuracy, while misaligned prompts, such as sports-style CoT on arithmetic tasks, produce low adherence and low accuracy (Yang et al., 28 Jul 2025).

The same analysis traces the probability of critical tokens through a logit-lens pipeline and finds that CoT probability distributions are more peaked and lower-entropy than standard prompts. In open-domain tasks such as GSM8K, CoT leads to fewer total activated neurons in later feed-forward layers, whereas in closed-domain tasks such as Coin Flip and AQuA it increases later-layer activation, producing an “amplifier” effect for discrimination among a small answer set. Larger models magnify these layer-wise differences (Yang et al., 28 Jul 2025). This suggests that CoT is not a single universal internal mechanism; it modulates computation differently depending on whether the task is broad-generation or small-label-set discrimination.

A complementary lexical and behavioral analysis frames CoT as a mixture of in-context learning and pretrained priors. Generated rationales can be decomposed into structure words, feature words, verbs, and named entities. On GSM8K, structure-word usage increases sharply under manual and task-agnostic CoT relative to zero-shot CoT, while feature-word distributions remain tied to question content, indicating continued reliance on pretrained semantic priors. The same study reports that at low shot counts, the effect of noisy exemplars is small because pretrained priors dominate, but as the number of exemplars increases, model behavior shifts toward in-context signals, so false answers or false rationales increasingly destabilize accuracy and token probabilities (Yang et al., 1 Sep 2025).

This literature helps resolve a common misunderstanding: CoT is not merely a post hoc verbal explanation layer attached to an unchanged internal decision. The reported changes in template adherence, entropy concentration, and activation profiles indicate that the presence and structure of intermediate reasoning tokens can redirect the generation process itself. At the same time, the persistence of pretrained priors and the instability under noisy demonstrations show that CoT remains a hybrid phenomenon rather than a pure symbolic planner.

5. Efficiency, compression, and search over reasoning paths

Because explicit CoT consumes substantial token budget, a large subfield studies how to preserve reasoning benefits while reducing generation cost. Stepwise Perplexity-Guided Refinement defines the importance of a reasoning step by the perplexity change caused by its removal. In few-shot prompting, this allows demonstration chains to be shortened while retaining nearly the same accuracy; for example, on DeepMind Math AL1, reducing from seven to four steps cuts generated tokens by roughly 30%–40% with less than a 1% drop in accuracy on most models, and on NBC, moving from twelve to nine steps saves about 20% of tokens while staying within 1–2% of the full-CoT baseline. In fine-tuning, “Min PPL (merge)” yields higher accuracy than random removal or highest-PPL removal at equal token budgets, with a 2–3 percentage-point gain on GSM8K under SFT for the same decoding length (Cui et al., 18 Feb 2025).

SoftCoT moves part of the reasoning process into continuous space. A frozen assistant LLM receives the instruction, question, and $c\in C$4 copies of $c\in C$5, producing hidden states for those placeholders; a trainable linear projection maps these “soft thought tokens” into the representation space of a frozen backbone LLM, which then generates rationale and answer tokens. Only the projection layer is trained. On five reasoning benchmarks, SoftCoT improves average accuracy over zero-shot CoT and other baselines, with the largest gains on StrategyQA and Date Understanding, while using far fewer soft thoughts than the 24 or more hard tokens needed by Assist-CoT (Xu et al., 17 Feb 2025).

Other approaches change decoding itself. Neural Chain-of-Thought Search treats reasoning as a search over a small set of high-level operators inserted at step delimiters and scores candidate operators with a dual heuristic for correctness and progress. Averaged over AMC23, ARC-C, GPQA, and GSM8K, the method yields a Pareto improvement in both accuracy and length: on Qwen-1.5B, $c\in C$6 percentage points in accuracy with $c\in C$7 length, and on Qwen-7B, $c\in C$8 points with $c\in C$9 length (Ling et al., 16 Jan 2026). Contrastive CoT modifies token logits at decoding time by subtracting an “amateur” distribution computed under reduced context from the expert CoT logits, improving some model-dataset pairs such as Mistral-7B on CommonSenseQA and Phi-1.5 on AQuA while harming others, which underscores the sensitivity of inference-time steering methods to prompt design and hyperparameters (Shim et al., 2024).

Compression theory places a harder limit on such efficiency programs. One line of work proves that skipping intermediate steps forces the model to learn higher-order logical dependencies whose learning signals decay exponentially with interaction order; on NatBool-DAG, ALiCoT uses an auxiliary alignment objective between latent compressed states and omitted explicit CoT steps and reports a 54.4x speedup while maintaining accuracy close to explicit CoT (Li et al., 29 Jan 2026). A related complexity-theoretic analysis distinguishes CoT scratchpads from compressed latent loops: CoT grows persistent mutable memory with each generated token, whereas a compressed loop has fixed recurrent state size. Under the assumption $q(c)$0, polynomial-length CoT can decide P-complete problems that compressed loops with polylogarithmic state cannot (Zhang, 29 May 2026). Together, these results suggest that not all “reasoning compression” is functionally equivalent to explicit scratchpad reasoning.

6. Distillation, semi-supervision, and specialized deployments

CoT has also been reinterpreted as a transferable supervision signal rather than merely an inference-time prompt. In symbolic distillation, a larger teacher generates multiple rationale-answer samples per unlabeled input, optionally filtered by gold labels, and a smaller student is trained with standard autoregressive loss on the concatenated rationale-plus-answer sequence. Besides improving benchmark accuracy, this procedure changes the quality of the student’s rationales: on 900 random QA instances, human evaluators preferred the distilled OPT-1.3B model over vanilla OPT-1.3B in 59% of cases, and found no significant preference between the distilled student and the GPT-3 teacher despite the large parameter gap (Li et al., 2023).

Semi-supervised Chain-of-Thought Learning extends the same logic to unlabeled questions. Semi-CoT samples multiple pseudo-CoTs per unlabeled example, groups the extracted answers, computes normalized semantic entropy,

$q(c)$1

and retains low-entropy chains as pseudo-demonstrations. In pilot experiments on AQuA, SVAMP, GSM8K, and MultiArith, accepted pseudo-CoTs had pseudo-answer precision from 91.36% to 100%, and the method yielded small gains on SVAMP and GSM8K, negative transfer on AQuA, and ceiling behavior on MultiArith (He et al., 1 Jul 2026). This result is important because it separates two properties often conflated in CoT research: reliability of pseudo-rationales and relevance of those rationales to new test questions.

Specialized deployments show that CoT models are increasingly embedded in hybrid scientific systems. In chemical engineering, a locally deployed architecture combined DeepSeek-r1:14b and Qwen2:7b with Gaussian-process or random-forest surrogates to predict solubility from only 30 experimental data points. The ML-LLM-CoT variant was markedly more efficient during construction, requiring 2 points to rethink and 4 total rethink iterations, compared with 5 points and 34 total rethink iterations for LLM-CoT. On 20 dissimilar molecules, the number of cases with prediction deviation above 100% fell from 7 for the Gaussian model and 6 for LLM-CoT to 4 for ML-LLM-CoT, which also achieved the highest solubility-judgment success count (Zhou et al., 17 Feb 2025).

Across these developments, the CoT model has expanded from a prompt engineering trick into a broader research object spanning probabilistic inference, computational complexity, mechanistic interpretability, training-signal design, and domain adaptation. The cumulative evidence does not support a single universal explanation. Instead, CoT appears to help when it reduces hidden-context ambiguity, aligns step transitions, supplies an appropriate reasoning template, or allocates persistent memory and test-time computation in a task-matched way; it hurts when trajectories mismatch, supervision is noisy or irrelevant, chain length crosses into overthinking, or compression removes state that the task irreducibly requires.