---
title: Chain-of-Thought in Transformers
url: https://www.emergentmind.com/topics/transformers-with-chain-of-thought
type: topic
---

# Chain-of-Thought in Transformers

Chain-of-thought (CoT) in transformers denotes the use of intermediate reasoning states before a final prediction, usually as autoregressively generated tokens that remain in context, but also as latent hidden-space updates, continuous thought vectors, summarized traces, or loop-aligned recurrent states. In this literature, CoT is treated not merely as a prompting format but as a computational resource: it can add effective serial depth, enlarge persistent memory, alter optimization dynamics, and expose or internalize algorithmic structure that immediate-answer transformers do not reliably realize [2305.15408][2310.07923][2605.30757].

## 1. Formal role of CoT in transformer computation

The foundational theoretical claim is that bounded-depth transformers without intermediate generation are sharply limited on serial and recursive problems. One line of work shows that bounded-precision transformers can be simulated by uniform \(TC^0\) circuits, which yields impossibility results for arithmetic formulas, the Circuit Value Problem, and Hidden Markov Model decoding when the model must answer directly after reading its input [2305.15408]. A tighter analysis for constant-depth transformers with constant-bit precision places the no-CoT regime in \(AC^0\), while log-precision retains the usual \(TC^0\) upper bound; with \(T\) CoT steps, however, constant-depth transformers with \(O(\log n)\) embedding size can solve any problem solvable by Boolean circuits of size \(T\) [2402.12875].

A more fine-grained complexity-theoretic account makes the dependence on CoT length explicit. For decoder-only transformers with a logarithmic number of decoding steps, the gain over standard transformers is limited; with a linear number of steps, regular-language recognition becomes available; and with polynomially many steps, the model class matches polynomial-time computation under the paper’s assumptions. The central correspondence is stated as
\[
\mathsf{TIME}(t(n)) \subseteq \mathrm{CoT}(t(n)) \subseteq \mathsf{SPACE}(t(n)+\log n),
\]
with the specific identification \(\mathrm{CoT}(\mathrm{poly}(n))=\mathsf{P}\) for the generalized pre-norm setting analyzed there [2310.07923].

These results collectively establish a standard formal view of CoT: the intermediate sequence functions as a serial computational trace. Rather than enlarging a transformer’s depth in the architectural sense, CoT reuses a fixed model over a growing context, so the output sequence itself becomes the medium through which recursive or dynamic-programming style computation is staged [2305.15408][2310.07923][2402.12875].

## 2. Step budget, memory regime, and lower bounds

The number of CoT steps is not a superficial design choice. In hard-attention transformers, systematic lower bounds show that many algorithmic problems require long scratchpads. For Boolean functions with average sensitivity \(\Theta(N)\), any unique-hard-attention CoT computation needs \(\Omega(N)\) steps. The paper instantiates this for Parity, sensitive bits of Multiplication, and Median, and gives an \(\Omega(|E|\log |V|)\) lower bound for reachability in a family of DAGs. It also shows that “dot-by-dot” pause-token scratchpads are even weaker: for Parity, any polynomial-length version is ineffective, and superpolynomial length is required [2502.02393].

A complementary perspective treats CoT as a persistent-memory mechanism. In the memory-budget separation framework, a CoT scratchpad stores intermediate state in emitted tokens and therefore has persistent memory
\[
M_{\text{CoT}}=\Theta(\ell \log |\Sigma|),
\]
which grows with scratchpad length. By contrast, a compressed latent loop carries only a fixed recurrent state \(z_t\) with persistent budget \(M_{\text{CLL}}=sdp\) bits, yielding
\[
\mathrm{CLL}[s,d,p,T] \subseteq DSPACE\big(O(sdp+\log n+\log T)\big).
\]
Under the stated standard complexity assumption, such compressed loops cannot decide \(P\)-complete problems when their persistent state is only polylogarithmic, whereas polynomial-length CoT can decide all languages in \(P\). Full sequence-state loops occupy a different regime because their persistent state scales with sequence length [2605.30757].

This memory view also motivates summarized and algorithmically efficient CoT variants. Low-precision softmax transformers with summarized CoT can simulate Turing machines with model size scaling logarithmically in a space bound rather than a time bound, and the same paper argues that summarized CoT is more efficient than standard CoT for space-bounded computation [2605.18079]. At a higher algorithmic abstraction level, CoT transformers can simulate Word RAM algorithms with only polylogarithmic overhead in \(n\); for flat instruction sets the overhead reduces to log-square, and for multiplication-free flat instructions it becomes only logarithmic, in contrast to the quadratic overhead incurred by known Turing-machine-based CoT simulations [2606.19697].

## 3. Learnability, sample efficiency, and optimization

Several papers argue that CoT matters not only because it enlarges expressivity, but because it changes what gradient-based training can discover. In a parity-learning setup, one work proves that a one-layer transformer without CoT is already expressive enough to represent any parity function, yet the sample requirement remains exponential, whereas CoT converts the problem into a sparse sequential dependency structure and reduces sample complexity to \(\tilde O(n)\). The same analysis shows that attention becomes sparse and nearly one-hot, with the learned map satisfying
\[
\left|\mathrm{softmax}(C + E(b) A E(b))[j,i] - 1(j=S[i])\right| < \frac{1}{n^8}
\]
after training on CoT-formatted data [2410.05459].

A separate theoretical study on the \(k\)-parity problem emphasizes process supervision. Without intermediate supervision, any finite-precision gradient-based method requires substantial iterations to solve parity from finite samples. With teacher forcing and a loss over all intermediate parity values,
\[
L(W)=\frac{1}{2n}\sum_{m=d+1}^{d+k-1}\|\hat x_m-x_m\|^2,
\]
the model can learn parity in one gradient update when \(n=\Omega(d^{2+\varepsilon})\). Even without teacher forcing, data augmentation and self-consistency checking yield efficient learning in \(\log k\) iterations with exponentially small final error [2410.08633].

The same theme appears in in-context learning for linear regression. On an in-context weight-prediction task, a one-layer linear transformer without CoT can implement only a single step of gradient descent, with optimal output \(\hat w_{\text{one-step}}=\frac{\eta^\*}{n}Xy\). When trained with CoT, the same architecture can learn to perform multi-step gradient descent autoregressively:
\[
\hat w_{k+1}=\left[I-\left(I-\frac{\eta}{n}XX^\top\right)^{k+1}\right]w^\*,
\]
and for \(k=\Theta(\log d)\) the evaluation loss becomes \(O(1/\operatorname{poly}(d))\) [2502.21212].

Implicit internalization is also learnable under suitable curricula. The Log-ICoT curriculum removes thinking tokens in geometric chunks rather than one at a time, proving that an \(L\)-layer transformer with \(L=\log_2 k\) training stages learns \(k\)-parity with polynomial sample complexity, total training samples \(O(B\log k)\) for \(B=\Omega(n^{2+\epsilon})\), and single-forward-pass inference after reasoning has been absorbed into hidden states [2605.28600].

## 4. Mechanistic accounts of what CoT-trained transformers compute

Mechanistic work on state tracking shows that CoT can correspond to identifiable algorithmic circuits rather than opaque performance gains. In a study of group-composition tasks, Transformer+CoT learns world-state tracking in a form consistent with a finite state automaton (FSA). Activation patching localizes the critical circuit to late-layer MLPs, especially the last three. The paper introduces compression and distinction metrics for neuron sets associated with hidden states and reports nearly 100% accuracy for both, 90% of neurons in layer 11 mapped to a unique state with F1-score at least \(0.2\), and roughly 15% with perfect state-subset correspondence. The same model remains robust to skipped steps and noisy scratchpads, but length generalization drops sharply on much longer sequences [2502.20129].

A solvable transformer model for CoT state tracking provides a training-dynamics account of how such circuits emerge. There, RoPE attention learns fixed-lag action retrieval while a specialized MLP logic module applies the retrieved permutation to the current state. The dynamics are summarized by three order parameters—attention retrieval \(A\), teacher-matrix alignment \(R\), and off-target logic overlap \(S\)—and training proceeds through a mixed-heuristic phase, an attention “lock-on” phase, and a rapid alignment phase that explains the sharp transition in rollout accuracy [2606.18164].

Continuous CoT introduces a different mechanism: superposition of multiple reasoning traces in latent space. On directed graph reachability, gradient-based training yields an index-matching logit that first grows and then stays bounded. The bounded regime balances exploitation of plausible next nodes with continued exploration over multiple candidate traces, so the latent thought becomes a superposition rather than a prematurely collapsed single path [2509.23365].

## 5. Externalized scratchpads, latent CoT, and loop-aligned variants

The field now distinguishes between explicit CoT, which externalizes reasoning as visible tokens, and several internalized variants. Explicit CoT remains the clearest case theoretically and mechanistically, but it incurs autoregressive inference overhead. Implicit CoT addresses that cost by progressively internalizing intermediate steps, as in Log-ICoT, while retaining the polynomial sample-efficiency guarantees established for parity [2605.28600].

Looped and recurrent models provide another route. RELAY aligns loop iterations with CoT reasoning steps and applies iteration-wise intermediate supervision during training of a Looped Transformer. According to the paper, this preserves length generalization, enables the looped model to predict CoT reasoning steps for unseen data, and then uses those generated chains to fine-tune an auto-regressive model, yielding significant improvements in the latter’s performance [2502.08482].

Whether depth recurrence by itself yields interpretable latent CoT remains contested. In the analysis of Huginn-3.5B, a depth-recurrent transformer with recurrent blocks reused at inference time, probing with the Logit Lens and Coda Lens finds limited evidence of interpretable latent CoT, strong block-dependent inconsistencies, and only marginal gains from increasing recurrence depth. On GSM8K, explicit CoT prompting reaches \(24.87/38.13\) strict/flexible accuracy at 64 recurrent steps, whereas no-CoT prompting ranges from \(3.11\) to \(4.93\) as recurrent steps increase [2507.02199].

These comparisons suggest a persistent divide between latent computation and externalized reasoning. External scratchpads remain the most transparent and, in several settings, the most effective; latent and recurrent approaches can improve efficiency or length generalization, but they do not automatically reproduce the structured, staged traces observed under explicit CoT [2502.08482][2507.02199][2605.28600].

## 6. Task structure, applications, and limits of the paradigm

The effectiveness of CoT depends on how the intermediate computation is represented. For arithmetic, one study isolates the order of decoder output tokens as a critical variable. Its pipeline trains on mixtures of candidate output orders, ranks them by early loss descent, and uses a two-stage hierarchical search over block-level and intra-block permutations. Across four order-sensitive arithmetic tasks, it identifies a learning-friendly order out of a few billion candidates; for multiplication it automatically recovers the reverse-digit order reported in prior work, and the discovered orders raise test success rate from about \(10\%\) to nearly \(100\%\) [2506.23875].

CoT is also being exported beyond natural-language reasoning. In wireless communications, CHOOSE introduces autoregressive latent reasoning steps inside a shallow transformer for in-context symbol detection. At query position \(t\), latent thought embeddings are updated as
\[
\tilde e_t^{j}=\mathrm{Transformer}(D_k,y_t,\tilde e_t^1,\ldots,\tilde e_t^{j-1}), \quad j=1,\ldots,C,
\]
and only the final thought \(\tilde e_t^C\) is projected to the prediction. On 16QAM at 30 dB SNR and 64QAM at 40 dB SNR, a 1-layer CHOOSE model with \(C=4\) achieves over 50% lower symbol error rate than a vanilla 1-layer in-context-learning transformer; a 2-layer CHOOSE with \(C=2\) matches a 4-layer vanilla model; and CHOOSE reaches similar or better error rates with 5–10 times fewer parameters, using about \(27{,}000\) parameters versus about \(400{,}000\) for an 8-layer deep ICL baseline. The paper attributes the modest extra computation to key-value caching and re-use of previous keys, values, and input embeddings [2506.21093].

At the same time, a common misconception is that CoT universally dominates direct prediction. A memorization-theoretic study gives a negative answer to that stronger claim. For finite reasoning datasets with \(N\) elements, both CoT and no-CoT fixed-precision transformers require \(\overline{\Theta}(N)\) parameters in the worst case; the necessary and sufficient conditions for memorization in the two settings do not imply each other; some datasets are memorizable only by no-CoT transformers and others only by CoT transformers; and some simple infinite datasets cannot be memorized by transformers with or without CoT [2511.01190].

The current literature therefore supports a nuanced conclusion. CoT can fundamentally extend the computational and learnable behavior of transformers, especially for inherently serial or compositional tasks, but its benefits depend on step budget, memory regime, ordering of intermediate states, supervision strategy, and the architectural interface through which reasoning is externalized or internalized [2310.07923][2410.05459][2511.01190].

Source: https://www.emergentmind.com/topics/transformers-with-chain-of-thought