Log-ICoT: Internalizing Chain-of-Thought in Transformers

• Log-ICoT is a staged training curriculum that internalizes chain-of-thought by absorbing intermediate reasoning into hidden states using a parity-tree structure.
• It trains transformers on the k-parity task in logarithmically many stages (L = log₂ k), preserving the sample efficiency of explicit chain-of-thought approaches.
• The approach integrates specialized architecture modifications and stabilization techniques to eliminate inference overhead while maintaining clear internal reasoning.

Searching arXiv for the target paper and closely related curriculum/ICoT work to ground the article in the latest literature. Log-ICoT Curriculum is a staged training procedure for implicit chain-of-thought (ICoT) in transformers. In the formulation analyzed in "Transformers Provably Learn to Internalize Chain-of-Thought" (Huang et al., 27 May 2026), it is designed to make intermediate reasoning steps that would normally be emitted as explicit tokens at inference become absorbed into hidden states during training. The paper studies this mechanism on the $k$-parity task, proves that an $L$-layer transformer trained under Log-ICoT learns $k$-parity with $\mathsf{poly}(n)$ samples and $L=\log_2 k$ training stages, and positions the curriculum as an alternative to explicit chain-of-thought that preserves sample efficiency while eliminating inference-time reasoning-token generation.

1. Conceptual definition

Log-ICoT is a curriculum for internalizing chain-of-thought rather than displaying it. The motivating distinction is between explicit CoT and ICoT. In explicit CoT, intermediate reasoning is written as a sequence of “thinking tokens” and consumed autoregressively at inference. This can substantially improve sample efficiency, but it makes decoding slow and computationally expensive because the model must emit a long reasoning trace before producing the answer. In ICoT, the same intermediate structure is kept inside the network, so the final answer can be produced in a single forward pass (Huang et al., 27 May 2026).

The specific novelty of Log-ICoT is curricular compression. Standard ICoT, as described in the paper, removes thinking tokens one at a time. If there are $k-1$ intermediate positions, this yields roughly $k-1$ training stages, which is linear in $k$. Log-ICoT instead removes thinking tokens in geometric chunks aligned with the levels of a parity tree, reducing the number of stages to $L=\log_2 k$. The paper presents this as the first theoretical analysis of ICoT for multi-layer transformers and as an extension of earlier one-layer parity guarantees to deeper architectures.

A common misconception is to treat ICoT as the absence of reasoning. The target formulation states the opposite: the intermediate reasoning steps are still present as computation, but they are internalized into hidden states rather than written into the token stream. Under this view, Log-ICoT is not a prompt-formatting trick; it is a training curriculum coupled to a particular architectural and optimization analysis.

2. Parity as the analytical testbed

The paper uses the $k$-parity task as a clean setting in which the role of intermediate reasoning can be isolated (Huang et al., 27 May 2026). The input is a random vector $L$0, together with a secret subset $L$1 of size $L$2, and the target is

$L$3

This problem is described as hard for standard gradient-based training without intermediate supervision: with only polynomially many samples and queries, one cannot achieve nontrivial accuracy. By contrast, if explicit chain-of-thought supervision provides the intermediate parity values in a recursive decomposition, parity becomes learnable with polynomial sample complexity. The task therefore isolates a precise phenomenon: explicit CoT makes learning feasible, but incurs an inference-time burden that scales with the length of the reasoning chain.

To expose the recursive structure, the paper assumes $L$4 and decomposes $L$5-parity into a complete binary tree of two-parities. The tree has height $L$6, the $L$7 leaves correspond to the original input bits $L$8 for $L$9, and the $k$0 internal nodes correspond to intermediate parity values. Each internal node satisfies

$k$1

where $k$2 and $k$3 are its two children, and the root $k$4 is the final target label. This binary-tree factorization is not merely representational; it is the structural basis for the curriculum. It implies a natural progression in which lower-level two-parities are learned before higher-level compositions.

3. Curriculum construction and staging

Log-ICoT organizes training by removing visible reasoning tokens in chunks corresponding to parity-tree levels (Huang et al., 27 May 2026). The curriculum is specified as follows:

• Stage 1: full CoT is visible.
• Stage $k$5 for $k$6: positions $k$7 are replaced by padding zeros, while later intermediate positions remain visible.
• Stage $k$8: all intermediate reasoning tokens except the final output are padded.

The stage boundary is

$k$9

so the number of revealed CoT positions shrinks by factors of $\mathsf{poly}(n)$0 across stages. The paper identifies this geometric reduction as the key distinction from standard ICoT. Rather than peeling off one thinking token per stage, Log-ICoT removes an entire parity-tree layer’s worth of reasoning at a time.

Method	Intermediate reasoning during training	Stage count / inference
Explicit CoT	Full reasoning tokens visible	Sample-efficient; inference requires $\mathsf{poly}(n)$1 sequential token generation
Standard ICoT	Thinking tokens removed one at a time	Roughly $\mathsf{poly}(n)$2 stages, linear in $\mathsf{poly}(n)$3
Log-ICoT	Thinking tokens removed in geometric chunks aligned with tree levels	$\mathsf{poly}(n)$4 stages; final answer in a single forward pass

The significance of this design is twofold. First, it preserves the parity-tree semantics of the target computation: each curriculum stage corresponds to a depth level in the recursive decomposition. Second, it yields a logarithmically staged training process. This suggests that the efficiency gain is not only at test time, where explicit reasoning tokens disappear, but also in the structure of the training curriculum itself.

4. Transformer architecture, supervision, and optimization

The proof in the target paper depends on a carefully specified transformer architecture rather than on an unconstrained generic transformer (Huang et al., 27 May 2026). The model uses single-head attention with fixed value matrices, a smooth link function $\mathsf{poly}(n)$5, and gated connections that isolate the role of each layer. The link function is required to satisfy

$\mathsf{poly}(n)$6

and is locally quadratic near the relevant points, for example

$\mathsf{poly}(n)$7

The local quadratic behavior is used in the analysis to make small approximation errors shrink as reasoning is composed across layers.

The curriculum and architecture are aligned so that layer $\mathsf{poly}(n)$8 learns parity-tree level $\mathsf{poly}(n)$9. In stage $L=\log_2 k$0, only the tokens corresponding to that next level are supervised, using the stage-$L=\log_2 k$1 squared loss

$L=\log_2 k$2

Training proceeds sequentially through the stages with fresh data at each stage and quantized gradient descent:

$L=\log_2 k$3

Quantization is part of the proof strategy. Once a layer has been learned, subsequent gradients become exponentially small, and rounding freezes the earlier layer’s weights. This allows later stages to be analyzed essentially in isolation.

Two technical stabilization mechanisms are emphasized. The first addresses representation collapse, described as the tendency of uniformly attended inputs to become nearly constant and thereby destroy token-distinguishing information. The paper uses fixed, level-dependent gates

$L=\log_2 k$4

so that only positions assigned to layer $L=\log_2 k$5 are actively rewritten. The second addresses error propagation through a customized causal mask that restricts each position to attend only to lower-level tokens, together with quantization, which prevents learned layers from drifting in later stages.

5. Theoretical guarantees and proof structure

The main theorem states that an $L=\log_2 k$6-layer transformer trained under Log-ICoT learns $L=\log_2 k$7-parity with $L=\log_2 k$8 samples per stage and $L=\log_2 k$9 stages, and achieves exponentially small final error (Huang et al., 27 May 2026). The paper presents this as matching the sample efficiency of explicit CoT while eliminating its inference overhead.

The proof is stage-by-stage and inductive. At stage 1, the gradient signal identifies the two correct children for each level-2 parity node. The signal is described as being of order $k-1$0, while the noise is much smaller. With the learning rate

$k-1$1

the correct attention logits are pushed to a large integer value $k-1$2, while non-children remain at zero after quantization. As a result, softmax attention concentrates almost entirely on the two true children:

$k-1$3

Because the link function maps the average of two $k-1$4 values to their product, the layer then computes the correct parity value with exponentially small error. The induction step repeats the same logic: once layers $k-1$5 are learned, their hidden states approximate the relevant intermediate parity values well enough that layer $k-1$6 effectively receives clean inputs and can learn the next tree level.

A hidden-state propagation lemma formalizes this compositional behavior. The relevant residual-stream block at layer $k-1$7 is either the original input bit $k-1$8 or the already computed parity value, depending on whether that position has already been “written.” The sample-complexity argument uses concentration: with $k-1$9 samples, all nontrivial empirical cross-correlations are small, of order

$k-1$0

which ensures that the gradient is dominated by the true child positions rather than by noise. The paper also highlights the analysis of fourth-order terms, because parity products generate many cancellations and the proof must count “trivial” tuples versus nontrivial ones to obtain the signal-plus-noise decomposition.

6. Empirical behavior, interpretation, and related curricula

The experiments are presented as matching the theoretical account (Huang et al., 27 May 2026). On a 4-layer transformer trained on $k-1$1 with $k-1$2, so that $k-1$3, the validation loss drops quickly in the first stage, spikes briefly when each new chunk of thinking tokens is removed, and then rapidly returns to near zero. The final accuracy reaches $k-1$4, and the learned attention maps show each layer focusing sharply on the correct pair of child positions for the parity node it is responsible for. The experiments are interpreted as visualizing progressive absorption of explicit reasoning traces into deeper layers: what was previously written as tokens is instead computed in hidden state.

This evidence bears directly on a common misunderstanding of ICoT. The point is not that the reasoning chain becomes unnecessary. Rather, the reasoning chain remains structurally necessary during learning, but the curriculum trains the model to realize that structure internally. A plausible implication is that the central distinction between explicit CoT and ICoT lies in where intermediate states are represented—token sequence versus hidden state—rather than in whether intermediate computation exists.

Log-ICoT should also be distinguished from curriculum methods that act on demonstrations rather than on internalized reasoning traces. "Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoning" (Ma et al., 21 Feb 2025) constructs a curriculum in-context learning strategy in which demonstrations are selected by problem-solving logic, using QDMR/BREAK-style operator sequences, and then ordered from easy to hard according to the number of reasoning steps. Its selection rule is prefix matching on operator sequences, and its best strategy is described as “prioritize diversity + w/ order.” That setting concerns in-context example selection and ordering, not staged removal of thinking tokens inside a transformer. The comparison is nevertheless informative: both approaches treat curriculum as a way to align training or prompting with latent reasoning structure, but Log-ICoT is specifically a hidden-state internalization curriculum for multi-layer transformers on $k-1$5-parity, whereas the logic-guided ICL method is a demonstration-selection procedure for complex reasoning tasks.

Taken together, these results position Log-ICoT as a theoretically explicit account of how a transformer can preserve the learnability benefits of chain-of-thought while removing test-time decoding overhead. By matching curriculum stages to parity-tree depth and removing reasoning tokens in geometric chunks rather than one by one, it reduces curriculum length from linear in $k-1$6 to $k-1$7, while keeping the final computation within a single forward pass.