Hidden Chain-of-Thought Prompting

• Hidden chain-of-thought prompting is a concept where internal multi-step reasoning is executed but not explicitly revealed, separating computation from visible rationale.
• It employs methodologies such as decoding search, filler-token substitution, and semantic compression to recover or condense hidden reasoning without altering input prompts.
• Empirical results show significant performance gains (e.g., GSM8K accuracy improvements and up to 3.8× speedups) while challenging the necessity of explicit rationale disclosure.

In recent LLM research, the label hidden chain-of-thought prompting has been applied to several related settings in which multi-step reasoning is preserved, surfaced, or exploited without relying on a conventional explicit rationale in the default output stream. One line of work shows that chain-of-thought (CoT) trajectories can be elicited from a pre-trained model by altering decoding rather than the prompt; another replaces explicit reasoning tokens with filler symbols while retaining recoverable internal computations; a third compresses verbose CoT into a learned special-token representation for faster inference. Taken together, these approaches recast CoT from a purely prompting-centered technique into a broader problem of decoding, representation, and interpretability (Wang et al., 2024, Bharadwaj, 2024, Liu et al., 2024).

1. Conceptual scope and formal variants

The earliest of the three formulations asks whether LLMs can reason effectively without any CoT instruction. Instead of modifying the prompt, it branches on the top-$k$ alternative tokens at the first decoding step and then greedily decodes each branch. The central claim is that CoT reasoning paths are often already present in lower-ranked continuations, and that decoding search can reveal them without prompt engineering (Wang et al., 2024).

A second formulation defines hidden CoT prompting more literally. Let $Q$ be a question, let $R=(r_1,\dots,r_m)$ be explicit reasoning tokens, and let $A$ be the final answer. Standard CoT prompting produces $S = R \circ A$. Hidden CoT prompting replaces every reasoning token $r_k$ with a filler token $f$, producing $F=(f_1,\dots,f_m)$ with $f_k \equiv f$, and trains the model to emit $S_{\mathrm{hidden}} = F \circ A$. The hypothesis, confirmed experimentally in that work, is that hidden states still carry the “would-have-been” reasoning token $Q$0 in their subspaces even though the observed target sequence contains only filler symbols (Bharadwaj, 2024).

A third formulation, denoted HCoT, treats hidden CoT as semantic compression. Standard CoT interleaves explicit reasoning steps $Q$1 with content blocks $Q$2. HCoT compresses each multi-step reasoning chain $Q$3 into a single learned vector $Q$4 emitted as a special token [CoT], and generates downstream content conditioned on $Q$5 rather than on the full reasoning text. The stated motivation is to preserve the benefits of CoT while reducing decoding length and latency (Liu et al., 2024).

Variant	Mechanism	Representative finding
CoT-decoding	Branch on top-$Q$6 first tokens, then decode greedily	Hidden CoT paths often appear among alternative continuations
Filler-token hidden CoT	Replace reasoning tokens with filler $Q$7 and probe hidden states	Rank-2 logits recover original reasoning tokens at high accuracy
HCoT compression	Encode reasoning in a learned [CoT] embedding	Shorter sequences with reported wall-clock speedups

These variants differ operationally, but they share a common premise: reasoning traces need not coincide with the visible top-1 text sequence. This suggests that “showing the rationale” and “performing the reasoning” are separable operations.

2. Decoding as a substitute for CoT prompting

The decoding-based method starts from the same simple prompt, "Q: …\nA:", and modifies only inference. At the first generation step, the model distribution $Q$8 is sorted, the top-$Q$9 initial tokens $R=(r_1,\dots,r_m)$0 are selected, and each branch is continued greedily:

$R=(r_1,\dots,r_m)$1

Each branch therefore yields a complete path $R=(r_1,\dots,r_m)$2 with its own reasoning trajectory. The method then extracts an answer span and scores the branch with a confidence gap over answer-token positions:

$R=(r_1,\dots,r_m)$3

A larger $R=(r_1,\dots,r_m)$4 is interpreted as stronger confidence in the decoded answer tokens and, empirically, as a cue that the preceding path contained a coherent CoT (Wang et al., 2024).

The empirical results are reported on PaLM-2 Large. On GSM8K, greedy decoding gives 34.8% accuracy, CoT-decoding with max $R=(r_1,\dots,r_m)$5 gives 61.5% accuracy, and CoT-decoding aggregated by $R=(r_1,\dots,r_m)$6 gives approximately 63%. On MultiArith, accuracy rises from 75.0% to 86.7%. On a year-parity task defined as even/odd birth year of 100 celebrities, accuracy rises from 57% to 95%. On symbolic reasoning tasks, reported gains include Coin Flip (2 flips) from 70% to 94%, Web-of-Lies (3 statements) from 76% to 87%, and Multi-step Arith (depth=2, length=4) from 0% to 16%. The GSM8K curve reportedly increases steadily as $R=(r_1,\dots,r_m)$7 grows from 1 to 10 and saturates around $R=(r_1,\dots,r_m)$8–10. The same study compares the method to standard CoT prompting on GSM8K, listing few-shot CoT at approximately 63%, zero-shot CoT at approximately 41%, and CoT-decoding aggregated at approximately 62%, all without changing the prompt or adding demonstrations (Wang et al., 2024).

The significance of these results is methodological. The paper argues that few-shot CoT prompts primarily serve to surface hidden reasoning into the top-1 path, whereas CoT-decoding can recover similar behavior by post-hoc decoding search. Within that framing, prompt engineering is not the only access route to step-by-step reasoning.

3. Hidden computations under filler-token CoT

The filler-token line of work studies a stricter form of hidden reasoning: the model is explicitly trained to emit filler symbols where rationale tokens would normally appear. The hidden states are defined layer by layer as

$R=(r_1,\dots,r_m)$9

and the key claim is that these hidden states still encode the original reasoning tokens even when the training loss only supervises filler outputs (Bharadwaj, 2024).

To recover those hidden computations, the paper applies the logit lens. With unembedding matrix $A$0 and hidden-state matrix $A$1, it computes

$A$2

followed by a row-wise softmax to obtain $A$3. At each CoT position, the logits are ranked. The reported phenomenon is highly specific: for the 4-layer model used in the experiments, $A$4 almost always at the final layer, whereas $A$5, the original reasoning token. Recovery during inference then uses a modified greedy rule: if the top token is the filler token, select the highest-scoring non-filler token instead (Bharadwaj, 2024).

The experimental setup is intentionally controlled. The model is a LLaMA-style 4-layer transformer with hidden size $A$6, 6 attention heads, and approximately 34M parameters. The dataset is a Match-3 / 3SUM variant: inputs are 7 tuples of 3 integers each, sampled uniformly from $A$7, with label equal to whether any triplet sums to 0 mod 10. The training set has 10M instances with 50% positive labels; the test set has 2k instances. The corruption rate is 4/3, CoT rate is 50%, the optimizer is Adam, the learning rate is $A$8, batch size is 256, and training lasts 5 epochs (Bharadwaj, 2024).

The quantitative results are correspondingly sharp. Across layers, the percentage of CoT positions whose top-1 prediction is filler rises from 8.3% at layer 1 to 96.7% at layer 4, while the percentage whose rank-2 prediction is the original token rises from 7.9% to 95.8%. Modified greedy decoding yields token-level recovery accuracy of 95.8%, compared to an approximately 5% random filler replacement baseline. Final-answer accuracy on the 3SUM test set remains 98.4% for explicit CoT, hidden CoT under standard greedy decoding, and hidden CoT with recovered tokens. The paper interprets this as evidence that the model performs core arithmetic or logic in early layers and then “overwrites” or “formats” predictions into filler tokens in later layers, suggesting a separation of concerns between internal computation and externalized text (Bharadwaj, 2024).

4. Semantic compression with learned hidden CoT tokens

The HCoT framework addresses a different bottleneck: explicit CoT often improves reasoning but lengthens output sequences and increases latency. HCoT therefore compresses each reasoning chain $A$9 into a single learned vector $S = R \circ A$0 emitted as a special token [CoT]. The downstream content block $S = R \circ A$1 is generated conditioned on $S = R \circ A$2 instead of on the full text of $S = R \circ A$3. The paper states that this yields the same accuracy with much shorter generated sequences and reports 1.5$S = R \circ A$4–3.8$S = R \circ A$5 speedups in wall-clock decoding time (Liu et al., 2024).

Training is split into two stages. Stage 1 trains an auxiliary CoT model $S = R \circ A$6 that, given $S = R \circ A$7, predicts the full reasoning text $S = R \circ A$8 and aligns $S = R \circ A$9 with $r_k$0. The alignment uses a symmetric contrastive objective over normalized mean-pooled hidden state $r_k$1 of the ground-truth $r_k$2 and normalized embedding $r_k$3 of the corresponding [CoT] token, with temperature $r_k$4 and trade-off coefficient $r_k$5. The paper describes the full auxiliary loss as cross-entropy plus $r_k$6 times the contrastive loss. Stage 2 freezes the auxiliary model parameters and fine-tunes an HCoT model $r_k$7 to predict the next content block conditioned on the fixed hidden reasoning representation (Liu et al., 2024).

The reported implementation uses LLaMa2-7B and LLaMa2-13B backbones, BF16, DeepSpeed ZeRO-3, 32$r_k$880GB GPUs, batch size 1 per GPU, gradient accumulation 8, typical fine-tuning learning rates such as $r_k$9, and up to 20 epochs. Inference uses greedy decoding with temperature $f$0 and $f$1. The inference workflow is staged: decode with HCoT until [CoT] is emitted, pause HCoT, run the auxiliary CoT model to produce $f$2, re-inject $f$3 as the [CoT] embedding, and resume generation. If an explicit rationale is needed, the auxiliary model can optionally generate the full $f$4 text after the fact (Liu et al., 2024).

The method is evaluated on mathematical reasoning, question answering, and agent invocation. Datasets include GSM8K, MATH, ScienceQA, and HotpotQA via ReAct. Reported LLaMa2-7B results compare CoT, HCoT, and HCoT-Contrast. On GSM8K, CoT is 36.85, HCoT is 37.15, and HCoT without contrast is 36.47. On MATH, 6.74 rises to 7.49 and then to 8.24 with contrastive learning. On ScienceQA, natural science goes from 80.99 to 83.13 to 84.24; social science goes from 65.80 to 63.89 to 62.77; language science goes from 86.64 to 88.45 to 88.36. On HotpotQA, 83.73 becomes 83.50 and then 84.94. The same paper reports that HCoT reduces the number of generated tokens by 33%–76%, with sequence compression rate $f$5-CR of 23.8%–66.9%, sequence-level speedups $f$6-$f$7–4.2$f$8, and wall-clock speedups $f$9-$F=(f_1,\dots,f_m)$0–2.8$F=(f_1,\dots,f_m)$1 after accounting for auxiliary-model overhead. For LLaMa2-7B, GSM8K has $F=(f_1,\dots,f_m)$2-CR $F=(f_1,\dots,f_m)$3 and $F=(f_1,\dots,f_m)$4-CR $F=(f_1,\dots,f_m)$5, corresponding to $F=(f_1,\dots,f_m)$6-$F=(f_1,\dots,f_m)$7 and $F=(f_1,\dots,f_m)$8-$F=(f_1,\dots,f_m)$9; MATH gives 1.79$f_k \equiv f$0 and 1.60$f_k \equiv f$1; ScienceQA gives 1.49$f_k \equiv f$2 and 1.41$f_k \equiv f$3; and HotpotQA gives 4.21$f_k \equiv f$4 and 2.79$f_k \equiv f$5 (Liu et al., 2024).

5. Empirical regularities across the three paradigms

Viewed jointly, the three papers identify a recurring empirical pattern: reasoning competence can survive, or even become more accessible, when explicit rationale text is suppressed, bypassed, or compressed. In the decoding-based study, the strongest results come from exploring lower-ranked first tokens and selecting paths by a confidence gap, yielding large gains over greedy decoding on GSM8K, MultiArith, year parity, Coin Flip, and Web-of-Lies (Wang et al., 2024).

In the filler-token study, hidden CoT does not degrade final-answer performance on the tested 3SUM task: explicit CoT, hidden CoT, and hidden CoT with recovered tokens all remain at 98.4% final-answer accuracy, while token-level rationale recovery reaches 95.8%. The implication drawn there is not merely that reasoning survives hidden output, but that explicit output is not a faithful window into internal computation (Bharadwaj, 2024).

In the semantic-compression study, compact hidden representations preserve or improve performance in several domains while reducing decoding cost. HCoT-Contrast is reported as top or near-top across most listed domains, and the paper attributes part of this effect to the contrastive objective, noting accuracy drops of approximately 0.6 to 1.4 percentage points when $f_k \equiv f$6 in the ablation labeled “$f_k \equiv f$7 wo Contrast” (Liu et al., 2024).

A common misconception is that CoT gains are exclusively a consequence of explicit rationale prompting. The combined evidence is narrower than that claim. One paper shows that CoT-like trajectories can be surfaced by decoding search alone; another shows that rationale tokens can be hidden behind filler outputs with no loss on the studied benchmark; and a third shows that a learned hidden token can substitute for verbose reasoning text in downstream generation. None of these findings implies that explicit CoT is irrelevant, but together they indicate that explicit rationales are only one interface to underlying reasoning processes.

6. Interpretability, controllability, and unresolved issues

The interpretability implications are strongest in the filler-token setting. The paper explicitly states that hidden CoT shows explicit output is not a faithful window into internal computation and that layer-wise probing suggests a separation of concerns: early layers for “thinking,” later layers for “speaking.” It also notes that the overwrite behavior is reminiscent of induction-head or copy-head circuits, in which useful content is computed and then reformatted to match output targets (Bharadwaj, 2024).

The decoding-based paper contributes a complementary interpretability claim. Because it does not alter prompts or apply instruction tuning, it argues that the resulting CoT paths expose intrinsic reasoning abilities that are otherwise obscured by standard greedy decoding. Its $f_k \equiv f$8 gap functions as a lightweight, unsupervised heuristic for identifying which branch “makes sense,” and the paper suggests possible extensions such as training a small verifier on $f_k \equiv f$9, adaptively re-branching at later steps, or integrating dynamic programming to search for CoT sub-paths (Wang et al., 2024).

The compression-based paper emphasizes controllability and systems-level efficiency. It lists robustness and control over explanation verbosity, the possibility of unhiding full $S_{\mathrm{hidden}} = F \circ A$0 text on demand, and the practical trade-off that two models and a contrastive objective increase training complexity. It also names several open directions: automatically determining when to produce full CoT versus a hidden token, joint training to reduce parameter cost, and alternative compression coding such as multiple tokens for higher speedups. These proposals are prospective rather than established results (Liu et al., 2024).

Across the three strands, the central unresolved issue is not whether hidden reasoning exists in some form, but how faithfully it can be recovered, compressed, selected, or audited at scale. The current evidence spans PaLM-2 Large on reasoning benchmarks, a controlled 34M-parameter transformer on a Match-3 / 3SUM variant, and LLaMa2-7B/13B on mathematical reasoning, ScienceQA, and HotpotQA. This suggests broad relevance, while also indicating that the phenomenon is presently documented through distinct operational definitions rather than a single unified formalism.