SoftCoT: Soft Chain-of-Thought Reasoning

• SoftCoT is a framework that shifts part of LLM reasoning into a continuous latent space while preserving explicit textual rationales.
• It employs a frozen assistant to generate latent thought tokens, which are aligned via a small, trainable projection module to the backbone LLM.
• Empirical results demonstrate improved performance over zero-shot CoT on various benchmarks with average gains of up to 4.8% in math, commonsense, and symbolic tasks.

SoftCoT, short for “Soft Chain-of-Thought,” is a framework for improving LLM reasoning by shifting part of the reasoning process from discrete token generation into a continuous latent space while leaving the backbone LLM unchanged. A lightweight frozen assistant model generates instance-specific soft thought tokens in its hidden space; a small trainable projection maps those vectors into the backbone LLM’s representation space; and the backbone then produces natural-language rationales and answers conditioned on these continuous prefix tokens. In this formulation, the observed reasoning trace remains textual, but its trajectory is steered by latent, differentiable, instance-adaptive prompts rather than solely by hard token decoding (Xu et al., 17 Feb 2025).

1. Problem setting and conceptual motivation

SoftCoT is motivated by limitations of standard chain-of-thought prompting in which a model emits explicit rationale tokens before the final answer. Given a tokenized question $\mathcal Q = [q_1,\dots,q_{|\mathcal Q|}]$, conventional “Hard-CoT” produces rationale tokens $\mathcal R = [r_1,\dots,r_{|\mathcal R|}]$ and answer tokens $\mathcal A = [a_1,\dots,a_{|\mathcal A|}]$ through autoregressive decoding:

$$r_{i+1} = \mathrm{LLM}(\mathcal Q;\mathcal R_{\le i}), \qquad a_{j+1} = \mathrm{LLM}(\mathcal Q;\mathcal R;\mathcal A_{\le j}).$$

This imposes a discrete vocabulary bottleneck, requires long token sequences for reliable performance, is vulnerable to error propagation across rationale steps, and interrupts end-to-end differentiability because the pipeline passes through logits, sampled tokens, and re-embedding rather than a continuous latent channel (Xu et al., 17 Feb 2025).

The underlying claim is not that textual reasoning is useless, but that natural language is a coarse interface optimized for human interpretability rather than necessarily for internal computational efficiency. The survey literature places SoftCoT within “latent representation compression,” alongside Coconut, CCoT, Heima, and related methods that attempt to compress or replace explicit textual reasoning with hidden-state computation in continuous space (Sui et al., 20 Mar 2025).

SoftCoT also responds to a specific failure mode of prior continuous-space reasoning methods. Coconut and CCoT move reasoning into latent space, but the original SoftCoT paper argues that they rely on full-model fine-tuning with language-modeling objectives and can suffer catastrophic forgetting on strong instruction-tuned backbones. The reported example is LLaMA-3.1-8B, where LM-based fine-tuning as used in Coconut and CCoT hurts performance relative to zero-shot CoT, making a frozen-backbone alternative attractive (Xu et al., 17 Feb 2025).

2. Architecture and representation flow

SoftCoT consists of three components: a frozen assistant model, a trainable projection module, and a frozen backbone LLM. The assistant is a smaller LLM, such as LLaMA-3.2-1B-Instruct in the LLaMA-based experiments, and its only role is to generate latent thought vectors for each instance. Its input is

$$\mathbf{x}_{\mathrm{assist}} = [\mathcal I, \mathcal Q, [\mathrm{UNK}]_{1:N}],$$

where $\mathcal I$ is a task-specific instructional context, $\mathcal Q$ is the question, and $[\mathrm{UNK}]_{1:N}$ are placeholder tokens reserved to host the soft thoughts. After the assistant processes this sequence, the final-layer hidden states at the placeholder positions are extracted as

$$\mathbf t_{\mathrm{assist}} \in \mathbb R^{N \times d_{\mathrm{assist}}}.$$

These vectors are not decoded through a softmax and are never converted into vocabulary tokens (Xu et al., 17 Feb 2025).

Because the assistant and the backbone live in different representation spaces, SoftCoT introduces a tokenwise linear projection

$$\mathbf t_{\text{soft}} = \mathrm{Linear}_\theta(\mathbf t_{\mathrm{assist}}),$$

with

$\mathcal R = [r_1,\dots,r_{|\mathcal R|}]$0

where $\mathcal R = [r_1,\dots,r_{|\mathcal R|}]$1 and $\mathcal R = [r_1,\dots,r_{|\mathcal R|}]$2. This projection is the only trainable module in the original method. The backbone LLM then receives

$\mathcal R = [r_1,\dots,r_{|\mathcal R|}]$3

where $\mathcal R = [r_1,\dots,r_{|\mathcal R|}]$4 is a fixed task-specific instruction prompt and $\mathcal R = [r_1,\dots,r_{|\mathcal R|}]$5 functions as a sequence of continuous prefix tokens. These vectors participate in attention exactly as additional input positions, allowing the frozen backbone to query them throughout rationale and answer generation (Xu et al., 17 Feb 2025).

The resulting design is a parameter-efficient wrapper rather than a modification of the backbone. In the scale described in the paper, if $\mathcal R = [r_1,\dots,r_{|\mathcal R|}]$6 and $\mathcal R = [r_1,\dots,r_{|\mathcal R|}]$7, the linear projection has about $\mathcal R = [r_1,\dots,r_{|\mathcal R|}]$8 million parameters plus biases, which is orders of magnitude smaller than a 7B–8B backbone. No LoRA or other PEFT method is applied to the backbone in the original system (Xu et al., 17 Feb 2025).

3. Training objective and inference behavior

SoftCoT targets tasks with annotated chain-of-thought rationales $\mathcal R = [r_1,\dots,r_{|\mathcal R|}]$9 and answers $\mathcal A = [a_1,\dots,a_{|\mathcal A|}]$0. During training, the backbone input is

$\mathcal A = [a_1,\dots,a_{|\mathcal A|}]$1

The backbone and assistant remain frozen. Only the projection parameters $\mathcal A = [a_1,\dots,a_{|\mathcal A|}]$2 are optimized with standard next-token prediction loss on the rationale and answer tokens, while the task prompt, question, and soft thought tokens are masked out of the loss:

$\mathcal A = [a_1,\dots,a_{|\mathcal A|}]$3

where $\mathcal A = [a_1,\dots,a_{|\mathcal A|}]$4. Gradients therefore flow through the projected soft thoughts into the projection layer, but not into either frozen LLM (Xu et al., 17 Feb 2025).

At inference time, the assistant first generates all soft thought tokens in a single forward pass; the backbone then autoregressively generates a natural-language rationale $\mathcal A = [a_1,\dots,a_{|\mathcal A|}]$5 and answer $\mathcal A = [a_1,\dots,a_{|\mathcal A|}]$6. The final answer $\mathcal A = [a_1,\dots,a_{|\mathcal A|}]$7 is extracted from $\mathcal A = [a_1,\dots,a_{|\mathcal A|}]$8 by a hand-crafted rule-based extractor $\mathcal A = [a_1,\dots,a_{|\mathcal A|}]$9, such as a regular expression for boxed math answers. The paper explicitly notes that “speculatively” does not mean speculative decoding for faster token generation; it refers only to the assistant generating the soft thought tokens before backbone decoding (Xu et al., 17 Feb 2025).

This training regime makes SoftCoT a form of soft prompt tuning, but with a distinctive source of prompts: the prompt vectors are not directly optimized free parameters, nor are they textual prompts, but instance-specific hidden states generated by a separate frozen assistant and aligned by a learned projection. A plausible implication is that SoftCoT combines two desirable properties that are usually separated in the literature: interpretability of explicit rationales at output time and differentiability of the upstream reasoning scaffold.

4. Benchmarks and empirical results

The original study trains and evaluates SoftCoT on five benchmarks: GSM8K, ASDiv-Aug, AQuA, StrategyQA, and Date Understanding from BIG-Bench. The first three are math-focused; StrategyQA targets commonsense reasoning; and Date Understanding targets symbolic reasoning. For ASDiv-Aug, the paper states that ASDiv is augmented by replicating each instance 5 times and randomly resampling numerical values to create novel problems, with the stated goal of testing true reasoning rather than memorization (Xu et al., 17 Feb 2025).

Across both evaluated backbones, SoftCoT improves over zero-shot CoT and over a harder-prompting baseline that appends assistant-generated text. The average accuracies reported in the main tables are as follows:

Backbone	Zero-Shot CoT Avg.	SoftCoT Avg.
LLaMA-3.1-8B-Instruct	68.21	70.52
Qwen2.5-7B-Instruct	70.29	75.06

For LLaMA-3.1-8B-Instruct, SoftCoT reaches 81.03 on GSM8K, 87.19 on ASDiv-Aug, 56.30 on AQuA, 69.04 on StrategyQA, and 59.04 on Date Understanding, with a reported average gain of about 2.3 points over zero-shot CoT. For Qwen2.5-7B-Instruct, the gains are larger: 85.81 on GSM8K, 88.90 on ASDiv-Aug, 72.44 on AQuA, 60.61 on StrategyQA, and 67.52 on Date Understanding, corresponding to an average increase of about 4.8 points over zero-shot CoT. On the LLaMA backbone, Coconut underperforms zero-shot CoT, which the paper interprets as an instance of catastrophic forgetting under full-model LM fine-tuning (Xu et al., 17 Feb 2025).

The ablation studies emphasize compactness and complementarity. On ASDiv-Aug with LLaMA-3.1-8B, accuracy peaks around $$r_{i+1} = \mathrm{LLM}(\mathcal Q;\mathcal R_{\le i}), \qquad a_{j+1} = \mathrm{LLM}(\mathcal Q;\mathcal R;\mathcal A_{\le j}).$$0 soft thought tokens at about 87.19%, whereas the hard assistant-prompt baseline requires about 24 textual tokens to reach similar performance. On GSM8K with self-consistency, SoftCoT improves both the single-chain and 10-chain settings, rising from 79.61 to 81.03 at $$r_{i+1} = \mathrm{LLM}(\mathcal Q;\mathcal R_{\le i}), \qquad a_{j+1} = \mathrm{LLM}(\mathcal Q;\mathcal R;\mathcal A_{\le j}).$$1 and from 90.37 to 90.98 at $$r_{i+1} = \mathrm{LLM}(\mathcal Q;\mathcal R_{\le i}), \qquad a_{j+1} = \mathrm{LLM}(\mathcal Q;\mathcal R;\mathcal A_{\le j}).$$2. The paper also reports that merely adding [UNK] tokens as thought markers slightly improves performance and reduces variance, likely acting as “pause tokens,” but SoftCoT remains the strongest configuration (Xu et al., 17 Feb 2025).

5. Relation to adjacent methods and later variants

Relative to standard CoT prompting, SoftCoT introduces learned, instance-specific continuous prompts while preserving explicit natural-language rationale generation. Relative to Coconut and CCoT, it avoids full-model fine-tuning by freezing the backbone and generating latent thoughts with a separate assistant. Relative to Heima, which compresses reasoning into a single continuous vector for multimodal tasks, the original SoftCoT paper presents SoftCoT as a backbone-agnostic, parameter-efficient wrapper for strong instruction-tuned LLMs (Xu et al., 17 Feb 2025).

The immediate extension of the method is “SoftCoT++,” which reframes SoftCoT as a test-time scaling method in continuous space. The SoftCoT++ paper argues that the original SoftCoT produces a single deterministic latent thought per input, so diverse exploration remains limited even if one applies self-consistency in the discrete reasoning stage. SoftCoT++ addresses this by introducing multiple specialized initial tokens $$r_{i+1} = \mathrm{LLM}(\mathcal Q;\mathcal R_{\le i}), \qquad a_{j+1} = \mathrm{LLM}(\mathcal Q;\mathcal R;\mathcal A_{\le j}).$$3 for the assistant and a contrastive regularizer that promotes diversity among the resulting soft thought representations. In the reported experiments, SoftCoT++ improves over SoftCoT-SC and conventional zero-shot CoT with self-consistency across GSM8K, ASDiv-Aug, AQuA, StrategyQA, and Date Understanding on both LLaMA-3.1-8B-Instruct and Qwen3-8B (Xu et al., 16 May 2025).

A later development, “LTA-Thinker,” positions itself as continuing the SoftCoT and SoftCoT++ paradigm while redesigning the latent thought generator and its optimization objective. Instead of a pretrained small LLM assistant, it uses a randomly initialized lightweight Transformer block as a learnable prior, and it adds Semantic Alignment Loss based on KL divergence plus Reasoning Focus Loss based on contrastive learning. The stated aim is to obtain higher-variance yet directionally constrained latent thought distributions and better test-time scaling efficiency (Wang et al., 16 Sep 2025).

SoftCoT also influenced adjacent work on chain-of-thought compression. “Upfront Chain-of-Thought” (UCoT) uses a small compressor to produce an “Upfront Thought” embedding and a large executor to generate a shorter explicit CoT and final answer. That paper explicitly cites SoftCoT as evidence that aligned continuous representations alone are not sufficient to reduce CoT length unless the model is also trained to connect the latent representation to correctness and compression objectives (Li et al., 9 Oct 2025).

6. Interpretability, limitations, and significance

SoftCoT preserves explicit textual rationales, but the soft thought tokens themselves are latent and not directly interpretable. The appendix examples discussed in the original paper show that the backbone still emits ordinary step-by-step reasoning, while the assistant’s contribution remains hidden in continuous form. The authors do not report PCA, clustering, or probing analyses of the latent tokens, so their internal semantics remain opaque. This suggests that SoftCoT improves reasoning behaviorally without yet providing a transparent account of what information the soft thoughts encode (Xu et al., 17 Feb 2025).

Several recurring misconceptions are explicitly addressed by the paper. SoftCoT does not replace textual reasoning with a purely latent answer generator; the backbone still decodes a natural-language rationale and answer. It is not speculative decoding in the standard systems sense; the assistant’s single-pass thought generation is called “speculative” only in the loose sense that it prepares candidate continuous thoughts before backbone decoding. It is also not a method for modifying or fine-tuning the backbone’s core parameters: the original formulation freezes both assistant and backbone and trains only the projection module (Xu et al., 17 Feb 2025).

The stated limitations are likewise specific. The original experiments cover LLaMA-3.1-8B and Qwen2.5-7B, so scalability to substantially larger backbones remains open. The method depends on a fixed assistant model that must produce useful hidden states; if the assistant is too weak or misaligned, the projection may learn to ignore it. Domain generalization beyond the evaluated math, commonsense, and symbolic tasks is not established, although zero-shot transfer on Date Understanding is reported as encouraging. The paper also emphasizes that SoftCoT does not remove the need for explicit reasoning paths: soft thoughts enrich the model’s internal probability space, but chain-of-thought decoding remains the main search mechanism for both reasoning quality and interpretability (Xu et al., 17 Feb 2025).

Within the broader efficient-reasoning literature, SoftCoT occupies a distinctive middle position. The survey on efficient reasoning classifies it under latent representation compression rather than prompt-only brevity control or RL-based length shaping, and this placement is apt: SoftCoT neither simply asks a model to “be concise” nor trains it with an explicit length reward. Instead, it supplies a compact, learned latent scaffold that can reduce reliance on long or multiple textual thought sequences while keeping the externally visible reasoning process intact (Sui et al., 20 Mar 2025).