Soft Thinking in LLMs

• Soft Thinking is a continuous reasoning paradigm for LLMs, replacing discrete token inference with full probability distributions.
• It improves performance on tasks by preserving uncertainty and exploring diverse reasoning paths using methods like Dirichlet resampling and the Gumbel-Softmax trick.
• Despite its benefits, challenges such as train–inference mismatch and single-threaded decoding persist, driving further research into stochastic soft reasoning.

Soft Thinking refers to a family of reasoning paradigms for LLMs that replace traditional discrete token-level inference with operations in a continuous concept space. Unlike standard Chain-of-Thought (CoT) reasoning, which samples or selects a single token at each step, Soft Thinking leverages the model’s full token probability distributions to synthesize intermediate “concept tokens,” represented as convex combinations in the embedding space. This continuous formulation enables preservation of uncertainty, richer semantic representation, and—when appropriately randomized—diverse exploration of reasoning paths. Key methods, limitations, empirical results, and ongoing research directions are delineated below.

1. Foundations: Soft Concept Tokens and Continuous Reasoning

In standard LLM decoding, each token generation step selects a discrete index $t_i$ from the vocabulary $V$, embedding it via a one-hot lookup $e(t_i)\in\mathbb{R}^d$. Soft Thinking generalizes this by forming a probability distribution $p_i\in\Delta^{|V|-1}$ over $V$ at each reasoning step—termed a “concept token.” The soft concept embedding,

$c_i = \sum_{k=1}^{|V|} p_i[k]\,e(k),$

resides in the continuous concept space

$$\mathcal{C} = \left\{ c = \sum_{k=1}^{|V|} \alpha_k e(k) : \alpha \in \Delta^{|V|-1} \right\} \subset \mathbb{R}^d.$$

This embedding preserves the full distributional semantics of potential next steps, allowing the model to operate in a high-dimensional, abstract representation space that smoothly interpolates between meanings encoded by discrete tokens. The full soft probability vector, rather than a sampled token, is used to generate the next step, and the eventual answer can revert to discrete decoding. This approach is training-free at inference: no model weights or architecture modifications are required, and the original embedding matrix $E$ is reused (Zhang et al., 21 May 2025).

2. Theoretical and Algorithmic Distinctions

Soft Thinking retains the two-stage flow of CoT reasoning—iterative "thinking" followed by final answer generation—but with crucial modifications. In the soft phase, at each step $i$, the LLM predicts a full token distribution $p_i$. Efficiency is achieved by filtering to the top-$n$ tokens and computing the corresponding renormalized distribution and embedding. A "Cold Stop" heuristic monitors the entropy of $p_i$ to prevent out-of-distribution (OOD) collapse by terminating the soft reasoning loop if the entropy remains low for $k$ consecutive steps.

Comparison Table: Discrete CoT vs. Soft Thinking

Property	Discrete CoT	Soft Thinking
Token selection	One token per step	Probability-weighted mixture
Intermediate state	One-hot embedding	Continuous convex combination
Path diversity	Sampled at sequence	Encoded in probability distribution
Downstream input	Token embedding	Soft concept embedding

Table: Core distinctions between standard token-based Chain-of-Thought and Soft Thinking. Both retain a sequence of steps, but Soft Thinking passes continuous-valued embeddings between steps (Zhang et al., 21 May 2025, Zheng et al., 9 Nov 2025).

3. Empirical Results, Benchmarking, and Efficiency

Soft Thinking has demonstrated measurable improvements on mathematical and code reasoning benchmarks. Across datasets such as Math500, AIME 2024, GSM8K, GPQA-Diamond, HumanEval, MBPP, and LiveCodeBench, Soft Thinking produced higher Pass@1 accuracy and reduced token usage compared to standard CoT. For instance, on QwQ-32B (math tasks), Soft Thinking raised Pass@1 from 83.84% to 86.32% and reduced token count by 11.6%; on DeepSeek-Qwen-32B (code tasks), it improved accuracy by 0.90 points and cut reasoning tokens by 19.1% (Zhang et al., 21 May 2025).

Additional qualitative analyses revealed that Soft Thinking generates more concise and interpretable solution traces. Early reasoning steps exhibit exploratory distributions, with final steps collapsing to one-hot predictions as uncertainty resolves. These distributional patterns reflect path diversity and improved error mitigation in early “fork” points.

4. Limitations: Train–Inference Mismatch and Single-Threaded Reasoning

A central limitation of vanilla Soft Thinking is its train–inference mismatch: LLM weights are optimized for sequences of discrete tokens, but Soft Thinking supplies continuous mixtures at test time. This can result in OOD behavior and unstable outputs, especially for long reasoning chains. Although Cold Stop reduces severe failures, it does not fully resolve this source of brittleness (Wang et al., 21 Nov 2025).

Crucially, empirical probing techniques such as Jensen–Shannon divergence comparisons, logit-lens analysis, and ROUGE-L overlap demonstrate that when presented with a soft concept embedding, autoregressive LLMs overwhelmingly follow the “dominant” (top-1) component, effectively reducing soft reasoning to greedy single-path decoding. As a result, latent parallelism is not realized in practice unless additional randomness is introduced (Wu et al., 5 Aug 2025).

5. Overcoming Greedy Pitfalls: Stochastic Soft Reasoning and RL Fine-Tuning

To recover the theoretical benefits of path diversity, recent work explores explicit injection of randomness into the soft tokens at each intermediate step. Strategies include:

• Dirichlet Resampling: Given a token distribution $p$, sample new distributions from Dirichlet($\gamma\cdot p$), with concentration parameter $\gamma$. Lower $\gamma$ increases randomness.
• Gumbel-Softmax Trick: Inject Gumbel noise into log-probabilities followed by a softmax at temperature $\tau$, yielding a continuous, randomly selected soft token. This technique provides tunable, controlled stochasticity, reliably outperforming both vanilla Soft Thinking and discrete sampling on math, QA, and code benchmarks by 0.5–2.5 points in Pass@1 or Avg@8 (Wu et al., 5 Aug 2025, Zheng et al., 9 Nov 2025).

RL fine-tuning approaches, such as Soft Concept Mixing (SCM) and SofT-GRPO, further mitigate the train–test gap and support robust stochastic soft reasoning. SCM introduces soft concept vectors during RL-based fine-tuning and fuses these into model hidden states, yielding stable improvements of 0.3–1.3 accuracy points over strong RL-CoT baselines and outperforming other latent reasoning methods such as Coconut and HRPO (Wang et al., 21 Nov 2025). SofT-GRPO innovates with Gumbel-Softmax sampling and reparameterization trick within the Group Relative Policy Optimization (GRPO) framework, achieving not only modest Pass@1 gains but substantial uplifts in Pass@32 (up to +2.19% over discrete RL baselines), especially under majority-voting aggregation (Zheng et al., 9 Nov 2025).

6. Advances in Latent-Space Reasoning and Test-Time Scaling

Beyond token-level soft concepts, related paradigms such as SoftCoT and its extension SoftCoT++ move reasoning entirely into the latent space by using a learned assistant and a projection network to construct high-level “soft thought” vectors. SoftCoT++ addresses the limits of fixed-latent decoding by generating multiple diverging latent reasoning chains with distinct initial tokens and contrastive training objectives to maximize latent diversity. At inference, majority-voting over multiple diverse latent chains yields further performance gains and is fully orthogonal to discrete token sampling. For example, on five reasoning benchmarks and two LLM backbones, SoftCoT++ consistently outperformed single-latent or self-consistency baselines (e.g. LLaMA-3.1-8B, GSM8K: SoftCoT-SC 90.63±0.39 vs. SoftCoT++ 90.99±0.25) (Xu et al., 16 May 2025).

7. Open Questions and Future Research Directions

Critical open challenges include:

• True Parallelism: Off-the-shelf LLMs remain single-threaded even under soft input, following only the most probable token chain unless randomness is explicitly injected or models are retrained for multi-path exploration (Wu et al., 5 Aug 2025).
• Train–Test Alignment: Robust soft reasoning under long chains or out-of-domain prompts will likely require dedicated pre-training or fine-tuning on soft concept sequences, as realized in SCM and SofT-GRPO (Wang et al., 21 Nov 2025, Zheng et al., 9 Nov 2025).
• Hierarchical and Structured Concept Spaces: Further generalizations may embed phrases, sentences, or hierarchical concepts as continuous entities, extending beyond token-level mixtures (Zhang et al., 21 May 2025).
• Combinatorial Scaling: Combining latent-path diversity (e.g., SoftCoT++), token-level self-consistency, and stochastic sampling offers additive gains and invites more exotic architectures for parallel reasoning (Xu et al., 16 May 2025).
• Theoretical Analysis: Understanding the bounds and optimality of linearization in continuous space, as well as the impact of randomization and contrastive diversity mechanisms, remains a fertile area for foundational paper.

In summary, Soft Thinking frameworks represent a class of continuous, distributional reasoning strategies that augment or surpass discrete token-level chain-of-thought by leveraging the LLM’s full output uncertainty. Continued methodological and empirical advances along these lines are poised to reshape best practices in LLM-based automated reasoning across scientific, mathematical, and real-world decision-making domains (Zhang et al., 21 May 2025, Wang et al., 21 Nov 2025, Zheng et al., 9 Nov 2025, Wu et al., 5 Aug 2025, Xu et al., 16 May 2025).