---
title: Soft Thinking in LLMs
url: https://www.emergentmind.com/topics/soft-thinking
type: topic
---

# Soft Thinking in LLMs

Soft Thinking refers to a family of reasoning paradigms for large language models (LLMs) that replace traditional discrete token-level inference with operations in a continuous concept space. Unlike standard Chain-of-Thought (CoT) reasoning, which samples or selects a single token at each step, Soft Thinking leverages the model’s full token probability distributions to synthesize intermediate “concept tokens,” represented as convex combinations in the embedding space. This continuous formulation enables preservation of uncertainty, richer semantic representation, and—when appropriately randomized—diverse exploration of reasoning paths. Key methods, limitations, empirical results, and ongoing research directions are delineated below.

## 1. Foundations: Soft Concept Tokens and Continuous Reasoning

In standard LLM decoding, each token generation step selects a discrete index $t_i$ from the vocabulary $V$, embedding it via a one-hot lookup $e(t_i)\in\mathbb{R}^d$. Soft Thinking generalizes this by forming a probability distribution $p_i\in\Delta^{|V|-1}$ over $V$ at each reasoning step—termed a “concept token.” The soft concept embedding,
$$
c_i = \sum_{k=1}^{|V|} p_i[k]\,e(k),
$$
resides in the continuous concept space
$$
\mathcal{C} = \left\{ c = \sum_{k=1}^{|V|} \alpha_k e(k) : \alpha \in \Delta^{|V|-1} \right\} \subset \mathbb{R}^d.
$$

This embedding preserves the full distributional semantics of potential next steps, allowing the model to operate in a high-dimensional, abstract representation space that smoothly interpolates between meanings encoded by discrete tokens. The full soft probability vector, rather than a sampled token, is used to generate the next step, and the eventual answer can revert to discrete decoding. This approach is training-free at inference: no model weights or architecture modifications are required, and the original embedding matrix $E$ is reused [2505.15778].

## 2. Theoretical and Algorithmic Distinctions

Soft Thinking retains the two-stage flow of CoT reasoning—iterative "thinking" followed by final answer generation—but with crucial modifications. In the soft phase, at each step $i$, the LLM predicts a full token distribution $p_i$. Efficiency is achieved by filtering to the top-$n$ tokens and computing the corresponding renormalized distribution and embedding. A "Cold Stop" heuristic monitors the entropy of $p_i$ to prevent out-of-distribution (OOD) collapse by terminating the soft reasoning loop if the entropy remains low for $k$ consecutive steps.

Comparison Table: Discrete CoT vs. Soft Thinking

| Property             | Discrete CoT            | Soft Thinking                         |
|----------------------|------------------------|---------------------------------------|
| Token selection      | One token per step     | Probability-weighted mixture          |
| Intermediate state   | One-hot embedding      | Continuous convex combination         |
| Path diversity       | Sampled at sequence    | Encoded in probability distribution   |
| Downstream input     | Token embedding        | Soft concept embedding                |

*Table: Core distinctions between standard token-based Chain-of-Thought and Soft Thinking. Both retain a sequence of steps, but Soft Thinking passes continuous-valued embeddings between steps* [2505.15778, 2511.06411].

## 3. Empirical Results, Benchmarking, and Efficiency

Soft Thinking has demonstrated measurable improvements on mathematical and code reasoning benchmarks. Across datasets such as Math500, AIME 2024, GSM8K, GPQA-Diamond, HumanEval, MBPP, and LiveCodeBench, Soft Thinking produced higher Pass@1 accuracy and reduced token usage compared to standard CoT. For instance, on QwQ-32B (math tasks), Soft Thinking raised Pass@1 from 83.84% to 86.32% and reduced token count by 11.6%; on DeepSeek-Qwen-32B (code tasks), it improved accuracy by 0.90 points and cut reasoning tokens by 19.1% [2505.15778].

Additional qualitative analyses revealed that Soft Thinking generates more concise and interpretable solution traces. Early reasoning steps exhibit exploratory distributions, with final steps collapsing to one-hot predictions as uncertainty resolves. These distributional patterns reflect path diversity and improved error mitigation in early “fork” points.

## 4. Limitations: Train–Inference Mismatch and Single-Threaded Reasoning

A central limitation of vanilla Soft Thinking is its train–inference mismatch: LLM weights are optimized for sequences of discrete tokens, but Soft Thinking supplies continuous mixtures at test time. This can result in OOD behavior and unstable outputs, especially for long reasoning chains. Although Cold Stop reduces severe failures, it does not fully resolve this source of brittleness [2511.16885].

Crucially, empirical probing techniques such as Jensen–Shannon divergence comparisons, logit-lens analysis, and ROUGE-L overlap demonstrate that when presented with a soft concept embedding, autoregressive LLMs overwhelmingly follow the “dominant” (top-1) component, effectively reducing soft reasoning to greedy single-path decoding. As a result, latent parallelism is not realized in practice unless additional randomness is introduced [2508.03440].

## 5. Overcoming Greedy Pitfalls: Stochastic Soft Reasoning and RL Fine-Tuning

To recover the theoretical benefits of path diversity, recent work explores explicit injection of randomness into the soft tokens at each intermediate step. Strategies include:

- **Dirichlet Resampling**: Given a token distribution $p$, sample new distributions from Dirichlet($\gamma\cdot p$), with concentration parameter $\gamma$. Lower $\gamma$ increases randomness.
- **Gumbel-Softmax Trick**: Inject Gumbel noise into log-probabilities followed by a softmax at temperature $\tau$, yielding a continuous, randomly selected soft token. This technique provides tunable, controlled stochasticity, reliably outperforming both vanilla Soft Thinking and discrete sampling on math, QA, and code benchmarks by 0.5–2.5 points in Pass@1 or Avg@8 [2508.03440, 2511.06411].

RL fine-tuning approaches, such as Soft Concept Mixing (SCM) and SofT-GRPO, further mitigate the train–test gap and support robust stochastic soft reasoning. SCM introduces soft concept vectors during RL-based fine-tuning and fuses these into model hidden states, yielding stable improvements of 0.3–1.3 accuracy points over strong RL-CoT baselines and outperforming other latent reasoning methods such as Coconut and HRPO [2511.16885]. SofT-GRPO innovates with Gumbel-Softmax sampling and reparameterization trick within the Group Relative Policy Optimization (GRPO) framework, achieving not only modest Pass@1 gains but substantial uplifts in Pass@32 (up to +2.19% over discrete RL baselines), especially under majority-voting aggregation [2511.06411].

## 6. Advances in Latent-Space Reasoning and Test-Time Scaling

Beyond token-level soft concepts, related paradigms such as SoftCoT and its extension SoftCoT++ move reasoning entirely into the latent space by using a learned assistant and a projection network to construct high-level “soft thought” vectors. SoftCoT++ addresses the limits of fixed-latent decoding by generating multiple diverging latent reasoning chains with distinct initial tokens and contrastive training objectives to maximize latent diversity. At inference, majority-voting over multiple diverse latent chains yields further performance gains and is fully orthogonal to discrete token sampling. For example, on five reasoning benchmarks and two LLM backbones, SoftCoT++ consistently outperformed single-latent or self-consistency baselines (e.g. LLaMA-3.1-8B, GSM8K: SoftCoT-SC 90.63±0.39 vs. SoftCoT++ 90.99±0.25) [2505.11484].

## 7. Open Questions and Future Research Directions

Critical open challenges include:

- **True Parallelism**: Off-the-shelf LLMs remain single-threaded even under soft input, following only the most probable token chain unless randomness is explicitly injected or models are retrained for multi-path exploration [2508.03440].
- **Train–Test Alignment**: Robust soft reasoning under long chains or out-of-domain prompts will likely require dedicated pre-training or fine-tuning on soft concept sequences, as realized in SCM and SofT-GRPO [2511.16885, 2511.06411].
- **Hierarchical and Structured Concept Spaces**: Further generalizations may embed phrases, sentences, or hierarchical concepts as continuous entities, extending beyond token-level mixtures [2505.15778].
- **Combinatorial Scaling**: Combining latent-path diversity (e.g., SoftCoT++), token-level self-consistency, and stochastic sampling offers additive gains and invites more exotic architectures for parallel reasoning [2505.11484].
- **Theoretical Analysis**: Understanding the bounds and optimality of linearization in continuous space, as well as the impact of randomization and contrastive diversity mechanisms, remains a fertile area for foundational study.

In summary, Soft Thinking frameworks represent a class of continuous, distributional reasoning strategies that augment or surpass discrete token-level chain-of-thought by leveraging the LLM’s full output uncertainty. Continued methodological and empirical advances along these lines are poised to reshape best practices in LLM-based automated reasoning across scientific, mathematical, and real-world decision-making domains [2505.15778, 2511.16885, 2511.06411, 2508.03440, 2505.11484].

Source: https://www.emergentmind.com/topics/soft-thinking