---
title: Learned Continue-Thinking Tokens
url: https://www.emergentmind.com/topics/learned-continue-thinking-tokens
type: topic
---

# Learned Continue-Thinking Tokens

Learned continue-thinking tokens are specialized symbols—discrete or continuous, explicit or latent—incorporated into large language model (LLM) sequences to trigger, extend, or compress the model’s internal reasoning between observed or generated content. These mechanisms are designed to allocate extra compute, enhance gradient flow, stabilize information dynamics, or adaptively focus on the most salient steps for complex reasoning tasks. Unlike naive repetition or static tokens, the learned variants are algorithmically or empirically associated with information-theoretic markers (such as mutual information peaks), optimized embedding representations, or explicit training signals, yielding direct improvements in accuracy, compute efficiency, or reasoning faithfulness compared to baseline approaches.

## 1. Formal Definitions and Construction of Continue-Thinking Tokens

Continue-thinking tokens manifest in multiple concrete forms:

- **Discrete thinking tokens**: In Herel & Mikolov's “Thinking Tokens” framework, a special symbol (notation: ‹T›) is inserted after each word $w_i$ in the input sequence, producing a new sequence $x' = [w_1,\langle T\rangle, w_2, \langle T\rangle, ..., w_L, \langle T\rangle]$, where $N$ copies of ‹T› can be interleaved per word [2405.08644]. These tokens serve purely as “think time”—no loss is backpropagated through them.
- **Information-theoretic peaks**: A distinct family emerges from tracking mutual information between the model's intermediate states $h_t$ and the gold answer's representation $h_y$, flagging “MI peaks” as the locus of true reflection; the associated natural-language tokens (e.g., "Hmm", "Therefore") are then selected as "thinking tokens" [2506.02867].
- **Planning or latent reasoning tokens**: In supervised or RL frameworks (e.g., SFT, DPO, GRPO), explicit markers such as `<think>...</think>` designate reasoning regions of the sequence. The policy may explicitly allocate probability mass for “thinking” versus “answer” modes [2510.16340]. 
- **Continuous—multiplex or soft—tokens**: Embeddings that represent mixtures of multiple possible next steps (e.g., via the multiplex merge of several candidate token embeddings at each step [2601.08808]), or softmax-weighted linear combinations with noise (the “soft tokens” of [2509.19170]), introduce more expressivity and stochasticity.
- **Learned embeddings for specialized use**: Single tokens (e.g., `<|continue-thinking|>`) added to the vocabulary and optimized for the forward pass while the backbone remains frozen [2506.11274].

Mechanistically, all continue-thinking tokens are unified by their role as either explicit or implicit triggers for extra computation, latent information processing, or controlled extension of the CoT (chain-of-thought) trajectory.

## 2. Training Protocols and Optimization Objectives

The methodology for learning or leveraging continue-thinking tokens varies significantly by framework:

- **Supervised augmentation:** Direct masking and loss omission for thinking tokens (delta mask $\delta_t$ in the objective $L(\theta) = -\sum_{t=1}^{T'} \delta_t \log P_\theta(y'_t | y'_{<t})$) [2405.08644].
- **Information-theoretic selection:** Thinking tokens are mined post hoc from generative runs of a base LLM by labeling tokens at MI peaks via Hilbert–Schmidt Independence Criterion (HSIC) applied to $(h_t, h_y)$ pairs [2506.02867].
- **Conditional token selection (compression):** Each reasoning token $y_i$ is assigned an importance score $r_i = \mathrm{PPL}_{\rm unc}(y_i) - \mathrm{PPL}_{\rm cond}(y_i)$ (measuring its drop in perplexity once the final answer is known), and only tokens above a quantile threshold are preserved for training, enforcing an adaptive compression ratio [2505.17827].
- **Reinforcement learning of embeddings:** In the RL-based approach, only the embedding $\theta_T$ for `<|continue-thinking|>` is updated through policy gradients; the reward is a function of answer correctness and required format, and generation is forced to substitute `<|continue-thinking|>` for the end-of-think token up to a preset budget [2506.11274].
- **Latent continuous token RL:** In “soft token” methods, the entire CoT phase is trained by policy gradient with softmax-weighted or multiplexed token embedding mixtures at each step; exploration is implemented via Gaussian noise injection into embedding space, and correctness signals guide reward [2509.19170, 2601.08808].

The joint effect is optimization for improved chain-of-thought expressivity, test-time scaling, exploration-exploitation balance, or data efficiency.

## 3. Architectural Integration and Inference Workflows

No fundamental architectural change is required for incorporating continue-thinking tokens:

- **Vanilla RNN/Transformer:** For explicit tokens (e.g., ‹T›), integration into RNN or Transformer models only requires insertion into the vocabulary and context window, and loss-masking during training [2405.08644].
- **Modular planning slots:** Chain-of-thought markers or reasoning regions (e.g., `<think>...<\think>`) are implemented as special tokens delimiting policy subspaces in SFT/DPO/GRPO [2510.16340].
- **Latent reasoning blocks:** The “Thinking States” method augments a backbone model with lightweight, unidirectional blocks for reasoning state generation, compression, and recurrent embedding update, supporting full parallelization via teacher-forcing [2602.08332].
- **Multiplex continuous tokens:** Token-wise branch-merge mechanisms via sampling $K$ candidates and merging their embeddings function as a drop-in to standard Transformer embedding layers, requiring only sequence-length increases to accommodate multiplex tokens [2601.08808].
- **Conditional token pruning/compression:** At each inference step, conditional importance thresholds determine which tokens are retained or suppressed on the fly, reducing unnecessary computation [2505.17827].
- **Budget-forcing at test time:** Fixed or learned continue-thinking tokens are injected explicitly during inference to extend CoT length and elicit further reasoning [2506.11274, 2506.02867].

This flexibility enables application across standard, distilled, and RL-optimized LLMs, with adjustments focused mainly on data augmentation, pre-processing, or embedding-table expansion.

## 4. Empirical Outcomes, Trade-Offs, and Theoretical Insights

Continue-thinking tokens yield diverse empirical gains:

| Approach                              | Computation Cost          | Accuracy/Metric Gains               | Additional Findings                                 |
|----------------------------------------|--------------------------|-------------------------------------|-----------------------------------------------------|
| Thinking Tokens (‹T›, N=1) [2405.08644] | Mild increase (seq. length) | - Maths queries: PPL reduced 16.8→13.1, 24.3→19.8   | More than N=1 degrades perplexity; no core arch. change needed |
| Info-theoretic peaks [2506.02867]       | No runtime overhead       | Suppressing thinking tokens: accuracy drops 50%→15% | Gains: +1–3% ABS on GSM8K/MATH500, +10–15% on AIME24; extendable to new domains |
| Conditional Token Selection [2505.17827]| Substantial token savings | GPQA: +9.1% accuracy at –13% tokens; –5% with –75.8% tokens | Performance robust to strong compression; proven redundancy in CoT |
| Learned embedding [2506.11274]          | No LLM retraining         | GSM8K: +4.2% ABS (vs. +1.3% for fixed “Wait”)       | Improved chain-of-thought quality, generalization to more continuation steps |
| Multiplex/Soft tokens [2509.19170, 2601.08808] | Trivial overhead         | Pass@32: +2–5% over hard CoT; higher OOD preservation | Training with soft tokens followed by hard inference optimal; higher diversity and stability |

Theoretical analysis confirms that mutual information concentration at select steps (MI peaks) matches tokens empirically important for correct reasoning, and that the overall error rate is bounded above and below by cumulative MI contributions at these steps [2506.02867]. Multiplex and continuous tokens realize information superposition, enabling more efficient encoding and exploration of reasoning paths [2509.19170, 2601.08808]. 

A notable limitation is diminishing returns or even performance drop with excessive token insertion (overthinking), as the model can lose track of context or introduce noise [2405.08644]. Likewise, MI estimation is performed via proxies (HSIC), and direct causal attribution of reasoning improvement to token mechanics is an open question [2506.02867].

## 5. Extensions: Modality, Supervision, and Domain Adaptation

Continue-thinking mechanisms have been deployed or critiqued across various domains and supervision regimes:

- **Latent reasoning with parallelization:** The “Thinking States” approach enables full teacher-forcing and parallel computation with natural-language “thought tokens” generated and re-compressed at fixed points in the input stream [2602.08332]. This yields state-tracking accuracy 100% in OOD (vs. 64.4% for CoT) and a 2× speedup.
- **Impact on machine translation:** Direct transfer of thinking tokens or synthetic CoT explanations to MT did not improve translation performance; only modular traces embedding actual translation drafts yielded notable gains, suggesting the utility of continue-thinking tokens is highly domain-dependent [2510.11919].
- **Planning and self-awareness:** Reasoning-trace tokens arising from planning policies (“think” slots) in SFT, DPO, or GRPO are subject to evaluation for faithfulness, OOD generalization, and self-introspection; results show RL-based policies yield stronger awareness and generalization but can decouple internal traces from final answers [2510.16340].
- **Adaptive and compressive agents:** The conditional token selection framework offers a pathway to fully differentiable gating of reasoning steps, extensible to multimodal or code-based tasks [2505.17827].

Overall, there is strong evidence that continue-thinking tokens’ impact varies with the reasoning domain, the quality of annotation, and the underlying decoding policy.

## 6. Practical Considerations, Limitations, and Perspectives

Learned continue-thinking tokens are a lightweight, modular tool for improving LLM reasoning performance, especially on complex mathematical or multi-step tasks. They can be introduced without core architecture changes, learned via RL or supervised masking, and interpreted via information-theoretic or empirical analyses. However, they require careful tuning: too many tokens can impair efficiency, and their utility is governed by task structure—mere natural-language “thoughts” are insufficient for tasks such as machine translation unless they encode explicit, intermediate solutions [2510.11919].

Open directions include dynamic token-insertion policies (learned, not static), differentiated multi-token schemes for hierarchical reasoning, and principled exploration strategies that go beyond random perturbation. Information-theoretic measurements of reasoning are still indirect, and extending these methods to non-math, non-English, or genuinely open-ended domains remains ongoing work.

The development of learned continue-thinking tokens demonstrates a convergence of data-centric, optimization-based, and information-theoretic advances in large-scale language model reasoning frameworks.

Source: https://www.emergentmind.com/topics/learned-continue-thinking-tokens