Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought

Abstract: While long, explicit chains-of-thought (CoT) have proven effective on complex reasoning tasks, they are costly to generate during inference. Non-verbal reasoning methods have emerged with shorter generation lengths by leveraging continuous representations, yet their performance lags behind verbalized CoT. We propose $\textbf{Abstract Chain-of-Thought}$, a discrete latent reasoning post-training mechanism in which the LLM produces a short sequence of tokens from a reserved vocabulary in lieu of a natural language CoT, before generating a response. To make previously unseen ''abstract'' tokens useful, we introduce a policy iteration-style warm-up loop that alternates between (i.) bottlenecking from a verbal CoT via masking and performing supervised fine-tuning, and (ii.) self-distillation by training the model to generate abstract tokens from the prompt alone via constrained decoding with the codebook. After warm-up, we optimize the generation of abstract sequences with warm-started reinforcement learning under constrained decoding. Abstract-CoT achieves up to $11.6\times$ fewer reasoning tokens while demonstrating comparable performance across mathematical reasoning, instruction-following, and multi-hop reasoning, and generalizes across LLM families. We also find an emergent power law distribution over the abstract vocabulary, akin to those seen in natural language, that evolves across the training phases. Our findings highlight the potential for post-training latent reasoning mechanisms that enable efficient inference through a learned abstract reasoning language.

• The paper introduces Abstract-CoT, replacing verbose chain-of-thought with concise, abstract token sequences to boost reasoning efficiency.
• It employs a warm-up stage using bottlenecked SFT and self-distillation followed by RL with GRPO, ensuring strong performance on diverse benchmarks.
• Empirical results demonstrate up to an 11.6× reduction in reasoning tokens with minimal performance loss, highlighting both efficiency and scalability.

Abstract Chain-of-Thought: Efficient Latent Reasoning with Discrete Token Traces

Introduction and Motivation

The paper "Thinking Without Words: Efficient Latent Reasoning with Abstract Chain-of-Thought" (2604.22709) interrogates the necessity of explicit, verbose rationales in LLM-mediated reasoning, proposing an alternative mechanism rooted in discrete latent representations. Verbalized chain-of-thought (CoT) has demonstrated substantial efficacy in facilitating multi-step reasoning across benchmarks, yet incurs significant inference-time computational cost and exacerbates RL training trace length. More fundamentally, recent evidence underscores that verbalized CoTs are frequently unfaithful, poorly aligned with the model's true latent computations, motivating the search for non-verbal intermediates that preserve performance while minimizing inference budget.

The Abstract Chain-of-Thought (Abstract-CoT) method introduced here replaces natural language rationales with bounded-length sequences composed from a reserved abstract token vocabulary, serving as a latent scratchpad. The central hypothesis is that a post-training pipeline can endow LLMs with the capacity to reason efficiently via learned discrete intermediation, rendering explicit language unnecessary for high performance.

Figure 1: Comparison of verbalized vs.\ abstract chain-of-thought; Abstract-CoT achieves equivalent reasoning outcomes with significantly fewer tokens.

Methodology

The Abstract-CoT framework extends the tokenizer with $M$ previously unseen, randomly initialized abstract tokens, demarcated by dedicated delimiters. The cold-start problem associated with these tokens is addressed via a policy iteration-style warm-up loop comprising: (i) bottlenecked SFT, leveraging attention masks to enforce an information bottleneck from verbal CoT to abstract sequence; and (ii) self-distillation, shaping the abstract generator to operate from the prompt alone. This alternation bootstraps semantically meaningful latent representations, enabling downstream RL to exploit the newly learned abstract codebook.

The warm-up stage is followed by warm-started RL with generative reward model-based scoring, employing GRPO and constrained decoding over the abstract vocabulary. The RL objective regularizes both the abstract and response distributions via KL to the warm-started reference policy, structuring exploration and stability.

Figure 2: Abstract-CoT training recipe: policy iteration warm-up alternating SFT and self-distillation, followed by warm-started RL with GRPO.

Empirical Results

Evaluation is performed on diverse benchmarks—MATH-500, AlpacaEval-LC-2.0, HotpotQA, AIME'25, GPQA-Diamond—spanning verifiable and non-verifiable reasoning regimes. Multiple post-training approaches are contrasted: pause token insertion, stepwise internalization (ICoT-SI), direct SFT, SFT with verbal CoT, SFT + RL, and Abstract-CoT variants (RL-only, warm-up only, warm-up + RL).

Strong numerical results substantiate the efficiency of Abstract-CoT:

• On Qwen3-8B, Abstract-CoT (warm-up + RL) achieves 90.8% accuracy on MATH-500 using only 144 reasoning tokens, compared to 92.6% for verbal CoT SFT + RL (1671 tokens), yielding a $\sim11.6\times$ reduction in reasoning token budget.
• For AlpacaEval, this configuration attains 60.8 win-rate with 225 tokens (SFT+RL: 58.4/496), and for HotpotQA, 58.8 F1 with 171 tokens (SFT+RL: 58.1/735).
• Similar trends emerge for Qwen3-4B and Granite-4.0-Micro, with Abstract-CoT outperforming or closely matching verbalized CoT, despite using far fewer tokens.

Compression ratios up to $10.4-11.6\times$ (MATH-500), $1.9-2.2\times$ (AlpacaEval), and $4.0-4.3\times$ (HotpotQA) are documented.

Figure 3: Abstract-CoT reduces generated token count drastically without compromising benchmark scores for both mathematical and instruction-following tasks.

Scaling and Ablation Studies

Vocabulary scaling ablations reveal clear trends: enlarging the abstract codebook up to $M=64$ enhances performance, after which saturation and slight decline occur; cold-start RL remains ineffective irrespective of vocabulary size. Warm-up is critical to embedding utility.

Figure 4: MATH-500 abstract vocabulary scaling; warm-up enables scaling, cold-start RL plateaus early.

Figure 5: AlpacaEval abstract vocabulary scaling; performance peaks at $M=64$ for warm-up + RL, supporting cross-task generality.

Figure 6: HotpotQA scaling—near-linear improvement for warm-up + RL up to $M=64$, then degradation; cold-start RL consistently lags.

Analysis of Abstract Reasoning Language

Abstract token usage distribution evolves during RL, manifesting a power-law structure analogous to Zipf's law in natural language. This emergent compositionality suggests the abstract reasoning language is nontrivial, capturing reusable concepts and indicating clustering behavior.

Figure 7: RL training induces power-law distribution over abstract tokens; the learned codebook reflects emergent concept reuse.

Permutation and truncation sensitivity analyses demonstrate that both Abstract-CoT and verbal CoT respond to order perturbations with degraded performance, confirming that sequence structure is essential for effective reasoning. However, the performance loss for Abstract-CoT under truncation is less severe, consistent with its training to reason with short sequences.

Practical and Theoretical Implications

Abstract-CoT constitutes an intermediate regime between full interpretability and latent-only computation: explicit CoT is replaced with a bounded, analyzable token segment. Practically, this enables large inference-time savings, making budgeted reasoning feasible; theoretically, the power-law and compositional effects raise new questions about concept formation and representation dynamics in post-trained discrete codebooks.

The discrete abstract interface may provide leverage for monitorability and auditability of LLM reasoning traces, promising more controllable, efficient intermediate representations. Hierarchical abstract reasoning languages or adaptive sequence length mechanisms could further enable scaling to longer-horizon or more complex tasks.

Figures

Figure 1: Explicit comparison of verbalized and abstract chain-of-thought—Abstract-CoT obtains equivalent answers with fewer reasoning tokens.

Figure 2: Policy iteration warm-up and RL stages, visualizing how abstract tokens are learned and optimized.

Figure 3: Plot of generated tokens vs benchmark scores, highlighting efficiency gains of Abstract-CoT.

Figure 7: Evolution and end-state of abstract token frequency distribution across RL training, showing emergence of power law.

Figure 4: Scaling ablation on MATH-500; vocabulary size directly affects performance gains.

Figure 5: Scaling ablation on AlpacaEval, confirming cross-domain robustness of Abstract-CoT.

Figure 6: HotpotQA scaling curves, with warm-up + RL outperforming cold-start for all vocabulary sizes.

Conclusion

Abstract Chain-of-Thought post-training induces efficient latent reasoning in LLMs by compressing rationales into short, bounded sequences of reserved discrete tokens, shaped via policy iteration warm-up and warm-started RL. Strong empirical evidence establishes performance parity with explicit CoT and substantial reductions in inference cost. Emergent power-law token dynamics and compositional sensitivity indicate the formation of a new, learned reasoning language, motivating future exploration in monitorability, budget-adaptive reasoning, and mechanistic concept analysis. This approach enables practical, scalable inference in environments where verbose, human-readable explanations are unnecessary, laying groundwork for further theoretical and practical advancements in latent reasoning interfaces.