SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs (2510.05069v1)

Abstract: Recent work shows that, beyond discrete reasoning through explicit chain-of-thought steps, which are limited by the boundaries of natural languages, LLMs can also reason continuously in latent space, allowing richer information per step and thereby improving token efficiency. Despite this promise, latent reasoning still faces two challenges, especially in training-free settings: 1) purely latent reasoning broadens the search distribution by maintaining multiple implicit paths, which diffuses probability mass, introduces noise, and impedes convergence to a single high-confidence solution, thereby hurting accuracy; and 2) overthinking persists even without explicit text, wasting tokens and degrading efficiency. To address these issues, we introduce SwiReasoning, a training-free framework for LLM reasoning which features two key innovations: 1) SwiReasoning dynamically switches between explicit and latent reasoning, guided by block-wise confidence estimated from entropy trends in next-token distributions, to balance exploration and exploitation and promote timely convergence. 2) By limiting the maximum number of thinking-block switches, SwiReasoning curbs overthinking and improves token efficiency across varying problem difficulties. On widely used mathematics and STEM benchmarks, SwiReasoning consistently improves average accuracy by 1.5%-2.8% across reasoning LLMs of different model families and scales. Furthermore, under constrained budgets, SwiReasoning improves average token efficiency by 56%-79%, with larger gains as budgets tighten.

• The paper presents a novel training-free inference framework that dynamically alternates between explicit and latent reasoning using block-wise entropy signals.
• It demonstrates significant improvements with up to 5% accuracy gains and token efficiency enhancements of up to 79% on various STEM benchmarks.
• Switch count control and early-answer checkpoints effectively suppress overthinking, reducing required samples by up to 72% compared to standard chain-of-thought methods.

SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs

Introduction

SwiReasoning introduces a training-free inference framework for LLM reasoning that alternates dynamically between explicit chain-of-thought (CoT) and latent reasoning modes. The framework leverages block-wise confidence signals, estimated from entropy trends in next-token distributions, to guide mode switching. Additionally, SwiReasoning incorporates a switch count control mechanism to suppress overthinking and improve token efficiency, especially under constrained token budgets. The approach is validated across multiple mathematics and STEM benchmarks and LLM families, demonstrating consistent improvements in both accuracy and efficiency over single-mode baselines.

Motivation and Background

Explicit CoT reasoning, while interpretable and robust, collapses the model's uncertainty at each step by committing to a single token, thereby discarding alternative hypotheses and limiting representational bandwidth. Latent reasoning, in contrast, operates in the continuous hidden space, preserving richer distributional information and multiple implicit reasoning paths. However, training-free latent reasoning faces two major challenges: (1) probability mass diffusion across many implicit paths, which introduces noise and impedes convergence, and (2) persistent overthinking, leading to inefficient token usage.

SwiReasoning addresses these limitations by dynamically switching between explicit and latent modes based on real-time confidence signals, and by bounding the number of mode switches to prevent excessive internal deliberation.

SwiReasoning Framework

The SwiReasoning framework consists of two core components:

1. Dynamic Mode Switching: The system alternates between explicit and latent reasoning blocks. Mode transitions are triggered by block-wise entropy trends: a decrease in entropy (rising confidence) prompts a switch to explicit reasoning, while an increase (falling confidence) triggers a switch to latent reasoning. Dwell windows are imposed to prevent oscillatory switching, with asymmetric window sizes reflecting the divergent nature of latent reasoning and the convergent nature of explicit reasoning.
2. Switch Count Control: To suppress overthinking, SwiReasoning limits the total number of Latent→Explicit switches. Early-answer checkpoints are introduced at switch boundaries, allowing the model to generate answers based on partial reasoning trajectories. Convergence and termination triggers are implemented via token injection queues, enforcing answer generation when the switch budget is exhausted.

   Figure 1: SwiReasoning framework. (a) Dynamic mode switching alternates between explicit and latent thinking based on block-wise confidence estimated from entropy trends. (b) A switch count control mechanism limits the maximum number of thinking-block transitions, suppressing overthinking before the final answer.

Implementation Details

• Explicit Reasoning: Standard token-level decoding using greedy or sampling policies.
• Latent Reasoning: At each step, the next input is a probability-weighted mixture of token embeddings, preserving the full next-token distribution.
• Signal Mixing: Embeddings of special tokens (e.g., >, ``) are blended into the latent input at mode transitions to align with learned reasoning patterns.
• Hyperparameters: Window size ($W$), entrance/exit mixing coefficients ($\alpha_0$, $\beta_0$), and switch budget ($C_{\max}$) are tunable per task and model.

Experimental Results

Reasoning Accuracy

SwiReasoning consistently outperforms standard CoT (sampling and greedy) and Soft Thinking (latent-only) baselines across all evaluated benchmarks and model families. On challenging benchmarks such as AIME24 and AIME25, absolute accuracy gains reach up to 5.0% for Qwen3-1.7B and 3.34% for Qwen3-8B.

Figure 2: Pass@1 accuracy under unlimited token budgets. On mathematics and STEM reasoning benchmarks, SwiReasoning yields improvements of up to $+2.8\%$ and $+2.0\%$, respectively.

Token Efficiency

Under limited token budgets, SwiReasoning achieves substantial improvements in token efficiency, with average gains of 56%–79% and peak efficiency gains up to 6.8× over CoT. The efficiency advantage is most pronounced on easier tasks, where early convergence is feasible.

Figure 3: Token efficiency (accuracy per token compared to standard CoT), under limited token budgets. Across reasoning LLM families and sizes, SwiReasoning brings average efficiency improvements of up to $+79\%$.

Figure 4: Token efficiency comparisons. SwiReasoning achieves the highest token efficiency throughout all token budgets in 13 out of 15 evaluations, with an efficiency improvement of $+84\%$ over CoT on average.

Pass@k Accuracy

SwiReasoning reaches maximal Pass@k accuracy with significantly fewer samples than baselines. For example, on AIME24, SwiReasoning achieves peak accuracy at $k=13$ versus $k=46$ for CoT, representing a 72% reduction in required samples.

Figure 5: Pass@k accuracy ($k \in [1,64]$) evaluation with Qwen3-8B on AIME 2024 and 2025 benchmarks. SwiReasoning achieves maximum reasoning accuracies $+50\%$ earlier compared to CoT on average.

Ablation Studies

Ablations on window size, signal mixing coefficients, and switch count confirm the necessity of intermediate dwell windows and appropriate mixing ratios for optimal performance. Excessively small windows lead to premature switching and noise exposure, while large windows hinder latent exploration. The exit bias ($\beta_0$) is particularly critical, with performance peaking near $\beta_0=0.7$.

Implementation and Deployment Considerations

• Computational Overhead: SwiReasoning operates entirely at inference, requiring no model retraining or architectural changes. The main overhead is the computation of entropy and soft embedding mixtures, which is negligible compared to model forward passes.
• Integration: The framework is compatible with Huggingface's model.generate() interface and can be layered atop existing efficiency techniques (e.g., KV cache compression, speculative decoding).
• Scalability: SwiReasoning is applicable to LLMs of varying sizes and families, with hyperparameters tunable per deployment scenario.
• Limitations: The approach relies on entropy as a proxy for confidence, which may not capture all forms of uncertainty. Further, the fixed switch budget may require task-specific tuning for optimal results.

Theoretical and Practical Implications

SwiReasoning demonstrates that dynamic alternation between explicit and latent reasoning, guided by confidence signals, yields Pareto-superior performance in both accuracy and efficiency. The framework provides a principled mechanism for balancing exploration and exploitation during inference, mitigating the limitations of single-mode reasoning. The results suggest that further integration with reinforcement learning or adaptive controllers could enhance reasoning capabilities, especially for tasks requiring long-horizon deductive chains.

Conclusion

SwiReasoning offers a robust, training-free solution for improving LLM reasoning by dynamically switching between explicit and latent modes and suppressing overthinking via switch count control. The empirical results substantiate consistent gains in both accuracy and token efficiency across diverse benchmarks and model families. The framework is readily deployable and compatible with existing LLM infrastructures, providing a practical pathway for enhancing reasoning performance in real-world applications. Future work may explore adaptive switch control and integration with RL-based training for further improvements.