Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning

Abstract: Latent chain-of-thought compresses reasoning by replacing visible reasoning traces with continuous hidden-state recurrence, but existing formulations are difficult to optimize with standard on-policy reinforcement learning (RL) and hard to interpret causally. Our key insight is that a single pair of explicit boundary tokens can address both issues at once: discrete entry and exit anchors make the latent block compatible with standard on-policy RL, and the same anchors offer a natural foothold for mechanistic analysis. Motivated by this, we propose SWITCH, a switchable latent reasoning framework. The model emits <swi> to enter latent mode and </swi> to exit. Because the boundaries are ordinary discrete tokens, the GRPO policy ratio is well-defined at every decision point. The same anchors also expose the latent steps to direct probing and causal intervention. We train the model with a visible-to-latent curriculum and a Switch-GRPO objective that propagates gradients through recurrent latent computation. SWITCH consistently outperforms prior hidden-state-recurrence latent reasoning approaches at similar scale. Mechanistic analysis through the boundary tokens further reveals three findings: (i) <swi> is a sharply localised, learned switching policy rather than a stylistic artefact; (ii) the latent step it opens performs problem-specific, causally important computation rather than acting as an inert placeholder; and (iii) that computation is concentrated at a single hidden-state transition on entry. Together, these results show that hidden-state-recurrence latent reasoning is both RL-trainable and open to direct mechanistic analysis, including of how on-policy RL itself improves the model from the inside.

• The paper introduces Switch, leveraging explicit tokens (<swi> and </swi>) to enable effective RL training and mechanistic analysis of hidden-state recurrence.
• It employs a novel three-phase training protocol—SFT, curriculum learning, and Switch-GRPO—that boosts math reasoning benchmarks by up to 25.7 points.
• The approach provides explicit control over accuracy-efficiency tradeoffs and uses systematic interventions to reveal the causal role of latent computations.

Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning

Introduction

The paper "Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning" (2606.13106) addresses two fundamental barriers in the latent chain-of-thought (CoT) paradigm for LLMs: (1) the incompatibility between hidden-state recurrence and standard on-policy RL techniques, and (2) the inaccessibility of latent computation to mechanistic analysis. Extending the hidden-state recurrence approach pioneered by Coconut, the authors introduce Switch, which leverages explicit boundary tokens (<swi> and </swi>) to enable both effective RL training and interpretability. Through a three-phase training protocol and a dedicated RL objective (Switch-GRPO), the framework outperforms prior baselines both in terms of mathematical reasoning accuracy and computability of internal states.

Figure 1: Switch overview. Three-phase training augments a Qwen3 base with switchable latent reasoning; during inference, a <swi> token triggers a latent block with recurrent hidden-state updates.

Methods

Architectural Innovation: Switchable Latent Blocks

Switch introduces three special tokens—<swi>, </swi>, and <latent>—to the underlying Qwen3-8B model. At inference, the model proceeds in standard text generation until emitting <swi>, after which it runs a sequence of latent (hidden-state-injected) steps, terminating with </swi> and resuming text output. Each latent step recursively feeds the previous step’s final-layer hidden state as the input embedding for the next (Equation 1), inheriting the modeling simplicity of Coconut-style recurrence while tying mode transitions to explicit, learned emission points.

Training Protocol

Switch’s training divides into three distinct stages:

1. Phase 1 (SFT): The model is fine-tuned to learn when to open and close latent blocks, using high-entropy regions in annotated CoT traces as positive examples for <swi>/</swi> placement.
2. Phase 2 (Curriculum): A curriculum replaces spans within <swi> blocks with <latent> positions, using a parallel schedule—shown to induce more robust latent computation than sequential alternatives.

   Figure 2: Sequential vs. parallel curriculum schedules. Parallel replacement enforces out-of-distribution generalization inside every <swi> span.

3. Phase 3 (Switch-GRPO): On-policy RL is conducted using the Group Relative Policy Optimization (GRPO) algorithm, with a custom likelihood computation that factors over text positions only—latent positions are deterministic and omitted from the policy gradient, yet they propagate reward signals via the boundaries.

   Figure 3: Switch-GRPO training trajectory with normalized latent invocations, token usage, and performance metrics across optimizer steps.

The reward used in RL is composite: strict answer correctness graded by math-verify, format validity, latent usage, and optionally a brevity component to induce shorter outputs (enabling exploration of accuracy-efficiency tradeoffs).

Experimental Results

Performance

On the MATH-500 and GSM8K benchmarks, Switch demonstrates 79.3% and 89.2% accuracy, respectively—substantially surpassing prior hidden-state-recurrence methods, with +25.7 points on MATH-500 vs. the best Coconut-style baseline. Relative to curriculum-only SFT, RL with Switch-GRPO further halves the model’s latent invocation rate while improving conditional accuracy on latent-invoked problems by +12.6 points.

Efficiency and Tradeoffs

Switch supports explicit control over the accuracy-efficiency Pareto frontier. By modifying reward parameters to emphasize brevity, the model trades a marginal amount of accuracy for shorter output sequences and effectively eliminates output truncation.

Figure 4: Accuracy-efficiency operating curve showing explicit user control over output length and accuracy tradeoff.

Figure 5: Token distributions comparing SFT, Switch-GRPO, and brevity-optimized variants.

Mechanistic Analysis

The introduction of explicit boundary tokens enables rigorous mechanistic study of latent reasoning dynamics.

Boundary Localization and Decodability

Analysis shows that <swi> emission is a sharply localized policy decision: the log-probability of emitting <swi> spikes by four orders of magnitude at annotated boundaries compared to non-boundary control positions. Linear probing of late-layer activations reveals that the boundary state is decodable at ≥88% accuracy, indicating robust internal computation of the switching policy.

Figure 6: Switch-window: $p(<swi>)$ exhibits a clean, isolated spike at the boundary; RL slightly softens but preserves the contrast.

Figure 7: Linear probe accuracy by layer for next-token-is-<swi>; boundary information emerges in late layers.

Causal Importance of Latent Steps

Intervention experiments—ablation, noise injection, and step skipping—demonstrate the necessity of correct latent updates: zeroing latent hidden states catastrophically degrades accuracy (e.g., a drop from 100% to 33% on a diagnostic subset), while random same-norm perturbations and block skipping are significantly less harmful, underscoring that it is the specific hidden-state update that encodes the reasoning step.

Figure 8: Latent-state intervention—accuracy and answer-change strongly degrade when latent state is zeroed but are robust to random-norm perturbation or omission.

Localization of Latent Computation

Logit lens and output probability analyses indicate that the majority of latent computation is concentrated immediately at the latent block’s entry. Without a minimum dwell constraint, the system tends to collapse latent blocks to single steps due to the exit-ready property at every latent position.

Limitations

Experiments are delimited to 8B-parameter Qwen3 models and mathematical reasoning tasks. The impact of scaling, transfer to multi-domain tasks, and combining hidden-state recurrence with samplable/vocabulary-mixture latents remain open questions. Furthermore, only the text segments propagate gradients in RL; the latent representations are primarily shaped by curriculum learning, not direct RL signals. Mechanistic analyses focus on confirming causal relevance and decodability, not on mapping reasoning trajectories in their entirety.

Implications and Future Directions

Switch demonstrates that hidden-state recurrence—if equipped with explicit, learnable boundary control—supports sound RL optimization and direct interpretability. This provides a constructive challenge to the view that only samplable (e.g., vocabulary-mixture) latent representations are RL-compatible, reopening the design space for latent CoT with richer mechanistic scrutiny. The explicit boundary mechanism also isolates discrete policy decisions, paving the way for more precise control and introspection in future LLM architectures that blend explicit and latent computation.

Potential avenues for further research include:

• Scaling analyses: Adapting and extending Switch to larger models, studying scaling laws for latent invocation rates and conditional accuracy.
• Cross-domain generalization: Deploying and evaluating in other reasoning-intensive domains (logical, scientific, multimodal) where latent compute can be critical.
• Hybrid latent design: Integrating recurrence-based and samplable latent approaches in a unified framework for increased flexibility and RL compatibility.
• Fine-grained mechanistic interpretability: Deeper, layer-wise analysis of latent computation and detailed intervention studies to trace multi-step reasoning dynamics.

Conclusion

Switch establishes that explicit, learned boundary tokens resolve both RL tractability and interpretability in recurrent latent CoT frameworks, outperforming prior hidden-state recurrence baselines by large margins while supporting direct policy analysis and causal testing. This result affirms that recurrent latent spaces, when combined with proper interface and policy optimization, are not merely architectural curiosity but effective and analyzable reasoning substrates for LLMs.