The Markovian Thinker (2510.06557v1)

Abstract: Reinforcement learning (RL) has recently become a strong recipe for training reasoning LLMs that produce long chains of thought (LongCoT). Yet the standard RL "thinking environment", where the state is the prompt plus all prior reasoning tokens, makes the state unbounded and forces attention-based policies to pay quadratic compute as thoughts lengthen. We revisit the environment itself. We propose Markovian Thinking, a paradigm in which the policy advances reasoning while conditioning on a constant-size state, decoupling thinking length from context size. As an immediate consequence this yields linear compute with constant memory. We instantiate this idea with Delethink, an RL environment that structures reasoning into fixed-size chunks. Within each chunk, the model thinks as usual; at the boundary, the environment resets the context and reinitializes the prompt with a short carryover. Through RL, the policy learns to write a textual state near the end of each chunk sufficient for seamless continuation of reasoning after reset. Trained in this environment, an R1-Distill 1.5B model reasons in 8K-token chunks yet thinks up to 24K tokens, matching or surpassing LongCoT-RL trained with a 24K budget. With test-time scaling, Delethink continues to improve where LongCoT plateaus. The effect of linear compute is substantial: we empirically estimate at 96K average thinking length LongCoT-RL costs 27 H100-months vs. 7 for Delethink. Analysis at RL initialization shows off-the-shelf reasoning models (1.5B-120B) often sample Markovian traces zero-shot across diverse benchmarks, providing positive samples that make RL effective at scale. Our results show that redesigning the thinking environment is a powerful lever: it enables very long reasoning without quadratic overhead and opens a path toward efficient, scalable reasoning LLMs.

• The paper demonstrates that segmenting reasoning into fixed-size chunks with a Markovian state enables linear scaling in compute and constant memory usage.
• It employs Delethink, a specialized RL environment that resets context at chunk boundaries, ensuring efficient long-horizon reasoning irrespective of token count.
• Empirical tests on math benchmarks show that Delethink outperforms traditional LongCoT methods, enabling scaling to reasoning traces of over 96K tokens.

The Markovian Thinker: Efficient Long-Horizon Reasoning via Markovian RL Environments

Introduction and Motivation

The paper introduces the Markovian Thinking paradigm for training LLMs to perform long-horizon reasoning with linear compute and constant memory, addressing the quadratic scaling bottleneck inherent in standard chain-of-thought (CoT) reinforcement learning (RL) environments. In the conventional LongCoT RL setup, the model's state is the concatenation of the prompt and all previously generated reasoning tokens, resulting in unbounded context growth and quadratic attention cost as the reasoning trace lengthens. This severely limits the practical scaling of reasoning length, both during RL training and inference.

The Markovian Thinking paradigm proposes a fundamental shift: the RL environment is redefined so that the policy (the LLM) always conditions on a constant-size state, regardless of the total reasoning length. This decouples the length of the reasoning trace from the context size, enabling efficient scaling to much longer chains of thought.

Delethink: Markovian RL Environment

The paper instantiates Markovian Thinking with Delethink, a novel RL environment that structures reasoning into a sequence of fixed-size chunks. Within each chunk, the model generates as usual, but at each chunk boundary, the environment resets the context to a fresh prompt containing the original query and a short carryover (the "Markovian state") from the previous chunk. The policy must learn, via RL, to encode all necessary information for seamless continuation into this bounded textual state.

This approach enforces a hard cap on the model's context size at every step, making the compute and memory requirements independent of the total reasoning length. The transition dynamics of the environment are modified such that, at chunk boundaries, only the carryover state is preserved, and all previous tokens are deleted from the context.

(Figure 1)

Figure 1: Delethink redefines the RL environment as a chunked, Markovian process, with context resets at chunk boundaries and only a short carryover state, in contrast to LongCoT's ever-growing context.

Theoretical and Empirical Analysis of Compute Scaling

The authors provide a detailed analysis of the computational complexity of Delethink versus LongCoT. In LongCoT, both training and inference scale quadratically with the number of thinking tokens due to the growing context. In Delethink, the cost is quadratic in the chunk size but only linear in the number of chunks, yielding overall linear scaling with reasoning length.

Empirical results confirm these predictions: for a 1.5B parameter model, training to an average thinking length of 96K tokens requires 27 H100-months with LongCoT-RL, but only 7 H100-months with Delethink. The per-GPU throughput remains constant as the thinking budget increases in Delethink, while it degrades linearly in LongCoT due to memory growth.

Figure 2: Delethink matches and surpasses LongCoT-RL in accuracy during RL training while using less compute; only Delethink continues to improve as the thinking budget is scaled beyond training limits.

Figure 3: Computational profiles show Delethink's linear scaling in FLOPs and constant memory, compared to LongCoT's quadratic scaling.

Empirical Results: Reasoning Performance and Test-Time Scaling

Delethink is evaluated on competition-level math benchmarks (AIME'24, AIME'25, HMMT'25), as well as out-of-distribution tasks (GPQA-Diamond, LiveCodeBench). With a 24K thinking-token budget, Delethink matches or outperforms LongCoT-RL 24K, and significantly outperforms LongCoT-RL 8K, confirming the necessity of extended reasoning.

A key result is that Delethink enables genuine test-time scaling: when allowed to reason with more tokens at inference than seen during training, Delethink continues to improve, while LongCoT-RL plateaus at its training limit. For some problems, solutions only emerge after reasoning with 100K+ tokens, despite the model being trained with a 24K budget.

Figure 4: On IID math tasks, Delethink outperforms LongCoT-RL 24K; on OOD tasks, Delethink matches or slightly beats LongCoT-RL 24K. Delethink's throughput remains constant as thinking scales.

Figure 5: Only Delethink continues to improve as the thinking budget is scaled at inference, while LongCoT-RL plateaus at its training limit.

Scaling to Extreme Reasoning Lengths

The linear compute scaling of Delethink enables training with much larger thinking budgets. The authors demonstrate Delethink with a 96K token budget, achieving mean trace lengths up to 42K and surpassing both the baseline and test-time-extended Delethink 24K. This is achieved with only 150 RL steps, highlighting the practical feasibility of scaling reasoning length by an order of magnitude.

Figure 6: Delethink 96K achieves higher accuracy and longer average trace lengths than Delethink 24K, demonstrating scalability to extreme reasoning lengths.

Markovian Tracing in Off-the-Shelf LLMs

A critical finding is that off-the-shelf reasoning LLMs (R1-Distill 1.5B–14B, GPT-OSS 120B, Qwen3-30B-A3B) already exhibit strong Markovian behavior zero-shot: when Delethink Tracing is applied at initialization (without any additional training or prompting), these models recover most of their LongCoT performance, and in some cases even surpass it. This provides a strong initialization for RL and suggests that Markovian Thinking is a natural fit for current SOTA models.

Figure 7: Delethink Tracing at initialization recovers most of LongCoT performance, indicating Markovian behavior in the base model.

Figure 8: SOTA LLMs (GPT-OSS-120B, Qwen3-30B-A3B) are capable of Markovian Thinking zero-shot, providing a strong initialization for Delethink training.

Ablations: Context and State Size

The paper includes extensive ablations on the per-chunk context size ($\mathcal{C}$) and the Markovian state size ($m$). Delethink is robust to reductions in context size down to 4K, with only modest performance degradation. However, at 2K, models struggle to terminate within budget and accuracy drops, indicating a lower bound on effective context size. The Markovian state size can be reduced significantly (e.g., $m \ll \mathcal{C}$) with little impact on accuracy for R1-Distill models, but larger models (Qwen3) benefit from larger state sizes, especially on long-trace tasks.

Figure 9: Ablation of the Markovian state size $m$ at fixed per-chunk context $\mathcal{C}=8\text{k}$; R1-Distill models are robust to small $m$, while Qwen3 benefits from larger $m$.

Figure 10: Scaling behavior of R1-Distill models under varying per-chunk contexts; 4K and 8K are effective, while 2K severely limits performance.

Limitations and Stress Tests

Delethink's Markovian approach is less effective on tasks that require persistent, large-scale memory, such as crossword puzzles where previously found words must be retained. In these stress tests, Delethink remains competitive but its zero-shot limits are evident, as the bounded state cannot always encode all necessary information.

Figure 11: On CrossWordBench, Delethink remains competitive but its zero-shot limits are evident due to the need for persistent memory.

Implementation and Practical Considerations

Delethink is implemented as a simple wrapper around standard transformer inference, requiring only that the context be reset at chunk boundaries and a short carryover state be appended. The approach is architecture-agnostic and can be combined with any attention variant (full, sliding, streaming). The RL objective is a straightforward adaptation of PPO to the chunked environment, with per-chunk loss normalized by the total number of thinking tokens.

A sample implementation using HuggingFace Transformers is provided, demonstrating the ease of integration into existing LLM stacks.

Implications and Future Directions

The Markovian Thinking paradigm, as instantiated by Delethink, demonstrates that long-horizon reasoning can be achieved with linear compute and constant memory, without architectural modifications. This opens the door to training and deploying reasoning LLMs that can think for hundreds of thousands or millions of tokens, previously infeasible due to quadratic scaling.

The results suggest that non-quadratic sequence architectures (e.g., linear attention, state-space models) may be particularly well-suited for reasoning tasks, as the Markovian environment decouples reasoning length from context size. The strong zero-shot Markovian behavior in current SOTA LLMs indicates that further scaling and optimization in this direction is both practical and promising.

Conclusion

The Markovian Thinker framework redefines the RL environment for reasoning LLMs, enabling efficient, scalable long-horizon reasoning by enforcing a constant-size state via chunked generation and explicit state passing. Delethink matches or surpasses standard LongCoT-RL in accuracy, achieves substantial compute savings, and uniquely enables test-time scaling far beyond training limits. The paradigm is robust, simple to implement, and compatible with existing LLM architectures. These results have significant implications for the future of scalable reasoning in LLMs, suggesting a path toward models capable of sustained, efficient, and arbitrarily long chains of thought.