Memory-Efficient Markov Chain of Thought

• Memory-efficient MCoT is a formal framework that enables multi-step reasoning by compressing prior context into a succinct state, enforcing the Markov property.
• It uses techniques like stateless reduction, self-correction, and latent state transitions to achieve O(1) or sublinear memory scaling, thus mitigating memory bottlenecks.
• Empirical results show that MCoT frameworks significantly reduce GPU memory usage and accelerate inference, improving performance in tasks such as mathematical reasoning and long-context QA.

A memory-efficient Markov Chain of Thought (MCoT) is a formal framework for enabling multi-step reasoning in LLMs under stringent memory and computational constraints. Instead of persisting the entire sequence of prior reasoning steps—as in conventional Chain-of-Thought (CoT) approaches—MCoT methods explicitly enforce (or approximate) the Markov property, wherein each step relies solely on a compressed state summarizing all relevant information from the preceding step. This design enables O(1) or sublinear memory scaling per reasoning step, supports arbitrarily long reasoning chains, and is foundational to a new class of scalable reasoning paradigms covering symbolic, retrieval-augmented, and latent-space models. MCoT variants have been empirically validated to both accelerate inference and reduce GPU memory while maintaining or improving task accuracy across mathematical reasoning, long-context QA, and multi-modal benchmarks (Yang et al., 2024, Zhang et al., 8 Apr 2026, Lei et al., 9 Apr 2026, Liu et al., 29 Sep 2025, Viteri et al., 2024).

1. Rationale and Core Principles

The rise of long-form CoT for complex problem solving in LLMs has exposed memory bottlenecks, especially in transformer-based models with quadratic scaling of KV-cache (key-value cache) and prompt length. In standard CoT, every step appends reasoning to the prompt, causing linear (or worse) memory growth in the number of steps $L$ and the number of tokens per step $N$, with total KV-cache cost $O(NL)$. MCoT frameworks address this by imposing stepwise marginalization over the history:

$p(s_t\,|\;q_{t'},s_{t'<t}) = p(s_t\,|\,q_t)$

This design collapses the dependency on prior full trajectory to a bounded state maintained at each step, which is either a short contiguous textual sub-question, a compressed vector, or a structured memory module. The Markovian property underpins MCoT’s memory efficiency and supports statistical independence between steps for training and inference (Yang et al., 2024, Viteri et al., 2024).

2. Canonical MCoT Algorithm and Variants

2.1 Stateless MCoT: “Derive and Reduce” Paradigm

A widely used MCoT pattern iteratively alternates between (a) deriving the next partial solution from the current sub-problem, and (b) reducing it to a yet-simpler sub-question, discarding all previous context except for the latest:

• Derivation: $s_t \sim \text{Model.generate\_step}(q_t)$ (where $s_t$ is text or code).
• Reduction: $$q_{t+1} \sim \text{Model.generate\_reduction}(q_t, s_t)$$.
• Memory: Only $(q_{t+1})$ is retained, enabling token and compute savings per step (Yang et al., 2024).

A Markovian property is strictly enforced:

$p(s_t, q_{t+1}|q_t)$

2.2 Self-Correction Mechanism

Some MCoT systems integrate an external code interpreter; if a derivation $s_t$ fails (e.g., produces an exception), a new sub-question is spawned locally until the code snippet is correct, then reduction continues. Self-correction operates within the same Markovian reduction mechanism (Yang et al., 2024).

2.3 Markovian Transformers

A Markovian transformer factors each next-token prediction through an intermediate, compact CoT state $N$0 of fixed size (e.g., a <400-token chunk) rather than the entire token history:

$N$1

Here, $N$2 is a CoT generator, and $N$3 is a CoT evaluator. This approach underlies highly memory-efficient language modeling (Viteri et al., 2024).

2.4 Hierarchical and Reversible MCoT

Cognitive Loop of Thought (CLoT) organizes the reasoning chain hierarchically. At each granularity $N$4 and timestep $N$5, the state:

$N$6

transitions via forward and backward (reversible) Markov processes. Cross-layer abstraction/refinement is explicitly parameterized, and a backward verification mechanism ensures local consistency at each level, supporting pruning of subordinate sub-problems upon high-confidence verification (Zhang et al., 8 Apr 2026).

2.5 MCoT with External Memory and Retrieval

Memory-driven MCoT integrates search over external knowledge with a two-component memory: (i) a semantic state memory $N$7 and (ii) an episodic trajectory log. The model iteratively issues sub-queries, updates the short-term memory, and prunes solved or irrelevant items via a judge agent, maintaining a strict per-step memory budget (Lei et al., 9 Apr 2026).

2.6 Latent Space MCoT

MARCoS models each reasoning step as a transition in a high-dimensional latent “thought” vector $N$8, decoupling reasoning (Markov chain in latent space) from explicit token generation. Only two consecutive latent states need to be stored at any time, and inference proceeds by updating latent vectors instead of generating hundreds of intermediate tokens (Liu et al., 29 Sep 2025).

3. Training Objectives and Datasets

Most MCoT approaches enable supervised or reinforcement learning at the per-step level:

• Maximum Likelihood: Training decomposes into $N$9 independent single-step likelihoods $O(NL)$0.
• Policy Gradient: An informativeness objective rewards the model for generating CoT traces that improve the accuracy of a frozen evaluator, optimized via REINFORCE or PPO (Viteri et al., 2024).
• ELBO (Variational): For latent state MCoT, per-step ELBOs encourage information-preserving transitions (Liu et al., 29 Sep 2025).
• Backward-Verification Loss: Joint forward+backward log-likelihoods are optimized to enable bidirectional reasoning and error-checking in hierarchically organized Markov chains (Zhang et al., 8 Apr 2026).

High-quality stepwise datasets such as MCoTInstruct and CLoT-Instruct provide $O(NL)$110^5 Markov chain fragments with step-level annotations for both forward and (if applicable) backward processes (Yang et al., 2024, Zhang et al., 8 Apr 2026).

4. Memory Efficiency and Computational Complexity

The central efficiency guarantee of MCoT arises from a bounded, often constant, per-step context:

• Token-based CoT: Context grows as $O(NL)$2.
• Stateless MCoT: Only current $O(NL)$3 state or $O(NL)$4 chunk ($O(NL)$5) is needed.
• Hierarchical/Reversible CLoT: Only the current layer’s KV cache ($O(NL)$6) and scalar verification summaries for each higher layer ($O(NL)$7) are retained, yielding $O(NL)$8 total memory (Zhang et al., 8 Apr 2026).
• Latent-space MCoT: Peak working memory is $O(NL)$9 with $p(s_t\,|\;q_{t'},s_{t'<t}) = p(s_t\,|\,q_t)$0, where $p(s_t\,|\;q_{t'},s_{t'<t}) = p(s_t\,|\,q_t)$1 is dimension, $p(s_t\,|\;q_{t'},s_{t'<t}) = p(s_t\,|\,q_t)$2 reasoning steps, $p(s_t\,|\;q_{t'},s_{t'<t}) = p(s_t\,|\,q_t)$3 tokens, and $p(s_t\,|\;q_{t'},s_{t'<t}) = p(s_t\,|\,q_t)$4 neuron banks, resulting in 10–20× lower memory than full autoregressive CoT (Liu et al., 29 Sep 2025).
• Retrieval-Augmented (MemCoT): Per-step memory is tightly bounded; only $p(s_t\,|\;q_{t'},s_{t'<t}) = p(s_t\,|\,q_t)$5 tokens and $p(s_t\,|\;q_{t'},s_{t'<t}) = p(s_t\,|\,q_t)$6 in-RAM state are used, with empirical reductions of $p(s_t\,|\;q_{t'},s_{t'<t}) = p(s_t\,|\,q_t)$785% in prompt size compared to full-context methods (Lei et al., 9 Apr 2026).

Empirical results demonstrate that MCoT/CLoT implementations consistently achieve substantial reductions in GPU KV-cache (e.g., from 280K to 78K tokens) and enable end-to-end throughput speedups up to $p(s_t\,|\;q_{t'},s_{t'<t}) = p(s_t\,|\,q_t)$815.7×, often with state-of-the-art accuracy (Yang et al., 2024, Zhang et al., 8 Apr 2026, Liu et al., 29 Sep 2025).

5. Application Domains and Empirical Performance

MCoT frameworks deliver advances in several areas:

Model/Approach	Application Domains	Typical Memory Reduction	Accuracy/Performance Gains
MCoT (Yang et al., 2024)	Math, QA	35%–40% GPU RAM, $p(s_t\,|\;q_{t'},s_{t'<t}) = p(s_t\,|\,q_t)$92× speedup	78.8% GSM8K (7B), 83.1% GSM8K (70B)
CLoT (Zhang et al., 8 Apr 2026)	Math reasoning	$s_t \sim \text{Model.generate\_step}(q_t)$0 vs $s_t \sim \text{Model.generate\_step}(q_t)$1; $s_t \sim \text{Model.generate\_step}(q_t)$250% cut	99.0% AddSub (GPT-4o-mini), $s_t \sim \text{Model.generate\_step}(q_t)$3 over CoT
MemCoT (Lei et al., 9 Apr 2026)	Long-context QA	$s_t \sim \text{Model.generate\_step}(q_t)$485% prompt size reduction	Overall F1 to 58.03 (GPT-4o-mini, LoCoMo)
MarCoS (Liu et al., 29 Sep 2025)	Latent reasoning	10–20× memory; $s_t \sim \text{Model.generate\_step}(q_t)$5 speedup	$s_t \sim \text{Model.generate\_step}(q_t)$6 GSM8K vs token CoT, up to 15.7× speedup
Markovian Transformer (Viteri et al., 2024)	LM, addition tasks	$s_t \sim \text{Model.generate\_step}(q_t)$7 context memory savings	Up to $s_t \sim \text{Model.generate\_step}(q_t)$8 accuracy improvement

Across benchmarks such as GSM8K, MATH, AddSub, LoCoMo, and LongMemEval-S, MCoT consistently achieves equal or superior accuracy with lower memory and computational costs. Backward verification and self-correction components further improve robustness and error detection.

6. Architectural Trade-offs and Limitations

• Memorylessness vs. Context: While enabling memory savings, strict Markov property can cause error propagation, as no backtracking is possible in vanilla MCoT. Reversible or hierarchical variants (e.g., CLoT) address this by adding backward verification and multi-tier checks (Zhang et al., 8 Apr 2026).
• Information Bottlenecks: Compressing the full previous context into $s_t \sim \text{Model.generate\_step}(q_t)$9 or $s_t$0 may discard subtle dependencies; advanced designs inject external retrieval, backward checks, or variational training to mitigate information loss (Liu et al., 29 Sep 2025).
• Code-based Self-Correction: Code execution error handling partially counteracts Markovian errors, but symbolic mistakes in pure text remain challenging (Yang et al., 2024).
• Parameter Efficiency: Some latent-based MCoT models (e.g., MARCOS) require larger parameter footprints to encode reasoning steps, trading off model size for working memory and speed (Liu et al., 29 Sep 2025).

7. Extensions and Future Directions

• Backward Reasoning and Pruning: CLoT’s reversible and hierarchical mechanisms support efficient, reliable chains while curbing error propagation and maximizing efficiency via dynamic pruning (Zhang et al., 8 Apr 2026).
• Hybrid Reasoning Frameworks: Integration with retrieval-augmented generation and recurrent inference networks promises to extend MCoT to very-long (multi-document) contexts (Lei et al., 9 Apr 2026).
• Flexible Informativeness Objectives: Markovian transformer approaches advocate learning maximally informative intermediate states for both interpretability and robustness (Viteri et al., 2024).
• Latent Chain-of-Thought: Greater theoretical and practical gains may arise from further compressing reasoning into continuous latent chains, decoupling “thinking” from “speaking,” and enabling step-level, reinforcement-driven reasoning (Liu et al., 29 Sep 2025).
• Backtracking and Monte Carlo Tree Search: Integrating MCTS or branching strategies may overcome the limitations of forward-only Markov chains and improve handling of uncertainty or ambiguity in sub-problem reduction (Yang et al., 2024).

MCoT frameworks are now foundational methods in scaling deep reasoning under memory constraints, with ongoing research into richer representational forms, backward reasoning, and hybrid architectures driving further advances (Yang et al., 2024, Zhang et al., 8 Apr 2026, Lei et al., 9 Apr 2026, Liu et al., 29 Sep 2025, Viteri et al., 2024).