Reasoning Cache in Transformer Inference

Updated 4 July 2026

Reasoning cache is a term that denotes a compact, reusable intermediate state essential for maintaining coherent multi-step inference in transformers and agent systems.
Techniques such as decode-time KV compression, semantic reuse, and structured memory updates are applied to balance performance, memory efficiency, and reasoning accuracy.
Empirical results demonstrate that selective caching minimizes redundancy, boosts throughput, and preserves chain-of-thought integrity even in long decoding sequences.

Searching arXiv for papers on reasoning-aware KV cache and related "reasoning cache" methods. “Reasoning cache” is a research term used in several closely related senses to denote reusable intermediate state for multi-step inference. In transformer inference, it most often refers to a KV cache treated not as a literal record of every past token, but as a constrained memory that should preserve the non-redundant backbone of a chain of thought; in multi-branch inference, it refers to shared or truncated computation across semantically similar branches and verification passes; in agentic systems, it denotes structured intermediate representations that are maintained and reused instead of repeatedly re-deriving the same latent or symbolic state (Cai et al., 30 May 2025, Sekar, 7 Jun 2026, Chillara et al., 22 Jan 2026, Lv et al., 16 Apr 2026).

1. Conceptual scope

Recent papers use “reasoning cache” in distinct but convergent ways. In decode-time KV compression, the cache is the minimal subset of hidden states sufficient to preserve coherent reasoning, key intermediate results, and answer formation. In multi-branch pipelines, the cache becomes shared reasoning context across branches or phases. In agent frameworks, cached objects need not be neural KV tensors at all; they may be structured intents, extracted facts, or other intermediate artifacts that stabilize downstream reasoning (Cai et al., 30 May 2025, Sekar, 7 Jun 2026, Chillara et al., 22 Jan 2026).

Interpretation	Representative mechanism	Cached object
Decode-time reasoning memory	R-KV, ReasonAlloc, CASK	Selected KV states from CoT
Cross-branch or cross-phase reuse	RKSC, MemShare, RedVisor	Prefix KV, branch states, verification states
Structured agent memory	SemanticALLI, IE-as-Cache	Intents, extracted records, visualization directives
KV as reusable representation	KV-CoE, KVClassifier	Pooled KV-derived features

A recurrent misconception is that reasoning cache is merely a synonym for ordinary prefix caching. The literature is more specific. Exact-prefix reuse in serving systems such as vLLM or SGLang is one boundary case, but several papers explicitly move beyond lexical identity toward semantic reuse, selective preservation of reasoning-critical state, or dynamic externalized memory (Sekar, 7 Jun 2026, Chillara et al., 22 Jan 2026).

2. Why long reasoning makes caching a first-class problem

In decoder-only transformers, the KV cache grows with sequence length because every new token appends keys and values that later queries must attend to. For reasoning models, this growth is driven primarily by decoding rather than prefilling: DeepSeek-R1-Distill-Llama-8B is reported to generate up to 32K tokens on math problems, with 4.1 GB of KV cache on top of 15.5 GB for weights, while MATH-500 and AIME-24 generations are reported to be 8–14 times longer than the ground-truth solutions and to exhibit 5–7 times higher unigram/bigram repetition (Cai et al., 30 May 2025).

This long-output regime differs from the classical long-prompt regime. Several papers argue that methods designed for long prompts and short generations do not transfer cleanly to chain-of-thought decoding. Attention-only token selection tends to preserve clusters of repeated reflective text, because semantically repetitive sentences strongly attend to each other, while sparse but crucial intermediate steps may be evicted (Cai et al., 30 May 2025). LongFlow makes the same systems-level point from another direction: prior KV optimization largely assumed long-input, short-output workloads, whereas reasoning models are dominated by long decoding and continuous cache growth (Su et al., 12 Mar 2026).

This shift changes both the objective and the failure mode. The objective is no longer simply reducing memory. It is preserving the latent state that carries the reasoning process forward. The failure mode is not merely reduced retrieval fidelity, but degraded chain-of-thought integrity: loops, overlong traces, broken verification, or loss of intermediate constraints.

3. Decode-time compression as reasoning-cache construction

A major line of work treats the reasoning cache as a compressed KV set optimized for ongoing chain-of-thought. R-KV formalizes this by scoring each token with a joint importance–redundancy objective,

$Z_i^h = \lambda I_i^h - (1-\lambda) R_i^h,$

where importance comes from recent-token attention and redundancy from cosine similarity among key states. The method keeps a fixed cache budget plus a small buffer, always preserves the last $\alpha$ “observation tokens,” and retains the highest-scoring non-redundant states. On mathematical reasoning, it preserves near-100% of full-KV performance using only 10–34% of the KV cache and reaches about 105% of full-KV accuracy at roughly 16–33% cache, with about 90% memory saving and 6.6 $\times$ throughput at 16K generation length in the reported Llama3-8B setting (Cai et al., 30 May 2025).

ReasonAlloc argues that uniform budgets across layers and heads are themselves a bottleneck. It introduces an offline layer-wise allocation based on an architecture-specific “Reasoning Wave” and an online head-wise reallocation based on current utility. The reported gains are largest at small budgets such as 128–512 tokens, and the method adds negligible overhead while outperforming uniform-budget R-KV, SnapKV, and Pyramid-RKV on MATH-500 and AIME 2024 (Liu et al., 9 Jun 2026).

Other work broadens the scoring signal. InfoKV introduces “Forward Influence” as an idealized forward-looking criterion and then approximates long-range usefulness with predictive entropy and layer-wise representation evolution, combining these with attention in a mixed score. The paper reports that InfoKV consistently outperforms attention-only baselines in both long prefilling and long decoding, and on IFEval some compressed settings surpass full-cache performance (Kai et al., 25 Jun 2026). LongFlow instead derives an importance proxy directly from the current attention computation,

$\text{LongFlowScore}(t_i)=\|\alpha_t^i v_i\|_1,$

and fuses attention, importance estimation, and eviction into a single Triton kernel; it reports up to 11.8 $\times$ throughput improvement with 80% KV compression and minimal accuracy impact (Su et al., 12 Mar 2026).

A second trend deemphasizes scalar token ranking in favor of structure. CASK argues that scorer refinement often fails to meaningfully reorganize the keep-set and reframes decode-time compression as preserving a protected core while selectively consolidating redundant scratch tokens. Kara likewise replaces threshold-triggered global compression with sliding-window compression over recent context, scores tokens with bidirectional attention, and expands selected tokens into flexible chunks through Token2Chunk; it reports consistent performance gains and better throughput behavior in its KvLLM serving stack (Kim et al., 13 Apr 2026, Han et al., 1 May 2026).

Head- and thought-level policies push this further. RLKV uses reinforcement learning to identify reasoning-critical heads and allocates full KV only to them, reporting 20–50% cache reduction with near-lossless performance (Du et al., 9 Oct 2025). ThinKV classifies chain-of-thought into reasoning, execution, and transition segments from attention sparsity, assigns thought-dependent quantization levels, and progressively evicts older segments according to a retention schedule; it reports near-lossless accuracy with less than 5% of the original KV cache and up to 5.8 $\times$ higher throughput over state-of-the-art baselines (Ramachandran et al., 1 Oct 2025).

4. Reuse across branches, passes, and repeated thought patterns

A second research direction treats reasoning cache as reusable computation across multiple reasoning paths rather than as a compressed single-trace memory. RKSC targets multi-branch pipelines such as self-consistency and verifier-based decoding. Its ASKS module performs semantic prefix caching by comparing branch and root hidden states with an exponentially weighted cosine similarity, its CGEE module skips verification when confidence is decisive or exits verification early when per-layer entropy stabilizes, and its RSBCM module manages block eviction by branch score and depth. Across five model families and four benchmarks, RKSC reports a mean 3.008 $\times$ speedup over a No-KV baseline, a 1.66 $\times$ improvement over vLLM-equivalent prefix caching, and a CGEE-induced error rate of 0.37% (Sekar, 7 Jun 2026).

MemShare addresses a different redundancy source: repeated intermediate reasoning steps within long chain-of-thought. It first identifies lexically similar steps by a bag-of-words cosine filter, then verifies block-level KV proximity with normalized Euclidean distance, and finally performs zero-copy block reuse through PagedAttention block tables. The reported throughput improvement reaches 84.79%, with better accuracy than eviction-based baselines at comparable affected ratios (Chen et al., 29 Jul 2025).

RedVisor shows that this reuse principle also applies to multi-phase safety pipelines. Its adapter generates an explicit reasoning path for prompt-injection localization in phase one, then the backbone answers in phase two with the adapter muted while reusing the same KV cache without a second prefill. The paper reports that unified RedVisor is more than twice as fast as a decoupled two-GPU version while preserving benign-task utility and improving attack resistance (Liu et al., 2 Feb 2026).

A related but more representational view appears in work that treats the KV cache as a free feature substrate. “Beyond Speedup -- Utilizing KV Cache for Sampling and Reasoning” shows that pooled KV features can support KV-CoE for output-free self-evaluation and a KVClassifier for fast/slow thinking switching, reducing token generation by up to 5.7 $\times$ with minimal accuracy loss on the reported settings (Xing et al., 28 Jan 2026). This suggests that reasoning cache can mean not only stored context for future attention, but also a reusable summary of the model’s internal reasoning trajectory.

5. Structured reasoning caches beyond neural KV

In agentic systems, the same design principle appears with symbolic or semi-structured artifacts instead of transformer KV tensors. SemanticALLI decomposes dashboard generation into Analytic Intent Resolution and Visualization Synthesis, then caches the resulting intermediate representations rather than caching only final responses. On its production-like evaluation, monolithic prompt-level caching saturates at a 38.7% hit rate, whereas the Visualization Synthesis stage reaches an 83.10% hit rate, bypasses 4,023 LLM calls, and serves hits with a median latency of 2.66 ms; the reported per-prompt token cost drops from about 59,906 to about 12,964 tokens (Chillara et al., 22 Jan 2026).

IE-as-Cache makes the same move in long-text agentic reasoning. It first generates a query-conditioned extraction schema, then extracts structured records into a cache, and subsequently alternates between reasoning over the cache and issuing seek_information actions that trigger further extraction and cache updates. The cache is explicitly treated as a compact, updateable working memory rather than a one-shot extraction product. On GPT-4o, the method reports 71.77 exact match on TACT versus 57.26 for ReAct and 61.29 for an IE-as-Tool baseline; removing the dynamic cache update drops TACT to 65.32, indicating that cache maintenance, not merely extraction quality, is central to the gain (Lv et al., 16 Apr 2026).

These systems broaden the meaning of reasoning cache. The common abstraction is no longer “KV compression,” but “persist the intermediate state that downstream reasoning repeatedly needs, and make that state small, structured, and updateable.”

6. Empirical regularities, misconceptions, and open problems

Across these papers, several empirical regularities recur. First, long chain-of-thought is strongly overcomplete. R-KV’s near-lossless operation at 10–34% KV budget, ThinKV’s reported near-lossless operation at under 5% of the original KV cache, and InfoKV’s occasional gains over full cache all suggest that storing every generated thought verbatim is unnecessary and can even be counterproductive (Cai et al., 30 May 2025, Ramachandran et al., 1 Oct 2025, Kai et al., 25 Jun 2026). A plausible implication is that compression sometimes acts as test-time regularization by removing redundant or distracting self-reflection.

Second, the useful unit of caching is heterogeneous. ReasonAlloc’s layer/head budgets, RLKV’s reasoning-critical heads, ThinKV’s thought types, and CASK’s core-versus-scratch partition all reject the assumption that layers, heads, tokens, or segments contribute uniformly to reasoning quality (Liu et al., 9 Jun 2026, Du et al., 9 Oct 2025, Ramachandran et al., 1 Oct 2025, Kim et al., 13 Apr 2026).

Third, exact lexical identity is often too weak a notion of reuse. RKSC’s semantic prefix sharing, MemShare’s similarity-based block reuse, and SemanticALLI’s IR-level reuse all show that repeated reasoning structure survives paraphrase or surface variation (Sekar, 7 Jun 2026, Chen et al., 29 Jul 2025, Chillara et al., 22 Jan 2026). Conversely, a common misconception is that better scalar scoring alone is the central problem. CASK explicitly challenges this, arguing that effective compression depends less on elaborating scorers than on preserving a protected core and consolidating scratch structure (Kim et al., 13 Apr 2026).

The limitations are equally consistent. Many methods are benchmarked mainly on mathematical reasoning and a few model families, so safe budgets may differ for code generation, dialogue, or multimodal reasoning (Cai et al., 30 May 2025, Liu et al., 9 Jun 2026). Several systems require nontrivial serving-stack integration, especially when compression must coexist with paged attention, custom kernels, or block managers (Han et al., 1 May 2026, Chen et al., 29 Jul 2025). Thresholds and calibration are often model-sensitive, as seen in RKSC’s semantic and confidence gates, InfoKV’s entropy settings, and ThinKV’s sparsity thresholds (Sekar, 7 Jun 2026, Kai et al., 25 Jun 2026, Ramachandran et al., 1 Oct 2025). Prompt-heavy regimes remain difficult: CASK describes a prefix-budget-exhausted boundary, and prefix handling remains a distinct systems problem from decode-stage reasoning preservation (Kim et al., 13 Apr 2026).

Taken together, the literature defines reasoning cache not as a single mechanism but as a design principle: preserve or reuse the intermediate state that actually carries reasoning, and do so with policies aligned to reasoning structure rather than with undifferentiated token retention. In neural form, that principle yields core-preserving, redundancy-aware, semantics-aware KV management. In agentic form, it yields structured intermediate memory that can be updated and reused across steps. The unifying claim is that efficient reasoning depends less on retaining everything the model ever produced than on retaining the specific state that keeps the reasoning trajectory coherent.