MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

Abstract: LLM agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a major memory bottleneck. Existing eviction policies generally apply the same attention- or recency-based rule to every token, ignoring semantic structure already available to the agent orchestrator. We introduce MemDecay, a training-free, region-aware KV-cache eviction policy. MemDecay assigns tokens region-specific base priorities and decay rates, refreshes retention scores when tokens receive attention, and evicts the lowest-scoring pages under a fixed cache budget while allowing critical regions to be pinned. We also provide a procedure for calibrating decay rates from measured attention lifetimes. We evaluate MemDecay at approximately 450 and 1,700 token contexts using Qwen2.5-1.5B and 3B. Across all settings, attention lifetimes differ by an order of magnitude across regions: system-token half-lives range from 148 to 189 decoding steps, compared with 14 to 16 for scratchpad tokens. Pinning preserves system-region facts at full-cache accuracy in every setting, while no baseline preserves more than 13 of 24. Region-aware retention remains effective as context grows, whereas recency-based retention collapses. Accumulated-attention retention performs better on unpinned content, however, and ablations identify attention-score normalization as the main limitation of the current formulation. These results establish semantic prompt structure as a robust signal for KV-cache management while clarifying how it should be combined with attention-based importance.

• The paper presents a novel region-aware eviction mechanism that assigns exponential decay and base priorities to tokens based on their semantic regions.
• It demonstrates that pinning critical regions like system instructions preserves essential context under strict memory constraints.
• Experimental results show improved recall and resource efficiency compared to traditional recency- or attention-based methods across varying context lengths and model scales.

MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

Problem Formulation and Motivation

MemDecay addresses the growing challenge of KV cache management in the deployment of LLM-based agents that accumulate long and heterogeneous execution traces. Traditional cache eviction strategies, such as those based on recency or global attention statistics, ignore the semantically diverse structure of agentic prompts—where system instructions, user turns, plans, tool outputs, retrievals, and intermediate reasoning are intermixed, but exhibit widely different optimal retention lifetimes. The uniform application of cache policies across functionally distinct prompt segments leads to resource waste and degraded quality of service in agentic, multi-turn, or tool-augmented LLM applications.

MemDecay proposes a region-aware, training-free eviction framework operational at inference time. It leverages orchestrator-provided region labels to distinguish semantic segments within the prompt, assigning each region a distinct base priority and temporal decay rate, then combines these “structural priors” with accumulated attention-derived importance to generate a composite retention score for each token. Pinning certain regions (e.g., system instructions) is supported, guaranteeing their survival during eviction.

Mechanism and Policy Design

The MemDecay retention score for each cache token is a sum of two terms: a region base priority that decays exponentially over the elapsed steps since last high attention, and a term proportional to the exponentially moving average (EWMA) of model attention received by the token during decoding. The decay rate and base priority are per-region hyperparameters, calibrated via attention lifetime measurements from agent traces. Whenever the fixed cache budget is exceeded, eviction is performed at page granularity, removing pages with the lowest average retention score, while always preserving pinned pages.

Formally, for token $i$ with region $r(i)$, the retention score at time $t$ is:

$$s_i(t) = b_{r(i)} \exp\big(-\lambda_{r(i)} \cdot \Delta t_i(t)\big) + \alpha\, a_i(t)$$

where $b_{r(i)}$ and $\lambda_{r(i)}$ are the base priority and decay rate of region $r(i)$, $\Delta t_i$ is token age since insertion or last attended (if exceeding threshold $\tau$), $a_i(t)$ is the EWMA attention, and $r(i)$0 is tunable.

MemDecay’s score reflects the empirical finding—contrary to many prior works—that semantic regions in agent traces exhibit highly divergent attention/reuse profiles: for example, system instruction tokens have order-of-magnitude longer attention half-lives than scratchpad or reasoning-token regions. This insight is operationalized by region-specific base and decay rates.

Experimental Analysis and Findings

MemDecay is evaluated on multi-stage, agent-style prompts traversing two tiers of context length (∼450 tokens, ∼1700 tokens) and two scales of open-source LLMs (Qwen2.5-1.5B and Qwen2.5-3B). The study is query-agnostic, i.e., evidence is compressed without knowledge of the upcoming probe, which is a notably stringent regime.

Region-conditioned attention lifetimes differ by up to an order of magnitude: system-region tokens exhibit half-lives of 148–189 steps, whereas scratchpad tokens have half-lives of 14–16 steps, robust across insertion order permutations and model scales. Retrieval regions, often assumed short-lived, show unexpectedly persistent attention, invalidating the naive classification favored by default heuristics.

Under tight budgets, strong claims are validated:

• Pinning system regions guarantees perfect recall of critical instructions: every system fact is retained at the full-cache ceiling for both short and long contexts, a property no other baseline (attention-based, recency-based, or random) matches.
• Conventional recency-based retention collapses with growing context, losing nearly all non-pinned content in long traces.
• Attention-based policies (H2O-style) improve with model scale, often outperforming MemDecay for unpinned user facts, since MemDecay’s aggressive decay prioritizes newer or structurally-protected tokens.

An ablation varying the attention term’s weight shows that it cannot rescue early tokens from being evicted: magnitude mismatch between the region-prior and attention contributions necessitates explicit normalization (e.g., with respect to uniform attention), and the page-based mean further dilutes outlier tokens. Thus, for older unpinned facts (notably user-region content at the head of the trace), attention is insufficient to override the structural expiration, and recall is low.

Computational overhead is minimal in the retention logic, but depends on attention statistics sampling; in real-world applications, attention sampling would likely use a larger stride or be repurposed from auxiliary computations.

Implications and Theoretical Considerations

MemDecay exposes and encodes the structural prior present in agent episodes. Its region-aware decay optimizes cache utility in LLM agent inference subject to memory constraints, particularly in agentic workflows that repeatedly interleave persistent background instructions with episodic evidence and ephemeral tool interactions.

The irreversibility inherent in current cache eviction schemes (once a token is evicted it cannot be recovered) remains a limitation identically shared by all non-tiered policies. The paper outlines possible mitigations, including tiered degradation—first quantizing or offloading evicted tokens before irreversible removal, enabling future recovery if subsequent queries demand it.

MemDecay’s efficacy, like any region-aware policy, is partly contingent on the availability and correctness of region labels. While orchestrators are trending toward exposing such metadata, deployment in less structured pipelines may require robust region classifiers, whose performance has yet to be evaluated.

Workloads with different cross-region copying behaviors, or with region semantics not tightly tied to token order, may challenge the assumptions underpinning MemDecay’s configuration, necessitating further calibration or more expressive conditioning (e.g., dependency-aware retention within plans and tool outputs).

Future Directions

Several important technical avenues remain open:

• Magnitude normalization of attention for unpinned recall, and online adaptation of decay rates by region/activity.
• Tiered retention, where “eviction” is replaced by progressive degradation (quantization, transfer to cold storage, etc.) prior to irrecoverable drop, is anticipated to improve recall in the presence of late-arising importance for old tokens.
• Scaling to production workloads: evaluating on 8B+ LLMs, context windows exceeding 4k tokens, and using traces derived from real-world benchmarks (e.g., SWE-bench, LongBench).
• Classifiers for region inference to support deployment in orchestrators lacking explicit region labeling.

Conclusion

MemDecay demonstrates that agent-specific, region-aware cache management for LLMs significantly outperforms recency- or global-attention-based schemes in realistic agent inference scenarios, particularly in preserving critical context under memory pressure. The method’s ability to guarantee the survival of protected regions while prioritizing retention according to empirically calibrated attention lifetimes represents a major advance in practical memory management for agentic LLM deployment. Remaining limitations—chiefly irreversibility and the need for attention normalization—identify clear opportunities for integration with tiered and dependency-aware cache management strategies.