LazyEviction Mechanism for Efficient LLM Inference

• LazyEviction is a token recurrence-aware eviction policy that leverages empirical attention patterns to retain contextually vital tokens during extended chain-of-thought reasoning.
• It employs an observation-driven strategy with an adjustable window and composite scoring to update metadata and efficiently manage GPU memory usage.
• Experimental results demonstrate 50–70% cache reduction with near state-of-the-art accuracy on benchmarks like GSM8K and MATH-500, outperforming traditional methods.

LazyEviction is a dynamic, observation-driven key-value (KV) cache eviction policy for Transformer-based LLM inference, optimized for long-range, chain-of-thought (CoT) reasoning tasks. By leveraging empirical analyses of attention patterns—specifically, the recurrence of token importance—LazyEviction prioritizes retention and delayed eviction of contextually critical tokens. This mechanism achieves substantial GPU memory reduction while maintaining or exceeding state-of-the-art accuracy relative to alternative KV compression strategies (Zhang et al., 19 Jun 2025, Zeng et al., 2024).

1. Token Importance Recurrence: Motivation and Analysis

LazyEviction is motivated by the empirical observation that, during extended CoT reasoning, the importance of context tokens is non-monotonic. Tokens may become quiescent (i.e., receive low attention) for numerous decode steps and later regain high influence. This phenomenon, denoted Token Importance Recurrence (TIR), reveals that critical information may be cyclically re-accessed far downstream. Formally, for each token $i$, its Maximum Recurrence Interval (MRI) up to step $t$ is defined as

$$\mathrm{MRI}_t[i] = \max\left\{ \mathrm{MRI}_{t-1}[i], TS_t[i] - TS_{t-1}[i] \right\}$$

where $TS_t[i]$ is the last timestep at which token $i$ achieved attention above threshold $\alpha$.

Empirical distributions of MRI on datasets such as GSM8K and MATH-500 show that over 95% of tokens have $\mathrm{MRI} > 1$ and 80% have $\mathrm{MRI} < 175$ even for sequences exceeding 8000 tokens. This long-tail distribution underpins the risk in naive, locality-driven eviction and motivates a lagged, recurrence-aware approach (Zhang et al., 19 Jun 2025).

2. LazyEviction Framework and Algorithmic Implementation

The LazyEviction policy maintains two per-token metadata vectors in the KV cache: the timestamp vector $\mathbf{TS}_t$ and MRI vector $\mathbf{MRI}_t$. The core eviction protocol is structured around the concept of an observation window $t$0, which is set to the 80% quantile of empirical MRI statistics (e.g., $t$1 for GSM8K).

At each generation step:

• Tokens' attention activations are measured. If $t$2, then $t$3; otherwise, $t$4.
• MRI values are updated accordingly.
• Every $t$5 steps, an eviction phase is triggered if the cache size $t$6.
• The lagged eviction selection keeps the $t$7 most recent tokens and the top $t$8 tokens by a composite importance score $t$9, defined as:

$$\mathrm{MRI}_t[i] = \max\left\{ \mathrm{MRI}_{t-1}[i], TS_t[i] - TS_{t-1}[i] \right\}$$0

$$\mathrm{MRI}_t[i] = \max\left\{ \mathrm{MRI}_{t-1}[i], TS_t[i] - TS_{t-1}[i] \right\}$$1

This windowed, batch-wise approach requires only a single sort per $$\mathrm{MRI}_t[i] = \max\left\{ \mathrm{MRI}_{t-1}[i], TS_t[i] - TS_{t-1}[i] \right\}$$2 steps ($$\mathrm{MRI}_t[i] = \max\left\{ \mathrm{MRI}_{t-1}[i], TS_t[i] - TS_{t-1}[i] \right\}$$3), compared to per-step sorting in baseline methods.

3. Integration with Transformer KV Cache and Variants

LazyEviction is implemented as a lightweight metadata extension to the standard Transformer attention module:

• Attention computation proper remains unmodified.
• The eviction phase prunes the $$\mathrm{MRI}_t[i] = \max\left\{ \mathrm{MRI}_{t-1}[i], TS_t[i] - TS_{t-1}[i] \right\}$$4 arrays by index, maintaining only the set $$\mathrm{MRI}_t[i] = \max\left\{ \mathrm{MRI}_{t-1}[i], TS_t[i] - TS_{t-1}[i] \right\}$$5 specified by the lagged selection.
• No modifications to softmax or matmul are introduced; only tensor slice management and metadata updates are required.
• Integration points are naturally compatible with standard model APIs (e.g., HuggingFace's past_key_values).

A parameterized, learnable variant of LazyEviction is realized by the Attention-Gate (AG) architecture (Zeng et al., 2024), which trains a small, dedicated self-attention block to predict per-layer, per-head, per-token binary eviction flags. The AG module generates a mask based on global context features, producing a contiguous or sparsely indexed cache. During optimization, a custom sparsity penalty is incorporated into the loss to control the balance between memory reduction and accuracy retention.

4. Experimental Results and Comparative Performance

Empirical evaluation demonstrates that LazyEviction consistently yields compression ratios $$\mathrm{MRI}_t[i] = \max\left\{ \mathrm{MRI}_{t-1}[i], TS_t[i] - TS_{t-1}[i] \right\}$$6 of 30% to 50% (i.e., 50–70% cache reduction) without substantial loss of accuracy. For example, on GSM8K at $$\mathrm{MRI}_t[i] = \max\left\{ \mathrm{MRI}_{t-1}[i], TS_t[i] - TS_{t-1}[i] \right\}$$7, the DS-Llama-8B model achieves 80.06% accuracy with LazyEviction versus 81.73% with FullKV; competing approaches such as H2O, TOVA, and R-KV incur accuracy drops of several percentage points. On MATH-500, LazyEviction at $$\mathrm{MRI}_t[i] = \max\left\{ \mathrm{MRI}_{t-1}[i], TS_t[i] - TS_{t-1}[i] \right\}$$8 slightly exceeds baseline FullKV accuracy (Table below) (Zhang et al., 19 Jun 2025, Zeng et al., 2024).

Method	DS-Llama-8B	DS-Qwen-7B
FullKV	81.73%	89.92%
RaaS	78.01%	85.37%
H2O	77.16%	87.04%
TOVA	72.25%	80.52%
R-KV	78.69%	88.33%
LazyEviction	80.06%	88.40%

Table: Accuracy on GSM8K ($$\mathrm{MRI}_t[i] = \max\left\{ \mathrm{MRI}_{t-1}[i], TS_t[i] - TS_{t-1}[i] \right\}$$9) (Zhang et al., 19 Jun 2025)

The Attention-Gate mechanism demonstrates comparable gains, reducing KV-cache by $TS_t[i]$050–60% with minimal or even negative loss in zero-shot and supervised evaluation settings, and supports trainable adaptation to diverse tasks (Zeng et al., 2024).

5. Practical Implementation Guidelines

Effective deployment of LazyEviction requires careful calibration of the observation window $TS_t[i]$1 and attention threshold $TS_t[i]$2:

• $TS_t[i]$3 should be set as the 80th percentile of empirical MRI, determined over a stratified sample (typically 1%) of evaluation data. Task-specific values: $TS_t[i]$4 for GSM8K; $TS_t[i]$5 for MATH-500.
• $TS_t[i]$6 discriminates "important" tokens. Recommended values: $TS_t[i]$7 (Llama-style); $TS_t[i]$8 (Qwen-style). Setting $TS_t[i]$9 too low yields excessive activations and MRI underestimation; too high forfeits valid activations.
• Metadata storage is modest: two int32 arrays per KV pair.
• Eviction logic is invoked every $i$0 generation steps and custom fitted to the compute/memory budget $i$1.
• LazyEviction does not confer benefit when $i$2 is consistently small (e.g., in language modeling without CoT).

6. Comparative Complexity and Limitations

Compared to baseline methods (Current-Evict/TOVA: $i$3; Cumulative/H2O: $i$4 over $i$5 steps), LazyEviction minimizes sorting cost to a single $i$6 operation per window, with $i$7 lightweight timestamp and MRI updates. The additional memory overhead for metadata is negligible relative to full KV storage.

Limitations include the necessity for offline computation of MRI statistics for each model-task pair and lack of support for dynamic $i$8 adjustment in response to workload variation. LazyEviction's efficacy diminishes in contexts where TIR is rare.

7. Relationship to Attention-Gate and Trainable Lazy Eviction

The Attention-Gate framework (Zeng et al., 2024), which can also be viewed as a trainable instantiation of lazy eviction, introduces a parameterized gating module that computes eviction flags using a small, learnable self-attention per layer. This allows adaptation to new tasks or fine-tuning scenarios via continual pre-training or supervised fine-tuning, with negligible impact on compute overhead and strong control over the cache-to-accuracy trade-off. A plausible implication is broader applicability of recurrence-aware, observation-driven cache eviction policies across LLM architectures and tasks with long context dependencies.

---

References:

(Zhang et al., 19 Jun 2025) "LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning" (Zeng et al., 2024) "In-context KV-Cache Eviction for LLMs via Attention-Gate"