---
title: 'PaFu-KV: Past & Future-Informed Cache Policy'
url: https://www.emergentmind.com/topics/past-and-future-informed-kv-cache-policy-pafu-kv
type: topic
---

# PaFu-KV: Past & Future-Informed Cache Policy

A Past- and Future-Informed KV Cache Policy (PaFu-KV) is a class of cache management strategies for transformer-based sequence models that selectively retains or evicts key–value (KV) memory entries based on both retrospective (past) and prospective (future) indicators of their semantic or functional importance. This policy synthesizes information from attention patterns, temporal reuse predictions, and model-internal salience scoring to minimize memory and bandwidth consumption during inference or generation, especially in tasks demanding long-context reasoning, multi-agent orchestration, or autoregressive video synthesis. PaFu-KV policies are realized via a diverse set of mechanisms—ranging from supervised and reinforcement learning to closed-form attention expectation computation and distillation-based salience estimation—with broad applicability across language, workflow, and video domains.

## 1. Motivation and Core Challenges

In transformer models, the KV cache linearly accumulates key and value tensors with each generated token or timestep, quickly resulting in substantial memory (e.g., 4.5 GB per instance at 32K tokens in Qwen3-4B) and compute overhead. Naïve retention (no eviction) ensures full context but severely limits throughput and feasible batch size. Uninformed or heuristic eviction (e.g., FIFO, LRU, or position-based thresholds) frequently discards tokens that, while seemingly unimportant immediately, can be essential for maintaining long-term dependencies, semantic coherence, or agentic control [2602.03203, 2507.07400, 2601.21896]. This is especially acute for:

- **Long-sequence reasoning and math models:** Complex dependencies across reasoning traces and head-specific semantic patterns cannot be captured by local recency/frequency alone.
- **Multi-agent workflows:** Hierarchical scheduling and data/control dependencies among agent invocations require anticipatory cache management to avoid recomputation [2507.07400].
- **Autoregressive video diffusion:** Sparse but persistent long-term spatiotemporal interactions between frames can be obscured by indiscriminate cache trimming, degrading temporal coherence [2601.21896].

A central challenge is the discrepancy between past and future importance—tokens largely ignored in past attention may be essential later, and purely forward-looking schemes are intractable or impossible in causal inference. PaFu-KV policies address this by integrating both retrospective and prospective signals.

## 2. Algorithmic Foundations and Principal Methods

### 2.1 Future-Informed Scoring: Golden Eviction and Expected Attention

Several works derive “future-informed” optimality by direct or estimated inspection of future attention:

- **Golden Eviction** computes an oracle eviction trace by leveraging future attention matrices across full (prompt + generation) traces. Each candidate KV pair is scored by the maximal pooled attention it will receive at any subsequent step; the policy retains the top-K pairs under a memory budget [2602.03203].
- **Expected Attention** [2510.00636] resolves the inaccessibility of future query vectors by modeling their distribution as Gaussian, then calculating the closed-form expected attention weight using the moment-generating function:
  $$
  \hat z_i = \exp\left(\frac{\bar\mu_q^\top k_i}{\sqrt d} + \frac{1}{2d} k_i^\top \bar\Sigma_q k_i \right)
  $$
  This yields a normalized attention proxy $\hat a_i$ for each key $k_i$ and enables compression without direct access to future tokens.

PaFu-KV generalizes these by combining such future-oriented proxies with past-statistics or by incorporating them in learning-based eviction policies.

### 2.2 Past-Informed Components

To capture retention value based on realized attention history, PaFu-KV also maintains:

- **Running or aggregated past attention scores:** E.g., exponentially moving averages or block maxima over $a_{t',i}$ (historical attention paid to key $i$) [2510.00636].
- **Statistics derived from kernel density estimation or frequency/recency predictions:** As in DEAP Cache, which models both future and historical cache characteristics in a multi-task LSTM pipeline [2009.09206].

The fusion may take the form $w_i = \alpha\bar a_i^{\text{past}} + (1-\alpha)\hat a_i^{\text{future}}$, with $\alpha$ hyperparameterized or head-adaptive.

### 2.3 Distillation and Salience Estimation

In non-language domains, PaFu-KV can be instantiated by distilling token- or frame-level salience from a bidirectional teacher into a lightweight student module:

- **Salience Estimation Head (SEH):** Trained to align its predicted salience $s_\theta$ with block-wise teacher attributions computed from full-sequence self-attention matrices; used to select the top-K tokens for cache retention [2601.21896].

## 3. Learning Frameworks: Supervision and Reinforcement

### 3.1 Supervised Distillation

Policies such as Golden Eviction are distilled into parameterized predictors (e.g., compact MLPs) which learn to estimate contribution scores or eviction eligibility using features extracted from KV tensors and attention-derived statistics [2602.03203]. Training employs pairwise ranking loss enforcing oracle-order equivalence among candidate KV pairs.

### 3.2 Online Policy Optimization

Eviction is also cast as a Markov Decision Process (MDP) to accommodate distributional drift between training (oracle labels) and actual inference dynamics:

- **States:** Current retained KV indices.
- **Actions:** Selection of which KV indices to keep under the budget.
- **Rewards:** Penalties such as mean-squared increase in modeling loss, particularly on low-entropy tokens (bottom 80%) which are most prone to loss spikes if crucial context is discarded.

The Guarded Regularized Policy Optimization (GRPO) algorithm is employed to refine the supervised policy, using clipped surrogate objectives and KL-divergence regularization for stable improvement [2602.03203]. Similar approaches appear in DEAP Cache with regret-minimization between frequency- and recency-based “experts” [2009.09206].

## 4. Structural and Computational Considerations

### 4.1 Data Structures

- **Radix-tree (trie) caches:** Employed in multi-agent workflows, where node- and subtree-based eviction priorities reflect the minimal steps-to-execution among all agents depending on the prefix [2507.07400].
- **Fine-grained, per-node eviction scores:** Calculated from anticipated “distance to reuse” and used to efficiently evict cache branches while maintaining coordination across concurrent workflows.

### 4.2 Complexity and Overhead

- Algorithmic steps, including priority queue updates and status tracking for overlapping CPU→GPU prefetch, are bounded by $O(\log N)$ per operation, with negligible metadata (<1% of raw KV usage).
- Scoring operations (e.g., MLP forward passes, expected-attention contraction) dominate the per-compression cost but are optimized to scale linearly or nearly linearly with cache size and model dimension [2510.00636, 2602.03203].

## 5. Applications and Quantitative Results

### 5.1 Language and Reasoning Models

PaFu-KV (as instantiated by ForesightKV) yields state-of-the-art efficiency-accuracy trade-offs: With only half the cache budget, it preserves ≥90% of full-context Pass@1 rates on math benchmarks (e.g., on Qwen3-4B + AIME2024: 1K PaFu-KV achieves 54.5 vs. 44.8 for 2K R-KV). Throughput improvements of up to 9.8× at 32K context are reported, with eviction overhead below 3% of total inference time [2602.03203].

### 5.2 Multi-Agent LLM Workflows

PaFu-KV, as implemented in KVFlow, realizes 1.8–2.9× reductions in prefill latency compared to LRU or HiCache, secures near-perfect cache hit rates for shared prefixes, and supports high concurrency with tunable prefetching aggressiveness [2507.07400].

### 5.3 Video Generation

In real-time AR video diffusion, PaFu-KV reduces KV cache size by 30–50%, with only a ~0.2% drop in generation quality, while improving metrics such as subject consistency (90.1% → 93.9%) and halving quality drift on long-horizon sequences [2601.21896].

### 5.4 General Cache Compression

Expected-Attention–based PaFu-KV methods, even in a training-free setting, closely track or surpass the uncompressed baseline across a variety of LLM architectures and benchmark suites (e.g., Qwen3-8B, Gemma3-12B), outperforming previous attention-score and value-norm heuristics at aggressive compression ratios [2510.00636].

## 6. Variants, Limitations, and Open Directions

- **Scoring model specificity:** Supervised and distillation-based policies are generally trained per model; cross-model transfer is an open question [2602.03203].
- **Limited gain under hardware bottlenecks:** When attention kernel memory costs are dwarfed by parameter size (e.g., with FlashAttention), further cache compression gives diminishing returns [2601.21896].
- **Dynamic head/layer allocation:** Static or globally uniform retention budgets may underutilize potential savings; adaptive, head-specific policy learning is suggested as a future avenue.
- **Extensions to multimodal/retrieval-augmented settings:** Handling heterogeneity in KV entry origin, modality, and reuse patterns remains a challenge [2602.03203].
- **Combination with speculative decoding and compression:** There is no published solution to end-to-end optimization uniting these axes, though synergistic approaches are plausible.

## 7. Summary Table: Key PaFu-KV Method Characteristics

| Method/Paper                                               | Salience Basis         | Learning Paradigm     | Primary Domain         |
|------------------------------------------------------------|-----------------------|----------------------|-----------------------|
| ForesightKV [2602.03203]                                   | Max future attention  | Supervised + RL      | Reasoning LLMs        |
| KVFlow [2507.07400]                                        | Steps-to-execution    | Heuristic/FM          | Multi-Agent LLM flows |
| Expected Attention + Past Proxy [2510.00636]               | Future expectation + Past mean | Training-free       | General LLM cache     |
| DEAP Cache [2009.09206]                                    | Frequency, recency    | Supervised + Hedging | General ML caches     |
| Salience Estimation Head [2601.21896]                      | Teacher-distilled     | Distillation         | Video diffusion       |

In sum, PaFu-KV denotes a family of cache eviction policies—rooted variously in supervised learning, reinforcement learning, closed-form probabilistic estimation, and distillation—that unify forward- and backward-looking signals to optimize cache footprint subject to maintained task performance. Their practical efficacy is established across language, agentic workflow, and video domains, with ongoing research addressing model adaptability, multimodal settings, and theoretical underpinnings.

Source: https://www.emergentmind.com/topics/past-and-future-informed-kv-cache-policy-pafu-kv