---
title: 'EpiCache Framework: Memory-Efficient KV Cache Management'
url: https://www.emergentmind.com/topics/epicache-framework
type: topic
---

# EpiCache Framework: Memory-Efficient KV Cache Management

EpiCache is a training-free Key-Value (KV) cache management framework designed to enable efficient, long-context conversational question answering (LongConvQA) with large language models (LLMs) under strict and constant memory budgets. The framework addresses the unbounded, linear growth of KV caches during multi-turn, extended conversations by combining three core innovations: block-wise prefill eviction, episodic KV compression, and adaptive layer-wise budget allocation. EpiCache maintains near-full-KV accuracy—retaining up to 95% of baseline accuracy under 4–6× compression—while drastically reducing GPU memory and decoding latency [2509.17396].

## 1. Problem Formulation: KV Cache Growth in LongConvQA

Autoregressive LLMs use a KV cache at each transformer layer to store past attention keys and values, such that for a dialogue of $N$ tokens, $L$ layers, and $H$ heads, the cache scales as $\|KV\| = L \times H \times N$. In LongConvQA settings, where dialogue length $N$ becomes very large (hundreds of turns or multi-day sessions), the memory required for storing KV caches quickly exhausts available GPU RAM: $M_{\text{peak}} \approx O(N)$. Traditional approaches like post-prefill eviction or query-dependent cache compression either induce unbounded peak memory or collapse relevant context, leading to degraded multi-turn accuracy. The goal is to satisfy, under a strict per-layer token budget $M$, that for all queries $q_i$,
$$
f_{\text{LLM}}(q_i \mid KV_H^C) \approx f_{\text{LLM}}(q_i \mid KV_H),
\quad \text{with} \quad \|KV_H^C\| \leq L \times H \times M,
$$
throughout prefill and decode, thus bounding peak memory across arbitrarily long conversations [2509.17396].

## 2. EpiCache Framework Overview

EpiCache constrains KV cache growth using a combination of block-wise prefill eviction, episodic KV compression, and adaptive layer-wise budget allocation:
- **Block-wise Prefill Eviction** processes the conversation in blocks of size $B$, evicting cache entries to maintain size $M+B$ at each block boundary.
- **Episodic KV Compression** segments the conversation into $E$ topical episodes via clustering, building separate compressed KV caches for each.
- **Adaptive Layer-wise Budget Allocation** assigns memory budget per layer according to measured sensitivity to key eviction, preserving representational integrity where most critical.

This ensemble of techniques allows EpiCache to cap GPU memory, preserve topic-relevant context, and satisfy the requirements of multi-turn dialogue systems without retraining [2509.17396].

## 3. Key Algorithms and Techniques

### 3.1 Block-wise Prefill Eviction

Block-wise prefill divides the input into non-overlapping blocks of $B$ tokens. After each prefill, token importance scores $s_i$ are computed (method in Section 4), and only the top-$M$ tokens by $s_i$ are retained; the rest are evicted. The process guarantees that the peak cache never exceeds $M+B$, regardless of the total dialogue history length.

Pseudocode summary:

```
Input: history H, total tokens N, budget M, block size B
KV_cache ← ∅
for t in 1…N step B:
    prefill B tokens x_{t:t+B-1}, update KV_cache
    compute s_i for tokens in KV_cache
    keep top-M by s_i
end for
```

This method ensures a flat GPU memory profile as $N$ increases, essential for deployment on resource-constrained hardware [2509.17396].

### 3.2 Episodic KV Compression

#### 3.2.1 Clustering into Episodes

Dialogue history $H$ is partitioned into $K$ segments $\{S_k\}$, each with $w_{\text{embed}}$ utterances. Segment embeddings $e_k$ are computed using a lightweight encoder (e.g., MiniLM, Qwen3). $K$-Means clusters $\{C_e\}_{e=1}^E$ are formed, and each cluster’s medoid is selected as the representative prompt $P_e$.

#### 3.2.2 Episode-Specific Eviction

For each episode $e$, block-wise prefill is rerun over the full history, appending $P_e$ after each block to bias attention scoring toward the episode’s topic. Token importance $s_i$ is computed as
$$
s_i = \max_{t \in \text{patch}} Attn(x_t \rightarrow x_i),
$$
ensuring selection of tokens most relevant to the corresponding episode. The top-$M$ tokens are retained to compose the episode’s compressed cache $KV^C_e$.

#### 3.2.3 Online Retrieval and Decoding

At inference, each query $q_i$ is embedded ($u_i = f_{\text{embed}}(q_i)$) and matched to the closest episode centroid. The associated compressed cache $KV^C_{\hat{e}}$ is loaded into GPU and used during decoding.

### 3.3 Adaptive Layer-wise Budget Allocation

Layer sensitivity to key eviction is determined by measuring representational drift:
$$
s_l = 1 - \frac{1}{H \cdot N} \sum_{h=1}^H \sum_{i=1}^N \cos(K^{\text{full}}_{l,h,i}, K^{\text{block}}_{l,h,i}),
$$
where $K^{\text{full}}_{l,h,i}$ and $K^{\text{block}}_{l,h,i}$ denote key states with and without block-prefill eviction, respectively. Sharpening sensitivities via exponent $\alpha$ ($\alpha \approx 1.1$–$1.3$), the total KV token budget is allocated as
$$
M_l = \frac{s_l^\alpha}{\sum_{j=1}^L s_j^\alpha} \cdot M \times L,
$$
assigning larger capacities to layers with higher sensitivity and thus maintaining QA performance [2509.17396].

## 4. Theoretical Guarantees and Empirical Results

- **Memory Guarantee**: Peak KV cache size is always $\leq M + B$, ensuring constant GPU memory usage $O(M)$ independent of conversation length.
- **Empirical Accuracy Bound**: Under compression ratios of $4$–$6\times$ ($M \approx B \approx N/6$), EpiCache achieves $>95\%$ of full-KV QA accuracy across benchmarks.
- **Latency and Memory Reduction**: On LLaMA-3.2-3B (at $M = 4$K), per-turn latency is reduced by up to $2.4\times$ (from $68.9$ ms to $28.1$ ms), and peak memory by up to $3.5\times$ (from $28.4$ GB to $8.2$ GB).
- **Scalability**: For contexts up to $100$K tokens ($M = 6$K), EpiCache approaches uncompressed performance, outperforming StreamingLLM, SnapKV, InfiniPot, KeyDiff, and KVzip-adapted baselines, which degrade at scale.

The following table summarizes benchmark results for EpiCache ($M=4$K, LLaMA-3.2-3B):

| Benchmark         | Accuracy Gain over Baseline | Compression Ratio | Latency Reduction (×) | Memory Reduction (×) |
|-------------------|----------------------------|-------------------|----------------------|----------------------|
| Realtalk, LoCoMo, LongMemEval | +20–40% absolute           | 4–6×             | up to 2.4           | up to 3.5           |

[2509.17396]

## 5. Implementation Details and Practical Considerations

- **Block Size $B$**: Larger $B$ yields faster coverage at the expense of higher temporary memory; smaller $B$ minimizes peak but slows prefill.
- **Episode Count $E$**: Higher $E$ (more episodes) can increase recall under tight memory but raises offline cache storage requirements.
- **Embedding Strategies**: Accuracy is not highly sensitive to the embedding model; compact encoders suffice for practical segmentation.
- **Retrieval Overhead**: Embedding, centroid matching, and KV loading contribute to $<5\%$ of per-turn latency, as episode switches are infrequent.
- **Limitations**: Fixed episode count $E$ and offline clustering; no support for dynamic $E$ or compressed/quantized KV storage. Performance depends on clustering quality and selection of $w_{\text{embed}}$.

## 6. Related Work

Contemporary baselines such as StreamingLLM, SnapKV, InfiniPot, KeyDiff, KVzip, and context manipulation techniques (e.g., PrefixLM, position-independent caching as in EPIC [2410.15332]) also target memory-constrained LLM inference. Unlike query-dependent strategies that collapse all context to a single query (often harming multi-turn coherence), EpiCache preserves topic- and episode-relevant context and amortizes cache management across episodes, achieving superior QA accuracy and resource efficiency.

## 7. Limitations and Future Directions

Current constraints of EpiCache include the static selection of episode count $E$, reliance on offline clustering, and the lack of quantized/compact KV representations. Future research directions proposed include dynamic adaptation of $E$, integration of KV-quantization schemes, and improvements in conversational segmentation. These extensions may further reduce memory requirements and enable finer-grained, context-adaptive compression without sacrificing QA fidelity [2509.17396]. A plausible implication is that incorporating quantization or dynamic clustering could push the efficiency-accuracy frontier further in long-context LLM serving.

Source: https://www.emergentmind.com/topics/epicache-framework