EpiCache: Episodic KV Cache Management for Long Conversational Question Answering (2509.17396v1)

Abstract: Recent advances in LLMs have extended context lengths, enabling assistants to sustain long histories for coherent, personalized responses. This ability, however, hinges on Key-Value (KV) caching, whose memory grows linearly with dialogue length and quickly dominates under strict resource constraints. An active line of research for reducing this overhead is KV cache compression, which seeks to limit cache size while preserving accuracy. Yet existing methods face two major limitations: (i) evicting entries after full-context prefill causes unbounded peak memory, and (ii) query-dependent eviction narrows the cache to a single query, leading to degraded accuracy in multi-turn conversations. We introduce EpiCache, a training-free KV cache management framework for long conversational question answering (LongConvQA) under fixed memory budgets. EpiCache bounds cache growth through block-wise prefill and preserves topic-relevant context via episodic KV compression, which clusters conversation history into coherent episodes and applies episode-specific KV cache eviction. We further design an adaptive layer-wise budget allocation strategy that measures each layer's sensitivity to eviction and distributes the memory budget across layers accordingly. Across three LongConvQA benchmarks, EpiCache improves accuracy by up to 40% over recent baselines, sustains near-full KV accuracy under 4-6x compression, and reduces latency and memory by up to 2.4x and 3.5x, thereby enabling efficient multi-turn interaction under strict resource constraints.

• The paper demonstrates that episodic KV cache management bounds memory growth and achieves up to 40% higher accuracy in long conversational QA tasks.
• The paper introduces a block-wise prefill and episodic clustering method to retain semantically relevant context while compressing the KV cache.
• The paper employs sensitivity-aware layer-wise budget allocation to reduce KL divergence and lower decoding latency and peak GPU memory usage.

Episodic KV Cache Management for Long Conversational QA: An Analysis of EPICACHE

Introduction

The EPICACHE framework addresses a central challenge in deploying LLM-based conversational agents for long-term, multi-turn interactions: the prohibitive memory growth of the Key-Value (KV) cache. As LLMs extend their context windows to hundreds of thousands or millions of tokens, the linear scaling of KV cache size with context length becomes a critical bottleneck, especially under resource constraints. Existing KV cache compression methods either fail to bound peak memory (post-prefill eviction) or degrade accuracy in multi-turn settings (query-dependent eviction). EPICACHE introduces a training-free, block-wise, and episodic KV cache management strategy that maintains a fixed memory budget while preserving topic-relevant context, enabling efficient and accurate Long Conversational Question Answering (LongConvQA).

Problem Formulation and Limitations of Prior Work

LongConvQA requires models to answer sequences of queries grounded in extended conversational histories, often spanning hundreds of turns and multiple sessions. The KV cache, which stores the Key and Value states for each token, grows linearly with the number of tokens, quickly exceeding available memory in practical deployments. Prior approaches to KV cache compression fall into two categories:

• Post-prefill eviction: Compression is applied after the entire context is prefetched, resulting in unbounded peak memory usage during prefill.
• Query-dependent eviction: The cache is pruned based on the current query, which narrows the retained context to a single query and degrades performance in multi-turn conversations.

Both approaches are inadequate for real-world LongConvQA, where memory must be strictly bounded and conversational context must be preserved across multiple turns.

EPICACHE: Methodology

EPICACHE introduces three key innovations:

1. Block-wise Prefill with Episodic Clustering

• Block-wise prefill: The input is processed in fixed-size blocks. After each block, eviction is performed to reduce the cache back to the memory budget $M$, ensuring that peak memory never exceeds $M + M_{block}$.
• Episodic clustering: The conversation history is segmented and clustered into semantically coherent episodes using sentence embeddings and K-means. Each episode is represented by a medoid segment, which serves as a patched prompt for guiding cache eviction.

2. Episodic KV Cache Compression

• For each episode, block-wise prefill is performed with the medoid segment appended as a patched prompt. Attention scores with respect to the patched prompt are used to select the most relevant tokens for retention in the episodic KV cache.
• All episodic caches are stored offline. At inference, the incoming query is embedded and matched to the closest episode centroid, and the corresponding episodic cache is retrieved for answer generation.

3. Sensitivity-Aware Layer-wise Budget Allocation

• Layer-wise sensitivity to block prefill eviction is measured by comparing Key state deviations under full and block-prefill masks.
• The global KV cache budget is distributed across layers in proportion to their measured sensitivity, with a sharpness hyperparameter controlling the allocation profile.
• This approach empirically reduces the KL divergence between block-prefill and full-KV predictions and consistently improves LongConvQA accuracy over uniform or pyramid-shaped allocations.

Implementation Details

Clustering and Embedding

• Conversation segments are embedded using lightweight sentence encoders (e.g., Qwen3-0.6B).
• K-means++ initialization is used for clustering, and the medoid segment of each cluster is selected as the representative patched prompt.

Block-wise Prefill and Eviction

• The block size $M_{block}$ is a tunable parameter, balancing memory and throughput.
• Patched prompts are used only for scoring and not retained in the cache.
• Token importance is computed via cross-attention from the patched prompt, and the top $M$ tokens are retained.

Episodic Cache Retrieval

• Episodic caches are stored in offline memory (e.g., CPU) and loaded to GPU as needed.
• Query-to-episode matching is performed via cosine similarity in the embedding space.
• Retrieval overhead is minimal, as episode switches are infrequent in natural conversations.

Layer-wise Budget Allocation

• Sensitivity is profiled once per model using a calibration batch.
• The allocation is static and reused across all experiments, incurring negligible overhead.

Empirical Results

Accuracy and Efficiency

• On three LongConvQA benchmarks (Realtalk, LoCoMo, LongMemEval) and four LLMs (LLaMA-3.2-3B, LLaMA-3.1-8B, Qwen2.5-3B, Qwen2.5-7B), EPICACHE achieves up to 40% higher accuracy than recent baselines under tight memory budgets (2K–4K tokens).
• Under 4–6x compression, EPICACHE sustains accuracy close to full KV cache.
• Decoding latency is reduced by up to 2.4x and peak GPU memory by up to 3.5x compared to full KV caching.
• The additional overhead from query embedding and cache retrieval is less than 5% of per-turn latency.

Ablation and Alternative Designs

• RAG-like approaches that directly input clustered segments with the query perform substantially worse than block-prefill-based episodic caching.
• The method is robust to segmentation window size, encoder choice, and number of medoids per episode.
• Increasing the number of episodes improves segmentation and accuracy under tight budgets, at the cost of maintaining more episodic caches.

Memory Scalability

• EPICACHE maintains superior accuracy over baselines as context length scales to 100K tokens, with accuracy approaching full KV as the cache budget increases.

Theoretical and Practical Implications

EPICACHE demonstrates that memory-bounded, episodic KV caching is feasible and effective for long-term conversational agents. The combination of block-wise prefill, episodic clustering, and sensitivity-aware budget allocation enables LLMs to maintain multi-turn conversational coherence under strict resource constraints. The framework is training-free, model-agnostic, and compatible with existing LLM architectures.

Notably, the results contradict the prevailing assumption that block-prefill eviction must entail severe accuracy degradation in multi-turn settings. By leveraging episodic structure and sensitivity profiling, EPICACHE achieves both bounded memory and high answer quality.

Future Directions

• Advanced clustering: More sophisticated, conversation-structure-aware clustering could yield even more coherent episodic boundaries.
• Adaptive episode count: Dynamically determining the optimal number of episodes per conversation could further improve scalability.
• Cache quantization: Integrating quantization into episodic caches would reduce storage and transfer costs.
• Integration with retrieval-augmented generation: Combining episodic KV caching with external retrieval modules may further enhance long-term memory and factuality.

Conclusion

EPICACHE provides a principled, efficient, and empirically validated solution to the KV cache memory bottleneck in long conversational QA. By bounding memory growth, preserving topic-relevant context, and optimizing cache allocation across layers, it enables practical deployment of LLM-based assistants in resource-constrained environments without sacrificing multi-turn conversational accuracy. The framework sets a new standard for memory-efficient conversational AI and opens several avenues for further research in episodic memory management and scalable LLM inference.