Recency/Frequency Adaptive KV Caching for Large Language Model Serving

Abstract: Key-value (KV) caching is a powerful technique for accelerating LLM inference and generation. Inference workloads are large and diverse, which makes them difficult to cache effectively. Existing cache management strategies adopt the least-recently-used policy for evicting cache blocks. However, LRU leads to multiple unrelated workloads flushing each other's caches. To address this, we integrate adaptive caching that dynamically allocates cache space between recently and frequently occurring KV blocks. Evaluations show that it improves the KV cache hit rate by up to 10.8% and reduces time to first token by up to 12.6% over naive vLLM on synthetic document question answering workloads, and 2.1% and 2.0% respectively on real-world conversation workloads. The method generalizes well to batch inference and demonstrates clear interpretability while effectively accommodating diverse workloads.

• The paper introduces an approach that integrates the ARC algorithm for adaptive KV caching in LLM serving, dynamically balancing recency and frequency-based requests.
• It demonstrates up to ~10.8% improvement in cache hit rates and 12.6% reduction in TTFT under diverse workloads, outperforming traditional LRU strategies.
• Empirical evaluations on document QA, multi-turn conversations, and batch inference confirm the method’s robustness, efficiency, and practical benefits for production systems.

Recency/Frequency Adaptive KV Caching for LLM Serving

Introduction and Motivation

The paper "Recency/Frequency Adaptive KV Caching for LLM Serving" (2606.21238) addresses a critical bottleneck in LLM inference: efficient management of GPU KV caches under diverse and dynamic workloads. While prefix caching can mitigate redundant computation in prefill and decoding stages, extant cache eviction strategies—primarily recency-based such as LRU—do not adequately accommodate mixed access patterns involving both locality and frequency. The authors propose to integrate the Adaptive Replacement Cache (ARC) algorithm into vLLM to dynamically partition cache space between recency and frequency, thereby improving hit rates and reducing latency (TTFT).

Figure 1: Overview of adaptive KV caching for LLM serving, depicting application-layer request distribution, GPU-level cache management, and ARC-based eviction integration.

Architecture and Algorithmic Details

The ARC algorithm logically partitions the cache into recency (L1) and frequency (L2) queues. Each is further split into physical caches (T1, T2) holding KV blocks and ghost caches (B1, B2) retaining only metadata for recently evicted items. This dual-queue design and ghost caching enable adaptive cache partitioning via feedback mechanisms: ghost cache hits inform tuning the relative sizes of T1 and T2 to match real-time workload characteristics.

Figure 2: Structure of the Adaptive Replacement Cache, illustrating recency/frequency queues and metadata-tracking ghost caches for adaptive partitioning.

The integration into vLLM aligns eviction decisions with paged-attention mechanisms and batch inference, supporting fixed-size KV blocks and parameter-free operation. The ARC-based evictor replaces the classic LRU strategy, dynamically balancing memory allocations between recency-dominated and frequency-dominated requests, as relevant in document QA (hotspot documents) and multi-turn chat (active user histories).

Numerical Evaluation and Ablation

Document QA Workloads

Experiments leverage QuALITY and WikiQA datasets with synthetic Zipf-distributed workloads to simulate realistic access patterns. ARC achieves up to ∼10.8% improvement in hit rate and up to 12.6% reduction in TTFT over LRU on QuALITY. For WikiQA, improvements reach ∼7.8% in hit rate and ∼7.9% in TTFT. The effectiveness of adaptive partitioning becomes more pronounced at smaller cache sizes, where replacement frequency is higher and static policies are less robust to workload shifts.

Figure 3: KV Cache Hit Rate comparison for QuALITY workload, demonstrating ARC and DBL outperforming LRU across cache sizes.

Figure 4: KV Cache Hit Rate for WikiQA workload illustrating the advantages of adaptive eviction over standard LRU.

Multi-Turn Conversation Workloads

Evaluation on production-scale conversational data from Qwen-Bailian (Trace A) validates ARC's efficacy in real-world settings. ARC increases hit rate by 1.1–2.1% and reduces TTFT by 1.0–2.0% compared to LRU. Notably, adaptive queue sizing becomes crucial as cache sizes grow, showing resilience to workload shifts and outperforming static two-queue (DBL) strategies at scale.

Figure 5: KV Cache Hit Rate on multi-turn chat workloads, revealing ARC’s performance gains in large-scale, dynamic scenarios.

Batch Inference

ARC maintains its performance advantages under varying batch sizes. TTFT reductions persist, albeit aggregate throughput increases, and the cache hit rate remains relatively stable as batch size grows, confirming the robustness of adaptive eviction in high-throughput, multi-request environments.

Figure 6: KV cache hit rate and TTFT across batch sizes on QuALITY, demonstrating ARC’s consistency and TTFT improvement as batches scale.

Adaptive Partitioning Dynamics

Tracking the T1/T2 (recency/frequency) ratio reveals ARC’s capacity to respond to changing workload distributions. In conversational traces, a tendency to reserve more space for recency emerges as cache scales, indicating that the frequency working set stabilizes at smaller cache sizes. In QuALITY, the ARC mechanism dynamically migrates allocation between queues according to windowed Zipf workload shifts, empirically supporting that adaptive partitioning yields substantive gains over static splits.

Figure 7: Dynamic evolution of the T1 ratio across workloads and capacities, exposing ARC’s responsiveness to workload characteristics and cache pressure.

Practical and Theoretical Implications

The integration of ARC into LLM serving frameworks demonstrates that conventional recency-based eviction policies are insufficient for high-throughput, mixed-workload environments. Dynamic recency-frequency partitioning is critical for maintaining high cache hit rates and minimizing TTFT, particularly at constrained memory budgets and under nonstationary workloads. The parameter-free, interpretable nature of ARC facilitates deployment in production systems. By exposing memory allocation dynamics, ARC enables workload-aware scheduling and could inform job placement or distributed cache architectures.

From a theoretical standpoint, the paper’s results highlight the inadequacy of simple locality-based heuristics and motivate further exploration into multi-factor eviction strategies, potentially incorporating prompt-length, positional locality, or learned predictors. The cross-compatibility with batch inference and multi-GPU setups foreshadows the extensibility of ARC for hierarchical and distributed KV cache systems.

Future Directions

Extensions of this work may focus on incorporating additional decision features in eviction policies (e.g., prompt-length-dependent latency, positional reuse probabilities) and explicit balancing among multiple dimensions of cache utility. Evaluating adaptive eviction in online inference, hierarchical disk/CPU offloading, and distributed GPU clusters represents a rich avenue for optimization in end-to-end large-scale LLM serving.

Conclusion

The adaptive KV caching approach utilizing ARC markedly advances the state-of-the-art in LLM serving. By dynamically allocating memory between recency and frequency-driven requests, it delivers improved KV cache hit rates and reduced TTFT across synthetic and real-world workloads, generalizes to batch inference, and offers interpretable cache-space allocation. The empirical and architectural evidence presented in this study strongly supports the adoption of adaptive eviction strategies as a foundational component for future LLM serving systems.