AdaptCache: Adaptive KV-Cache Compression

• AdaptCache is a system that dynamically compresses and places key-value cache entries based on future reuse, quality sensitivity, and transfer delay.
• It employs per-entry decisions on compression algorithm, rate, and storage-tier placement to boost DRAM hit rates and reduce time-to-first-token.
• Empirical evaluations on LongBench datasets show AdaptCache achieves up to 2.4× lower delay and significant quality improvements compared to fixed compression baselines.

AdaptCache is a storage and compression system for LLM key-value caches that treats reused prefill state as a first-class serving artifact rather than as a uniform blob to be kept in DRAM when possible and otherwise spilled to SSD. Its defining idea is entry-wise adaptivity along three coupled decisions—compression algorithm, compression rate, and storage-tier placement—with the explicit goal of reducing time to first token (TTFT) while keeping generation quality close to the uncompressed baseline (Feng et al., 28 Aug 2025). In this formulation, KV-cache management is not merely an eviction problem or an offloading problem; it is a joint compression-and-placement problem driven by future reuse, quality sensitivity, and transfer delay.

1. Problem setting and serving bottleneck

AdaptCache is motivated by workloads in which LLM serving systems repeatedly encounter shared prefixes such as chat history, retrieved documents, codebases, or multi-agent transcripts. These systems cache the KV state produced during prefilling and reuse it for future requests that share context, thereby avoiding redundant attention computation over the shared prefix (Feng et al., 28 Aug 2025). The paper argues, however, that once such applications scale, KV-cache reuse itself becomes the bottleneck because the total cache footprint becomes too large for fast memory.

The paper gives two concrete capacity illustrations. With two H100 nodes and 150 GB of CPU memory, Llama-3.1-70B-Instruct can only keep about 500K tokens of KV cache, roughly the chat history for only 30 users. In agentic workloads, a single run can involve millions of tokens and about 3 TB of KV cache (Feng et al., 28 Aug 2025). Under these conditions, hierarchical storage using DRAM and SSD is necessary for full retention, but prior DRAM/SSD systems still suffer high delay because DRAM is limited, most KV-cache hits come from SSD, and SSD loads are slow.

A common misconception is that any cache hit is intrinsically latency-reducing. AdaptCache explicitly rejects that assumption. Its analysis is that when most hits are served from SSD, the average cache-hit service time becomes dominated by disk fetches, so the system has a cache, but the “hit” is often not fast enough to help (Feng et al., 28 Aug 2025). This observation is the immediate rationale for increasing the DRAM hit rate rather than merely increasing total cache capacity.

2. KV-cache-native storage hierarchy

The paper characterizes AdaptCache as a KV-cache native storage hierarchy. The hierarchy is centered on two levels: DRAM / high-speed memory, which should store the most reusable and best-compressed KV entries, and SSD / low-speed storage, which should hold less reused or less compressible entries where the cost of disk access is accepted (Feng et al., 28 Aug 2025). What makes the hierarchy “native” is that placement is conditioned on entry-specific utility rather than on recency alone or on fixed-size cache-line abstractions.

This storage model is coupled to lossy compression. Some prefixes have high future reuse frequency and can be compressed aggressively with limited quality loss; others are fragile and should be kept at higher quality or relegated to disk. The system therefore uses compression as a first-class mechanism for increasing effective DRAM capacity and, in turn, DRAM hit rate (Feng et al., 28 Aug 2025). The paper’s central empirical claim is that static compression is the wrong abstraction because different entries require different methods and rates, and storage tier should be part of the same decision.

This positioning distinguishes AdaptCache from approaches that treat KV cache primarily as a memory-reduction target during a single long-context decode. For example, CAKE frames inference-time KV-cache eviction as a “cake-slicing problem,” allocates cache size across layers according to attention dynamics, and reports that it maintains model performance with only 3.2% of the KV cache while achieving over 10x speedup in decoding latency compared to full cache for 128K-token contexts with FlashAttention-2 (Qin et al., 16 Mar 2025). This suggests that CAKE and AdaptCache operate at different points in the design space: CAKE emphasizes layer-aware eviction within a decoding session, whereas AdaptCache emphasizes cross-request reuse, DRAM/SSD placement, and TTFT under large reused-prefix corpora.

3. Adaptive lossy compression framework

AdaptCache’s core mechanism is adaptive lossy KV-cache compression on a per-entry basis. For each KV cache entry, the system jointly decides the compression algorithm, the compression rate, and the storage tier placement (Feng et al., 28 Aug 2025). The motivation is semantic heterogeneity across contexts: some text is important mainly at the beginning and end and may be suitable for token dropping, whereas other text is dense in information and may be better served by quantization. A fixed compression method is therefore suboptimal.

The architecture contains three components: Estimator, Policy optimizer, and Executor. The Estimator performs offline profiling of quality–delay tradeoffs. It profiles device transfer delays, decompression overhead, quality-vs-compression curves for each compression method, and future reuse frequency of each entry, estimated from historical hit frequency. The device-timing profile uses dummy questions, and the quality profile samples ten entries from each dataset and uses GPT-4o-generated questions to evaluate how compression affects generation quality (Feng et al., 28 Aug 2025). The Policy optimizer chooses compression and placement decisions, and the Executor applies the selected compression and storage actions.

The target operating policy is conceptually straightforward. Frequently reused and compressible entries should be pushed into DRAM at high compression so that more future requests hit DRAM. Rarely reused or hard-to-compress entries can stay on SSD or be stored with less aggressive compression. Algorithm selection depends on context characteristics: token dropping may preserve important boundary tokens in contexts where middle content is less critical, while quantization may be better for contexts that are information-dense throughout (Feng et al., 28 Aug 2025). The paper attributes AdaptCache’s advantage over fixed-compression baselines to this refusal to force all KV caches through one compression recipe.

4. Utility model and online decision policy

The paper formulates AdaptCache as an optimization over cache-entry utility under capacity constraints. Each entry’s utility combines expected future reuse, generation quality under a compression choice, and loading delay under that choice (Feng et al., 28 Aug 2025). The utility definition is

$$\mathit{Utility}(i)=\mathit{Freq}(i)\cdot \left( \alpha \times \mathit{Quality}(i, M_i, R_i) - \frac{\mathit{size}(i, M_i, R_i)}{\mathit{Bandwidth}} \right)$$

where $i$ is the KV cache entry, $\mathit{Freq}(i)$ is the estimated future occurrence frequency, $M_i$ is the compression method for entry $i$, $R_i$ is the compression rate, $\alpha$ controls the quality-delay tradeoff, $\mathit{size}(i, M_i, R_i)$ is the compressed size, and $\mathit{Bandwidth}$ is the effective transfer bandwidth (Feng et al., 28 Aug 2025).

The overall optimization is stated as an NP-hard Multi-Choice Knapsack Problem (MCKP). Because exact optimization is not tractable online, AdaptCache uses a greedy strategy inspired by textbook MCKP solutions. For each entry and each possible compression-rate transition, it computes a marginal utility drop:

$$\frac{\mathit{Utility}(i,m) - \mathit{Utility}(i,n)}{\mathit{size}(i)\times (m-n)}$$

where $i$0 and $i$1 are two different compression rates (Feng et al., 28 Aug 2025). This quantity is interpreted as the utility lost per unit of space saved if the system compresses further or evicts the entry. The greedy policy then chooses, for each storage tier, the action with the smallest marginal utility drop.

The resulting decision logic is explicitly utility-per-byte oriented rather than fit-based. In practical terms, AdaptCache prefers compressing or keeping entries that give the best delay-quality return on scarce storage, while evicting those whose utility contribution is too low (Feng et al., 28 Aug 2025). A second common misconception is therefore that the objective is maximum compression. The paper’s actual objective is maximum utility under a quality-delay tradeoff.

5. Evaluation design and quantitative findings

AdaptCache is evaluated on 1,100 contexts drawn from six LongBench datasets—SAMSum, QMSum, TriviaQA, HotpotQA, RepoBench, and LongCoder—covering three task types: Summarization, Question answering, and Coding (Feng et al., 28 Aug 2025). Because the datasets do not contain request arrival timestamps, arrivals are synthesized using a Poisson distribution with varying request rates. The hardware and model setup uses one NVIDIA A100 GPU, 100 GB DRAM, 400 GB SSD, Llama-3.1-8B-Instruct, and SSD read throughput of 1 GB/s. The baselines are Without Compression, KIVI LRU, StreamingLLM LRU, and Prefill. The primary latency metric is TTFT, and generation quality is measured by task-specific metrics F1, ROUGE-L, and CodeBLEU, defined as similarity between the answer generated after compression and the original prefill answer (Feng et al., 28 Aug 2025).

The headline result is that AdaptCache consistently outperforms fixed strategies across all tasks. The paper reports 1.43–2.4× lower delay / TTFT savings compared with the strongest fixed-compression baseline and 6–55% quality improvements at the same delay (Feng et al., 28 Aug 2025). Against naive prefill and offloading, AdaptCache reduces TTFT by 56%, while maintaining within about 15% quality drop. Compared to KIVI, it reduces TTFT by 69% at the same quality. Compared to StreamingLLM, it improves quality by 15–89% at the same TTFT.

The paper also presents an illustrative coding-task result on DRAM hit rate. KIVI LRU achieves only 38% DRAM hit rate when quantized to 2 bits, whereas AdaptCache, depending on the quality-delay tradeoff setting, achieves 81%, 56%, 44%, and 11% hit rates on DRAM (Feng et al., 28 Aug 2025). This is significant because it operationalizes the paper’s main systems claim: the improvement comes not only from compressing more, but from moving a materially larger fraction of useful hits from SSD into DRAM.

6. Interpretation, limitations, and broader significance

The paper attributes AdaptCache’s gains to two interacting effects. First, it raises the DRAM hit rate by compressing selective entries aggressively, which reduces slow SSD fetches that would otherwise dominate TTFT. Second, it preserves quality better than fixed compression by allocating “quality budget” where it matters, prioritizing entries with high reuse or appropriate redundancy structure and choosing the compression algorithm per entry (Feng et al., 28 Aug 2025). This suggests that entry-wise heterogeneity, rather than raw cache size alone, is the central systems variable in reused-prefix serving.

The main practical implication is that LLM serving stacks using simple DRAM/SSD offloading policies are leaving performance on the table. AdaptCache argues that adaptive compression can make long-context and multi-turn serving much more scalable, and that the relevant objective is not maximum compression but maximum utility under a quality-delay tradeoff (Feng et al., 28 Aug 2025). In a broader adaptive-caching perspective, this aligns with prior work that analyzes adaptability as a balance between learning accuracy and adaptation speed rather than steady-state behavior alone (Li et al., 2017).

The limitations are equally explicit. The greedy policy is not globally optimal because the optimization is NP-hard. The estimator relies on offline profiling and historical frequency estimates, so its accuracy depends on workload stability. The evaluation is preliminary and limited to one model family and LongBench-style workloads. The paper reports results only for a DRAM and SSD hierarchy, not broader tiers such as network storage or multi-node disaggregation (Feng et al., 28 Aug 2025). A plausible implication is that deployment quality will depend heavily on how well offline profiles transfer to live serving distributions.

Taken together, AdaptCache advances a specific thesis about LLM systems design: KV-cache entries should not be treated as identical objects. Instead, they should be compressed and placed according to future reuse frequency, compression sensitivity, and storage-tier cost. Within the scope evaluated, that shift yields substantial TTFT savings and quality improvements over static baselines, and reframes KV-cache management as a utility-optimization problem native to reused-context serving (Feng et al., 28 Aug 2025).