DeltaKV: Long-Context KV Cache Compression

• DeltaKV is a KV cache compression framework for long-context LLM inference that compresses semantic residuals relative to referenced tokens.
• It leverages long-range similarity and shared latent structure in KV states to reduce memory footprint while ensuring near-lossless accuracy.
• Integrated with Sparse-vLLM, DeltaKV achieves up to 2× throughput gains and reduces KV cache memory to 29% without sacrificing next-token prediction.

DeltaKV is a KV cache compression framework for long-context LLM inference that reduces the rapidly growing memory footprint of the key-value cache while preserving model quality. It is motivated by the observation that long-context deployment is fundamentally bottlenecked by the linear growth of KV cache memory, which limits batch size, context length, and throughput in settings such as autonomous agents, long-chain reasoning, and creative writing. Rather than evicting tokens, DeltaKV compresses semantic residuals relative to retrieved historical references, and when coupled with Sparse-vLLM it is reported to reduce KV cache memory to 29\% of the original while maintaining near-lossless accuracy on LongBench, SCBench, and AIME, with up to $2\times$ throughput improvement over vLLM in long-context scenarios (Hao et al., 8 Feb 2026).

1. Problem setting and motivation

DeltaKV addresses long-context inference in LLMs, where two costs scale unfavorably with sequence length: attention cost is expensive for long sequences, and KV cache memory grows linearly with context length. In practice, the KV cache quickly becomes the main memory bottleneck. The stated consequence is reduced feasible batch size, reduced feasible context length, and lower throughput in deployments involving multi-turn dialogue, agentic workflows, code understanding, legal/financial document analysis, long-chain reasoning, and creative writing (Hao et al., 8 Feb 2026).

The framework is positioned against three existing design patterns. Token eviction saves memory but can permanently discard information. Dynamic sparsity can keep accuracy but often retains the full cache in GPU memory and relies on offloading. Prior compression methods may be GPU-unfriendly or reconstruct the whole cache, which reduces real speed gains. DeltaKV is designed to avoid that tradeoff by compressing the cache in a way that is both information-preserving and hardware-friendly.

A central distinction is therefore not simply between dense and sparse attention, but between approaches that remove tokens and approaches that re-encode token states. This suggests that the paper treats the KV cache primarily as a structured representation with internal redundancy, rather than as an immutable memory to be either fully kept or partially discarded.

2. Empirical basis: long-range similarity and shared latent structure

DeltaKV is built on two empirical observations about KV caches. The first is long-range inter-token similarity: semantically similar tokens are often far apart in the context, not just nearby. The paper reports that over 60\% of the most similar tokens are separated by more than 16 positions. This directly challenges locality-based compression methods that only compare neighboring tokens (Hao et al., 8 Feb 2026).

The second observation is that KV vectors are anisotropic and contain highly shared latent components. A small number of high-norm latent directions explain a large fraction of the shared structure among tokens. The paper’s SVD analysis shows a steep spectral decay in raw KV states, implying strong redundancy. After subtracting historical references, the residuals become much flatter, lower-magnitude, and more noise-like.

These observations motivate the claim that much of a token’s KV state is not unique content but shared structure, and that the distinctive part of the state is the deviation from a suitable historical prototype. The paper further supports this view with analysis showing much smaller residual norms, a distribution concentrated near zero, and improved suitability for quantization. Conceptually, the method is described as similar to predictive coding: if a good reference already explains most of the state, only the delta must be stored.

3. Residual-based formulation

For each token, DeltaKV uses the concatenated KV state

$$\mathbf{kv}_i = \mathrm{Concat}(\mathbf{K}_i,\mathbf{V}_i) \in \mathbb{R}^{2d_k}.$$

The method operates on pre-RoPE KV states so that similarity comparisons are position-invariant (Hao et al., 8 Feb 2026).

Instead of searching over all past tokens, DeltaKV maintains a strided reference set

$$\mathcal{T} = \{ \mathbf{kv}_t \mid t \bmod s = 0,\ t < i \},$$

where $s$ is the stride. For token $i$, it retrieves the top-$k$ nearest historical tokens:

$$\mathcal{R}_i = \operatorname{arg\,top}k_{\mathbf{kv}_j \in \mathcal{T}} \left(-\|\mathbf{kv}_i - \mathbf{kv}_j\|_2^2\right).$$

It then forms the reference mean

$$\overline{\mathbf{KV}_R} = \frac{1}{k}\sum_{j\in\mathcal{R}_i}\mathbf{kv}_j.$$

Compression is performed on a latent residual:

$$\mathbf{z}_\Delta = f_c(\mathbf{KV}) - f_c(\overline{\mathbf{KV}_R}),$$

where $f_c$ is the compressor. In the MLP version,

$$\mathbf{kv}_i = \mathrm{Concat}(\mathbf{K}_i,\mathbf{V}_i) \in \mathbb{R}^{2d_k}.$$0

The decompressor reconstructs the residual:

$$\mathbf{kv}_i = \mathrm{Concat}(\mathbf{K}_i,\mathbf{V}_i) \in \mathbb{R}^{2d_k}.$$1

with

$$\mathbf{kv}_i = \mathrm{Concat}(\mathbf{K}_i,\mathbf{V}_i) \in \mathbb{R}^{2d_k}.$$2

The final reconstruction adds the reference back:

$$\mathbf{kv}_i = \mathrm{Concat}(\mathbf{K}_i,\mathbf{V}_i) \in \mathbb{R}^{2d_k}.$$3

This formulation replaces the binary decision of whether a token should be kept or dropped with a reconstruction strategy in which shared information is borrowed from retrieved historical tokens and token-specific information is encoded as a compressed correction. The practical consequence, as described in the paper, is that repeated or shared information is factored out and only the remaining difference needs to be stored.

4. Training objective and workflow

DeltaKV is trained with a hybrid loss

$$\mathbf{kv}_i = \mathrm{Concat}(\mathbf{K}_i,\mathbf{V}_i) \in \mathbb{R}^{2d_k}.$$4

Here, $$\mathbf{kv}_i = \mathrm{Concat}(\mathbf{K}_i,\mathbf{V}_i) \in \mathbb{R}^{2d_k}.$$5 is an MSE reconstruction loss between original and reconstructed KV states, and $$\mathbf{kv}_i = \mathrm{Concat}(\mathbf{K}_i,\mathbf{V}_i) \in \mathbb{R}^{2d_k}.$$6 is a next-token prediction loss to ensure that the compressed cache still supports generation quality (Hao et al., 8 Feb 2026).

The appendix describes a training procedure with three stages. First, a ground-truth forward pass produces original KV states. Second, for each layer and token, the system retrieves top-$$\mathbf{kv}_i = \mathrm{Concat}(\mathbf{K}_i,\mathbf{V}_i) \in \mathbb{R}^{2d_k}.$$7 references from the strided reference set, computes residuals, compresses and reconstructs them, and accumulates reconstruction loss. Third, it computes the final next-token prediction loss from the reconstructed pass.

The training setup is described as lightweight: about 160M tokens for standard 7B/8B models, about 8 GPU hours on a single RTX PRO 6000 for standard models, and around 14 GPU hours for the 32B model. The role of the hybrid objective is explicit: pure reconstruction loss could preserve numerically accurate KV states while still harming generation, whereas the NTP term aligns compression with end-task usefulness. The paper also notes that exact cache reconstruction is not always necessary; the ablation suggests that some numerical reconstruction error is tolerable as long as next-token behavior is preserved.

5. Sparse-vLLM and system realization

A major claim of the work is that compression alone does not guarantee speedups. To convert compression gains into actual throughput gains, the paper introduces Sparse-vLLM, an inference framework designed for sparse and compressed KV layouts (Hao et al., 8 Feb 2026).

The stated motivation is that standard frameworks such as vLLM and SGLang assume page-based, full-cache memory management. That assumption is suitable for dense attention but awkward for sparse attention, irregular KV layouts, per-token compressed storage, and selective decompression. Sparse-vLLM therefore decouples memory management from model execution.

Its architecture has three main components. The first is a modular CacheManager that handles physical allocation and logical-to-physical mapping. It supports per-layer independent mappings for physical eviction methods, global shared mappings for logical masking methods, and a heterogeneous storage scheme for DeltaKV. For DeltaKV specifically, it manages a Full Pool for high-precision tokens and a Latent Pool for compressed residual vectors.

The second is a Sparse Controller that orchestrates execution flow. In pre-forward, it builds a logical sparse view and identifies which tokens need reconstruction. In post-forward, it updates the cache lifecycle, compresses overflowed tokens, and reclaims full-precision storage.

The third is a set of GPU-friendly kernels. Sparse-vLLM uses optimized Triton kernels for indirect addressing over non-contiguous KV layouts, batch L2 distance for fast reference retrieval, and fused reconstruction that combines gathering, averaging, residual addition, and writeback. The significance of this design is explicit in the paper: irregular layouts often kill throughput if handled with generic CPU-side logic or many tiny GPU kernels.

6. Reported empirical performance

The headline memory result is that DeltaKV reduces KV cache memory to 29\% of the original size while maintaining near-lossless performance on LongBench, SCBench, and AIME (Hao et al., 8 Feb 2026). The paper characterizes these datasets as representative of different long-context demands, including multi-turn settings and reasoning-intensive tasks.

On LongBench, DeltaKV is reported to match or nearly match strong baselines such as OmniKV while using much less memory. On SCBench, it remains strong in multi-turn settings, where static eviction methods can fail badly because they discard tokens that later become important. On AIME, it retains strong reasoning performance, which the paper presents as evidence that the method is not limited to simple retrieval-style tasks.

The system-level throughput results are presented as concrete examples of deployment behavior:

Context length	DeltaKV	vLLM
128k	187.0 tokens/s	143.2
256k	120.6 tokens/s	70.2
512k	67.7 tokens/s	33.1
900k	38.9 tokens/s	18.6

At 512k, the paper describes the comparison as roughly $$\mathbf{kv}_i = \mathrm{Concat}(\mathbf{K}_i,\mathbf{V}_i) \in \mathbb{R}^{2d_k}.$$8. It also states that Sparse-vLLM itself has only minor overhead under full-attention settings, implying that the observed gains come from actual compression and sparsity rather than from substituting an unrelated runtime baseline.

7. Tradeoffs, limitations, and interpretation

The paper identifies several implementation considerations. Reconstruction overhead still exists, because the current prototype does not fully fuse reconstruction into attention. Python-level control flow is also described as a bottleneck, since some view construction and slot management are not yet fully pushed into low-level fused kernels (Hao et al., 8 Feb 2026).

Hyperparameter tradeoffs are explicit. Larger compressed dimension $$\mathbf{kv}_i = \mathrm{Concat}(\mathbf{K}_i,\mathbf{V}_i) \in \mathbb{R}^{2d_k}.$$9 improves fidelity but increases memory. More references $$\mathcal{T} = \{ \mathbf{kv}_t \mid t \bmod s = 0,\ t < i \},$$0 can help, but too many references add noise and overhead. Smaller stride $$\mathcal{T} = \{ \mathbf{kv}_t \mid t \bmod s = 0,\ t < i \},$$1 improves accuracy but increases reference storage and retrieval cost. The system also uses a hybrid strategy in which selected layers remain uncompressed, because not all layers contribute equally. Quantization is described as promising but partial: in the main setup, only the compressed residuals are quantized to keep accuracy high, while broader quantization would require careful engineering.

These limitations clarify several common misconceptions. One is that KV compression automatically yields end-to-end acceleration; the paper argues that specialized system support is required. Another is that compression must reconstruct the original cache exactly to preserve behavior; the ablation suggests otherwise, provided next-token behavior is maintained. A further implication is that DeltaKV should not be understood as a token pruning method. Its defining principle is to preserve information by storing a semantic delta relative to retrieved context rather than by deciding that a token is expendable.

Taken together, these elements position DeltaKV as a residual-based alternative to eviction-centric long-context inference. The broader claim advanced by the work is that long-context LLMs need not merely keep fewer tokens; they can store smarter representations of the tokens they keep.