WindowKV: Efficient KV Cache Compression

Updated 7 January 2026

WindowKV is a family of techniques for compressing the key-value cache in transformer LLMs by adaptively selecting and compressing past tokens.
It employs task-adaptive group-wise window selection and intra-group index sharing to reduce memory usage while preserving semantic coherence.
Frequency-domain variants use streaming IWDFT and sparse IDFT to achieve unbiased compression, improving perplexity and throughput in long-context tasks.

WindowKV is a family of techniques for compressing the key-value (KV) cache in autoregressive transformer-based LLMs. KV caches hold the layerwise keys and values corresponding to all previous tokens for efficient self-attention. WindowKV strategies are designed to dramatically reduce the memory and compute overhead of long-context inference by adaptively selecting which segments of the KV cache are retained—either via task-adaptive local window selection in token space, or by compressing past tokens into the frequency domain—while preserving semantic coherence and minimizing performance degradation. The WindowKV methodology has evolved in several directions, including task-adaptive group-wise window selection (Zuo et al., 23 Mar 2025) and Fourier-based unbiased spectral compression (Li et al., 26 Jul 2025).

1. Foundations: KV Caching and Compression Rationale

In transformer LLMs, the key-value cache stores the per-layer projections of all previous hidden states to avoid redundant computation during autoregressive decoding:

For each past token, every layer $\ell$ caches $K^\ell \in \mathbb{R}^{(i-1) \times d_k}$ and $V^\ell \in \mathbb{R}^{(i-1) \times d_v}$ .
Attention for the next token is computed as $A^\ell = \mathsf{softmax}\left(Q^\ell (K^\ell)^\top / \sqrt{d_k}\right)$ with the output $A^\ell V^\ell$ .

The memory footprint grows linearly with context length, number of layers, and projection size. For a 7B-parameter model and $10^5$ tokens, the KV cache requires tens of gigabytes, becoming the dominant GPU memory cost during inference. Compressing the KV cache enables longer contexts, higher throughput, and significantly lower hardware requirements—a critical concern for scalable LLM deployment (Zuo et al., 23 Mar 2025).

2. Task-Adaptive Group-Wise Window Selection

WindowKV (Zuo et al., 23 Mar 2025) introduces dynamic window selection in token space:

The token sequence is partitioned into an observation window (last $\alpha$ tokens, capturing immediate context) and a review context (preceding tokens).
Attention scores between the observation window's queries and the review context's keys yield token importance scores:

$t_j = \sum_{i=n-\alpha}^{n} A_{ij}, \quad j=1,\dots,n-\alpha.$

The review context is divided into non-overlapping windows of size $\omega$ .
A lightweight classifier tags the task as either information localization (e.g., QA) or aggregation (e.g., summarization). Window scoring adapts accordingly:

$s_k = \frac{1}{\min(p, \omega)} \sum_{t \in \mathrm{Top}-p(\mathbb{W}_k)} t,$

where $p=\omega$ for localization and $p<\omega$ for aggregation.

For each group of layers, only the top $B^h$ windows are retained (budgeted via a pyramidal allocation across groups and layers).

This preserves semantic continuity by keeping contiguous segments and adapts retention to task characteristics. Selection is performed once per group, sharing indices among layers to reduce overhead.

A key innovation is intra-group index sharing for efficiency. The model's layers are divided into $H$ groups; within a group, the "anchor" layer performs the expensive attention-based selection, and all other layers reuse its indices:

$\mathbb{I}_\ell = \mathbb{I}_{\ell_g}, \forall \ell \in \mathbb{G}_h,$

where $\ell_g$ is the first layer in group $\mathbb{G}_h$ . This reduces window selection cost from $O(m)$ to $O(m/\gamma)$ , with $\gamma$ being group size. The observation window is always fully retained to allow fluid, recent generation. Only the "prefill" or input encoding phase applies compression; generation-phase pruning is not performed in the base method.

4. Frequency-Domain KV Cache Compression

An alternate and complementary approach, denoted FAEDKV ("Frequency-Adaptive Infinite-Window for KV cache") but also described as a WindowKV variant (Li et al., 26 Jul 2025), reframes cache compression in the frequency domain:

The core operation is a streaming, normalized Infinite-Window Discrete Fourier Transform (IWDFT) applied to keys and values as new tokens arrive:

$S_{t+1}[k] = W_k \left((N-1)/N\cdot S_t[k] + (1/N) x[t+1]\right)$

with $W_k = e^{-j(2\pi/N)k}$ and $x[t+1]$ the new vector.

The accumulated spectral coefficients $S_t[k]$ represent the DFT of all preceding tokens, such that

$S_t[k] = \frac{1}{N} \sum_{n=1}^{N} x[n] e^{-j(2\pi/N) k n}$

meaning each token contributes equally.

At each layer, only a subset $B^*$ of the most informative frequencies are retained, selected via offline ablation (measuring perplexity change when frequency bands are zeroed).
For attending, the historical context is reconstructed via sparse inverse DFT (IDFT) over retained bins.

This method is provably unbiased: neither recency nor position bias is introduced, addressing weaknesses of fixed-window and token-eviction schemes.

5. Complexity, Efficiency, and Empirical Performance

WindowKV achieves substantial memory reduction with minimal performance degradation.

Memory and Compute Complexity:

Method	Memory Scalability	Compute Overhead
Full KV	$O(m n d)$	$O(m n^2 d_k)$
Task-Adaptive WindowKV	$O(m\alpha d + H B_{\max} d)$	Window selection: $O(\tfrac{m}{\gamma} \alpha (n-\alpha) d_k)$
FAEDKV (Spectral WindowKV)	$O(r N d)$	Per step: $O(r N \log N)$

With $\alpha \ll n$ , $B_{\max} \ll n$ , or spectral compression ratio $r \ll 1$ , both variants reduce KV memory by an order of magnitude.

Experimental Results (LongBench & Needle-in-a-Haystack):

On LongBench (Qwen2.5-1.5B, KV size 2048, $\approx12\%$ retention), task-adaptive WindowKV achieves an average score of 32.75 versus 33.63 for full KV (Zuo et al., 23 Mar 2025).
FAEDKV on Llama3-8B ( $r=0.094$ ) yields up to 22% better average perplexity than H2O/SnapKV on QA tasks (Li et al., 26 Jul 2025).
Needle-in-a-Haystack: WindowKV attains superior and position-agnostic retrieval, with FAEDKV maintaining flat performance across early and late token retrieval.
On A100-40GB, WindowKV (task-adaptive, KV size=512) achieves 17% higher throughput with an 88% reduction in memory (Zuo et al., 23 Mar 2025).
FAEDKV improves per-token latency by 10–30% on Llama3-8B for long sequences (Li et al., 26 Jul 2025).

6. Limitations and Potential Directions

Several constraints are acknowledged:

Fixed window size in task-adaptive WindowKV may omit fine-grained evidence if windowing is too coarse.
Intra-group index sharing can degrade if attention patterns shift abruptly within a group.
Task-adaptive WindowKV presently compresses only the prefill phase; output-phase compression for generation is unaddressed.
FAEDKV requires custom IWDFT/IDFT kernels and per-layer band selection; it cannot extend positional length beyond the model’s trained maximum.

Prospective extensions include adaptive, variable-length windowing, head-wise token retention, online re-scoring during generation, and applying spectral compression to the generation phase for ultra-long outputs (Zuo et al., 23 Mar 2025).

7. Relationship to Prior Work and Comparative Analysis

WindowKV methods systematically address the shortcomings of previous KV cache compression schemes:

Token Eviction (H2O, SnapKV): Simple, low memory but recency-biased; often “lost-in-the-middle.”
Learned Projections (LoCoCo, ActivationBeacon): Flexible representation but require retraining and degrade old context.
Task-Adaptive WindowKV: Task-aware, contiguous selection; moderate overhead, high preservation of functional context (Zuo et al., 23 Mar 2025).
Spectral (FAEDKV) WindowKV: Training-free, unbiased across sequence, superior information density, but depends on spectral kernel implementation and is limited to compression (not extension) (Li et al., 26 Jul 2025).

WindowKV thus encompasses both contiguous, semantically coherent window selection for practical task adaptation and frequency-domain compression for unbiased long-context representation. Both approaches have demonstrated state-of-the-art balance of memory efficiency, retrieval capability, and inference speed in LLMs across industrial and research benchmarks.