---
title: Freq-Domain KV Cache Compression
url: https://www.emergentmind.com/topics/frequency-domain-kv-cache-compression
type: topic
---

# Freq-Domain KV Cache Compression

Frequency-domain KV cache compression encompasses a family of techniques that exploit the spectral (frequency) structure of the key and value matrices stored in transformer-based models to reduce memory and computational requirements at inference time. Unlike score-based or geometric compression, frequency-domain approaches analyze and modify the distribution of KV activations using transforms such as the Discrete Cosine Transform (DCT) or Discrete Fourier Transform (DFT), revealing intrinsic energy concentration patterns and enabling informed pruning or projection. This class of methods achieves significant cache reduction and acceleration, with minimal loss in downstream accuracy, and underpins recent algorithms such as FlashCache, FAEDKV, and EliteKV [2511.16786][2507.20030][2503.01586]. These approaches render KV cache management more efficient and in some cases agnostic to token position, making them particularly attractive for large-context and multimodal transformers.

## 1. Spectral Properties of KV Matrices in Transformers

Empirical studies on multimodal and language transformers reveal that the frequency-domain energy of the per-layer key and value matrices is predominantly concentrated in low-frequency bands. Specifically, it is observed that over 90% of the total energy in both keys and values lies in the low-frequency region, as measured after applying a 1D DCT or DFT along the cache sequence axis [2511.16786]. This regularity suggests that much of the information encoded by the KV cache is smooth in token position and can be well-represented by a small subset of spectral components.

In the context of RoPE-based attention, each attention head implicitly encodes a distribution over base rotation frequencies, further motivating frequency selection as a means of compression [2503.01586].

## 2. Frequency-Domain Transformation and Principal Component Extraction

To exploit these spectral properties, the KV caches $K^\ell, V^\ell \in \mathbb{R}^{N\times D}$ (where $N$ is context length, $D$ the hidden size, and $\ell$ the layer) are transformed via DCT [2511.16786] or DFT [2507.20030] along the token axis. Formally, for DCT:

$$
C_k^\ell[m] = \alpha[m] \sum_{i=0}^{N-1} k_i^\ell \cos[\pi m(i+1/2)/N]
$$

with energy spectrum $P_k^\ell[m] = |C_k^\ell[m]|^2$ and $\alpha[m] = \sqrt{1/N}$ for $m=0$, $\sqrt{2/N}$ otherwise [2511.16786]. For DFT/IWDFT, analogous formulas apply.

Low-pass filtering is then performed by zeroing all coefficients above a cutoff frequency $\omega = \gamma N$ (with $0<\gamma<1$). The inverse transform (IDCT or IDFT) of this truncated spectrum yields the "base" KV sequence, representing the principal, smooth component.

In EliteKV, the intrinsic frequency preference of each RoPE attention head is identified, and only the top-$r$ "elite" frequency components are retained for rotation, restoring linearity to other dimensions [2503.01586].

## 3. Outlier KV Definition and Recognition

High-frequency KV pairs—tokens whose keys and/or values substantially deviate from the base—are disproportionately critical for model inference. The deviation is quantified as:

$$
\operatorname{Dev}[i] = \|k_i^\ell - k_{\rm base}^\ell[i]\|_2^2 + \|v_i^\ell - v_{\rm base}^\ell[i]\|_2^2
$$

Pairs with the largest deviations are termed "Outlier KVs" (Editor's term), found via sorting. Ablation studies indicate that selectively removing Outlier KVs causes pronounced drops in task accuracy, confirming their significance [2511.16786].

Automated recognition modules (such as the Outlier KV Recognition Module in FlashCache) prioritize the retention of these KVs under a per-layer budget constraint.

## 4. Compression Algorithms and Cache Retention Strategies

Frequency-domain KV cache compression encompasses a pipeline comprising:

- **Spectral transform:** Apply DCT/DFT to project K/V caches to frequency space.
- **Spectral selection/pruning:** Use ablation studies, energy criteria, or head-specific metrics to select bins or frequency bands to retain. In FAEDKV, per-layer DFT bins are split into contiguous chunks and the most information-rich chunks are identified by observing perplexity drops when ablated [2507.20030].
- **Inverse transform:** Invert the selected spectrum to reconstruct a compressed cache, or represent the cache in the retained frequency bins.
- **Outlier identification:** Compute and sort deviations to select critical KVs (FlashCache).
- **Dynamic budget allocation:** Compute per-layer outlier energy ratios and normalize across layers to allocate retention budgets, maximizing the share of outlier energy preserved [2511.16786].

EliteKV introduces RoPElite, which restores linearity by keeping only frequency dimensions empirically found to be important for each head, then applies joint low-rank projection (J-LRD) that shares the reduced subspace between keys and values [2503.01586].

FAEDKV employs a novel Infinite-Window DFT (IWDFT) to maintain a compressed, frequency-domain cache that incorporates all tokens with equal weight, ensuring unbiased retention and allowing efficient, per-token updates without multiple passes or sliding windows [2507.20030].

## 5. Computational and Memory Savings

The principal benefit of frequency-domain KV cache compression is a substantial reduction in cache memory and associated compute at each decoding step:

- **Memory reduction:** For a retention ratio $\rho\in(0,1]$, compressed cache size is $M_{\rm cmp} \approx \rho M_{\rm full}$, with $\sim 70–85\%$ savings typical [2511.16786][2507.20030][2503.01586].
- **Runtime speedup:** Since cache length is reduced, the dominant attention cost per decoding step drops from $O(N \cdot D)$ to $O(\rho N \cdot D)$, resulting in speedups up to $1/\rho$.
- **Overhead:** The additional computation for DCT/DFT/IDFT and sorting is $O(D N \log N)$ at most, and negligible relative to attention for large $N \gg D$ [2511.16786].
- **EliteKV:** By combining RoPElite and J-LRD, cache size can be reduced by 75% with negligible performance loss, and associated speedups of $1.2\times$–$1.5\times$ are observed [2503.01586].

## 6. Experimental Results and Evaluation

Empirical evaluations demonstrate the efficacy of frequency-domain methods:

| Method        | Decoding Speedup | KV Memory Reduction | Accuracy Δ vs Full |
|---------------|------------------|---------------------|--------------------|
| Full Cache    | 1.00×            | 0%                  | 0.0%               |
| LOOK-M        | 1.30×            | 60%                 | –4.2%              |
| MEDA          | 1.40×            | 70%                 | –3.1%              |
| FlashCache    | 1.69×            | 80%                 | –0.3%              |

(Results on Qwen2.5-VL-7B, ρ=0.2, tasks on MileBench [2511.16786])

Across six multimodal and language benchmarks, FlashCache matches or exceeds other compression methods, maintaining accuracy drops under 1%. FAEDKV outperforms state-of-the-art eviction-based baselines by up to 22% in tight memory settings, with especially uniform retrieval accuracy throughout the sequence on "Needle-In-A-Haystack" tasks [2507.20030].

EliteKV achieves a 75% reduction in KV cache with less than 1% average performance loss after minimal up-training (0.6% of pre-training tokens), and remains robust across different model scales [2503.01586].

## 7. Practical Integration and Compatibility

Frequency-domain cache compression frameworks impose minimal constraints on model architecture. Notably, methods such as FlashCache are "attention-score-free" and fully compatible with efficient attention kernels, including FlashAttention [2511.16786]. 

FAEDKV maintains a position-agnostic, training-free, and unbiased representation, recommending a practical pipeline: (i) one-off frequency ablation, (ii) per-layer cache pruning, and (iii) efficient GPU/CPU mixed-precision implementation [2507.20030]. EliteKV further demonstrates adaptability to RoPE-based models via local structure modifications and minor up-training.

A practical implication is that frequency-domain KV cache compression can be realistically deployed to production-scale transformers, providing significant memory and time savings without material accuracy loss, and regardless of input modality or context length.

---
**Principal references:**
- "Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach" [2511.16786]
- "FAEDKV: Infinite-Window Fourier Transform for Unbiased KV Cache Compression" [2507.20030]
- "EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection" [2503.01586]

Source: https://www.emergentmind.com/topics/frequency-domain-kv-cache-compression