FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning. We introduce FreqDepthKV, an inference-time cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe assigns attention heads to shared-depth, residual-depth, or exact cache modes according to their contribution to reconstruction-sensitive attention logits, allowing the compression policy to adapt to prompt structure without retraining. Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV preserves task accuracy under substantially smaller cache budgets. With a 32k-token prefill window, FreqDepthKV reaches 58.3 Exact Match, 63.0 F1, 32.5 ROUGE-L, and 48.1 pass@1, closely matching full KV while outperforming prior compressed-cache methods. It also improves decoding throughput to 70.4 tokens/s, reduces TTFT to 2.06 seconds, and lowers peak KV memory to 6.2 GB, achieving a 3.9x effective compression ratio.

• The paper introduces FreqDepthKV, which combines DCT-based depth factorization, sparse high-frequency residuals, and logit-aware routing to achieve 3.9× compression while reaching 58.3 EM, 63.0 F1, and 70.4 tokens/s.
• FreqDepthKV evaluates shared-depth, residual-depth, and exact cache modes during prefill, retaining layer-specific tokens according to attention-logit perturbation rather than relying only on reconstruction error or cache saliency.
• The method closely matches full-cache accuracy and outperforms MiniCache and token-selection baselines, especially on retrieval and code tasks, while remaining retraining-free but leaving joint quantization, eviction, and dynamic decoding policies for future work.

Motivation and problem statement

Long-context inference shifts the dominant cost of LLM serving from parameter storage to the key-value (KV) cache, whose footprint grows linearly with context length across layers, heads, and positions. Existing compression strategies—token eviction (H2O, StreamingLLM, SnapKV, PyramidKV), quantization (KVQuant, KIVI), and inter-layer sharing (MiniCache)—treat redundancy as either a token-level or precision-level phenomenon. The paper's central observation is that adjacent transformer layers exhibit correlated depth-wise cache structure, but that uniform depth sharing, as in MiniCache (Liu et al., 2024), can erase sparse layer-specific evidence. A layer that appears redundant on average may still carry high-frequency residual information needed to disambiguate a needle sentence or preserve a code dependency; once removed, attention logits shift enough to change generated answers even when aggregate reconstruction error is low. This failure mode is consistent with recent analyses of workload-dependent compression risk [(2607.06519) cites chen2025pitfalls, haverbeck2026risk].

Method

FreqDepthKV factorizes the KV cache along the depth dimension within blocks of $B$ adjacent layers using a fixed DCT basis $F_B$ ($B=4$ in middle layers, $B=2$ near model boundaries). The first coefficient group forms a low-frequency shared-depth component stored once per block; remaining groups encode high-frequency layer deviations stored sparsely.

The key mechanism is an online routing probe evaluated during prefill. For each head, candidate caches are reconstructed under three modes—shared-depth (low-frequency only), residual-depth (shared plus sparse residual set), and exact—and scored by a reconstruction-aware loss comparing original versus reconstructed attention logits over probe query positions sampled from recent tokens, document boundaries, and high-entropy attention rows:

$$\mathcal{L}_{b,h}(m)=\frac{1}{|\mathcal{P}|}\sum_{t\in\mathcal{P}}\Big\|\frac{Q_{\ell,h,t}\widehat{K}^{(m)\top}_{\ell,h}}{\sqrt{d_h}}-\frac{Q_{\ell,h,t}K_{\ell,h}^{\top}}{\sqrt{d_h}}\Big\|_2^2+\lambda\,\Omega(m)$$

where $\Omega(m)$ is normalized memory cost. Residual retention is prioritized by per-token logit-perturbation scores, keeping top-$r_{b,h}T$ tokens' high-frequency coefficients. Routing is computed once after prefill (with optional periodic refresh), enforced against a target memory budget by tuning $\lambda$, and fused into the attention kernel so full caches are never materialized. The method requires no retraining and is orthogonal to token eviction and quantization, though experiments evaluate it standalone to isolate its effect.

Main results

Under a 32k-token prefill window with a targeted $3.8\times$ average compression ratio, FreqDepthKV achieves the best compressed-cache accuracy on all aggregate metrics while also leading in efficiency:

Method	EM	F1	ROUGE-L	pass@1	Tokens/s	TTFT (s)	Peak KV (GB)	Ratio
Full KV	58.7	63.4	32.8	48.6	38.2	2.91	24.0	1.0×
PyramidKV	56.4	61.2	31.4	45.8	63.0	2.24	7.4	3.2×
KIVI	57.1	61.8	31.7	46.4	60.9	2.29	6.9	3.5×
MiniCache	56.6	61.0	31.3	45.6	65.5	2.18	6.6	3.6×
FreqDepthKV	58.3	63.0	32.5	48.1	70.4	2.06	6.2	3.9×

Relative to MiniCache, it improves EM by 1.7 points, F1 by 2.0, ROUGE-L by 1.2, and pass@1 by 2.5 while reducing memory from 6.6 GB to 6.2 GB and raising throughput from 65.5 to 70.4 tokens/s. Against PyramidKV, the strongest token-selection baseline, it gains 1.8 F1 and 2.3 pass@1 points at 1.2 GB less memory. Gains are largest on retrieval-sensitive settings (Needle-in-a-Haystack, Qasper, code completion); summarization shows smaller but consistent improvements, indicating low-frequency depth components already capture much redundant document context. This supports the paper's design claim: aggressive depth sharing is safe only when paired with residual preservation and logit-aware routing.

Ablations

Each component contributes measurably. Replacing the DCT factorization with direct adjacent-layer averaging drops F1 from 63.0 to 61.3 and pass@1 from 48.1 to 46.0 at similar memory, showing that preserving high-frequency depth deviations matters more than minimizing average reconstruction error. Removing sparse residuals yields the fastest variant (72.6 tokens/s, 5.8 GB) but costs 1.6 EM and 2.3 pass@1 points; shared-depth-only is worse still (56.1 EM, 45.1 pass@1). Removing online head routing loses 1.3 F1 points via misallocated exact capacity; removing the reconstruction-aware loss causes the router to prefer cheap modes that preserve cache norms but perturb attention rankings. Dropping exact mode hurts most on needle retrieval and code tasks. Block size controls the trade-off: $B=2$ everywhere nearly matches accuracy but reduces throughput to 65.9 tokens/s; $F_B$0 reaches $F_B$1 compression but loses 0.9 F1. The mixed default ($F_B$2 middle, $F_B$3 boundary layers) is the best balance.

Limitations and open questions

The paper concedes two principal limitations. First, routing decisions are fixed after prefill and held static during decoding; whether generation-aware policies that migrate heads between modes as dependencies emerge would help multi-turn reasoning and long code completion remains open. Second, all results are reported for standalone depth-frequency compression; the claimed orthogonality to quantization and token eviction is asserted rather than empirically demonstrated, and the proposed joint depth-frequency/bit-width allocation problem is untested. Additionally, the evaluation uses a single base decoder configuration, so the sensitivity of the routing probe to model scale and architecture is not established. Training-time adaptation to make adjacent-layer structure more compressible is identified as unexplored.

Conclusion

FreqDepthKV treats inter-layer KV redundancy as frequency-structured rather than uniformly shareable, combining DCT-based depth factorization, sparse high-frequency residual storage, and logit-reconstruction-aware head routing into a retraining-free inference-time method. It closely matches full-cache accuracy (58.3 EM, 63.0 F1, 32.5 ROUGE-L, 48.1 pass@1) at a $F_B$4 compression ratio with 70.4 tokens/s decoding throughput, outperforming both uniform depth sharing (MiniCache) and token-selection baselines under matched budgets. The ablations confirm that robustness under aggressive budgets depends jointly on all five components, with the reconstruction-aware routing signal distinguishing the method from saliency-only eviction policies.