EliteKV: Scalable KV Cache Compression

• EliteKV is a framework that compresses the key-value cache in RoPE-based Transformers by selectively restoring key linearity through elite chunk selection.
• It employs a greedy algorithm per attention head to identify essential frequency chunks, enabling joint low-rank projection across keys and values.
• Empirical results on LLaMA2 models demonstrate up to a 75% reduction in KV cache memory with negligible accuracy drop, confirming its efficiency.

EliteKV is a framework for scalable key-value (KV) cache compression in rotary position embedding (RoPE)-based transformer networks. By jointly exploiting selective restoration of key linearity at the attention-head level and low-rank projection across keys and values, EliteKV enables highly flexible trade-offs between memory footprint and model fidelity, supporting variable compression ratios with minimal computational overhead and negligible loss in downstream performance (Zhou et al., 3 Mar 2025).

1. Background: RoPE-Based Attention and KV Cache Bottlenecks

Rotary position embedding (RoPE) encodes relative position information by rotating each two-dimensional chunk of a query or key vector by an angle proportional to sequence position. Formally, for a head of dimension $h$, the $i$th chunk at position $t$ is rotated:

$$q_{t,i}^{\text{rot}} = R(t\cdot\theta_i)q_{t,i}, \quad k_{t,i}^{\text{rot}} = R(t\cdot\theta_i)k_{t,i}$$

where $R(\theta)$ is the standard rotation matrix: $$R(\theta) = \begin{bmatrix}\cos\theta & -\sin\theta\ \sin\theta & \cos\theta\end{bmatrix}$$

This approach allows each head to attend to different frequency components along the sequence, but introduces nonlinear dependencies between position and key chunks. After applying RoPE, low-rank approximation or conventional cache compression techniques become less effective because the rotational transformation must either be stored or re-applied at each decode step, eroding potential memory savings.

2. RoPElite: Per-Head Frequency Preference Selection

The RoPElite procedure identifies the subset of frequency chunks ("elite" chunks) in each attention head that are essential for maximizing attention score fidelity. The selection algorithm is a greedy process:

• For each head, given a target number $r$ of chunks to keep as rotated ("elite"), iteratively add the chunk whose removal least changes the attention score, measured by the $L^1$ distance between the full-RoPE and partial-RoPE attention scores.
• The set of remaining chunks is treated as linear (i.e., not rotated), restoring linearity and enabling subsequent low-rank compression.

Once elite chunk sets $\mathcal{I}_e$ are identified, the attention score $s_{m,n}$ for positions $m,n$ is computed as:

$$s_{m,n} = \sum_{i\in \mathcal{I}_e} q_{m,i}^\top R((m-n)\theta_i)k_{n,i} + \sum_{i\notin \mathcal{I}_e} q_{m,i}^\top k_{n,i}$$

Empirically, frequency preferences differ across heads and tend to generalize across model scales (e.g., from 7B to 13B parameter LLMs).

3. Joint Low-Rank Projection for KV Cache Compression

Following restoration of partial linearity by RoPElite, EliteKV applies a joint low-rank decomposition (J-LRD) across both the unrotated key matrix ($K$) and value matrix ($V$), resulting in a shared small intermediate state $S$. For a given layer with sequence length $L$ and key/value dimensionalities $h$, the optimization is:

$$\min_{U,S,W}\|K-US\|_F^2 + \|V-SW\|_F^2, \quad \text{rank}(S)=r$$

where $U\in\mathbb{R}^{L\times r}$, $S\in\mathbb{R}^{r\times h}$, $W\in\mathbb{R}^{r\times h}$. Alternating least-squares and SVD techniques are used for efficient, closed-form computation.

The resulting storage requirements drop from $O(2Lh)$ (for full KV cache) to $O(Lr+2rh)$, a significant reduction when $r\ll h$.

4. Integration and Decoding Workflow

During inference, EliteKV reconstructs key and value representations for attention using the low-rank projections:

• For each new decoding step, only the elite key chunks are rotated as per RoPE; the remaining key and all value chunks are reconstructed via the shared low-rank state $S$.
• The updated attention equations are:

$Q_t = x_tW_Q$

$K' = [\text{RoPElite-rotated chunks}, US]W_K$

$V' = SW_V$

$$\alpha = \text{softmax}(Q_tK'^\top/\sqrt{d}), \quad o_t = \alpha V'$$

Notably, no additional per-token RoPE computation or rotation is incurred at decode time beyond what is necessary for the elite chunks, preserving throughput parity with unmodified models.

5. Uptraining and Empirical Results

EliteKV requires brief adaptation (uptraining) on a small data subset to recover maximum performance after restructuring:

• 0.6% of RefinedWeb data, matching the original LLaMA2 training distribution.
• Optimizer: AdamW ($\beta_1=0.9$, $\beta_2=0.95$), constant learning rate, batch size 512, sequence length 4096.
• Convergence is typically achieved in a few thousand steps.

Performance benchmarks on LLaMA2-7B (eight zero-shot benchmarks) demonstrate:

Cache fraction	Method	Avg(8)
100%	LLaMA2	58.14
50%	GQA	55.33
50%	EliteKV	57.72
25%	GQA	52.59
25%	EliteKV	57.30
12.5%	GQA	50.31
12.5%	EliteKV	55.67

• At 25% cache size, EliteKV is within 0.8 points of full-cache accuracy.
• Language modeling perplexity drop is under 0.01.
• KV cache memory is reduced by up to 75% with negligible performance loss.

6. Context, Limitations, and Extensions

EliteKV achieves compressibility-flexibility trade-offs not feasible with prior approaches, largely due to its joint exploitation of head-level RoPE structure and low-rank K/V sharing. J-LRD (joint low-rank decomposition) consistently outperforms S-LRD (separate SVD) at fixed cache budgets.

Noted limitations include:

• Some uptraining on pretraining-distributed data is required, especially as compression ratios decrease; very large models trained on 15T tokens may require additional adaptation.
• Potential future avenues involve combining EliteKV with quantization, alternative linear attention mechanisms, or applying it to other positional encodings.

RoPElite-derived frequency patterns are stable across scales, and larger models converge more rapidly during uptraining. A plausible implication is that EliteKV’s methodology is robust to deployment across a variety of contemporary RoPE-based Transformer architectures (Zhou et al., 3 Mar 2025).