VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs

Abstract: The key-value (KV) cache is the main memory bottleneck in long-context LLM inference. Two leading training-free families are both structurally limited: token-selection methods (SnapKV, Ada-KV) score importance from an observation window and evict low-scoring tokens, but eviction is irreversible -- so when the importance signal degrades under query-agnostic reuse, accuracy collapses by 11-15 points; uniform low-rank coding keeps every token but spends equal rank everywhere, wasting budget. We observe that both failures share one cure: rank should be allocated, not evicted. We present VarRate, a training-free KV codec that assigns each token a variable low-rank budget by its query salience, keeping every token at a nonzero rank. Comparable adaptive-rank codecs reach this allocation only through training; VarRate requires none. Because no token is dropped, it degrades by only 3.5-5.5 points where query-aware selection collapses. At a matched 20% budget on LongBench (16 tasks), VarRate stays within 0.8 points of the uncompressed model on both Llama-3.1-8B and Qwen2.5-7B. Averaged over the two, it is the strongest matched-memory compressor. It significantly beats its uniform-rank ablation on both models. Against KVzip, a method purpose-built for query-agnostic reuse, it is accuracy-equivalent in three of four settings and within a point overall, at about one-eighth the prefill overhead.

• The paper introduces VarRate, a training-free codec that uses water-filling to assign query-aware low-rank budgets per token while retaining every token, avoiding the irreversible losses of selection methods.
• At a matched 20% KV-cache budget, VarRate achieves 45.69 on Llama-3.1-8B and 48.00 on Qwen2.5-7B, outperforming uniform-rank coding by 2.22 and 17.50 points respectively while strictly dominating Palu across tested budgets.
• VarRate loses only 3.53–5.50 points when compressed caches are reused across queries, matching KVzip in three of four settings at about one-eighth its prefill overhead, but requires roughly 10–12% minimum budget and does not match quantization on some summarization tasks.

Motivation and problem statement

Long-context inference in LLMs is bottlenecked by the key–value (KV) cache, whose size grows linearly with prompt length and dominates GPU memory at decode time. The paper identifies two dominant training-free compression families and argues that each fails for a complementary structural reason. Token-selection methods (SnapKV, PyramidKV, Ada-KV) score tokens from a recent observation window—an essentially query-dependent signal—and irreversibly evict low scorers; when a compressed cache is reused across queries (prefix caching, multi-turn dialogue), the signal goes stale and every misjudged token is lost permanently. Uniform low-rank coding (Palu) keeps every token but assigns identical rank to all, wasting budget on trivial tokens and starving critical ones. The paper's central thesis is that both failures share one remedy: rank should be allocated rather than evicted or spread uniformly.

The VarRate codec

VarRate is a training-free KV codec that assigns each token a variable low-rank budget proportional to its query salience. Per layer, post-RoPE keys are un-rotated to their pre-RoPE form (RoPE inflates numerical rank) and stacked with values into one per-token vector $z_t \in \mathbb{R}^D$, so a single basis serves all heads under grouped-query attention. A strided set of anchor tokens plus the most recent window are stored exactly; every other token is coded as a residual to the mean of its $c$ nearest anchors. Residuals are projected onto a shared per-layer PCA basis calibrated offline from six unlabeled contexts—no gradients, no fine-tuning.

The allocation itself is water-filling: SnapKV salience scores $\hat{s}_t$, min–max normalized over coded tokens, set ranks $$r_t = \mathrm{clip}(r_{\min} + \lambda\hat{s}_t,\ r_{\min},\ R)$$, with $\lambda$ chosen so total storage matches the flat codec's budget at the same target ratio $\kappa$. Two properties hold by construction: comparisons are memory-matched, and since $r_{\min} > 0$, no token is ever dropped—a stale signal merely spends the budget suboptimally, coarsening a misjudged token rather than destroying it. Because the basis is ordered by singular value and orthonormal, truncation is nested (Matryoshka-style) without any training. The projection is computed once at full rank $R$ and masked, so variable rate adds no arithmetic over fixed-rank coding; coefficients remain quantizable, composing VarRate with the orthogonal quantization axis.

To the authors' knowledge this is the first training-free method setting graded per-token rank from a query-salience signal; prior adaptive-rank codecs either train a gate (DynaKV) or split by fixed position (OjaKV).

Accuracy at matched budget

On LongBench (16 tasks, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct), at a 20% KV budget VarRate lands within 0.3 points of the uncompressed ceiling on Llama (45.69 vs. 46.01) and 0.8 on Qwen (48.00 vs. 48.79), with the best two-model mean of any matched-memory compressor (46.85). The per-model picture is carefully qualified: on Llama it ties PyramidKV and SnapKV (paired bootstrap CIs straddle zero); on Qwen it wins significantly (+3.10 over PyramidKV, $p{=}0.001$; +1.75 over SnapKV, $p{=}0.022$). Against memory-matched KIVI-2 it ties on both models—the paper explicitly declines to claim a win over quantization, noting that KIVI-4's nominal edge on Qwen comes at 31% memory, 1.55× what other methods spend.

The strongest result is against the paper's own uniform-rank ablation ("flat"): +2.22 points on Llama ($c$0, winning 13/16 tasks) and +17.50 on Qwen ($c$1, 15/16). Qwen's four-KV-head cache is especially hostile to uniform low-rank coding—flat collapses to 30.50 overall, losing two-thirds of its multi-document QA score—and salience-weighted allocation recovers nearly all of it. Against Palu, VarRate strictly dominates at every budget tested: at keep-20%, Palu scores 0.4 on passage retrieval versus VarRate's 99.5, becoming viable only near its designed keep-50–70% regime. Notably, even the paper's own flat codec beats Palu at matched budgets, separating a basis-construction gain from the larger salience-allocation gain.

Robustness under query-agnostic reuse

Compressing a document once with the question absent collapses query-aware selection: SnapKV loses 12.96 and 14.66 points on Llama and Qwen respectively, Ada-KV 11.01 and 15.28, while the uncompressed control barely moves. Expected Attention, despite being built for this regime, collapses hardest (−21.68), because its future-query estimate derives from a prefill that normally contains the question. VarRate degrades by only 3.53 and 5.50 points—the direct consequence of allocation over eviction.

Against KVzip, which repairs the signal by re-encoding the context with the full model, the outcome is honestly mixed. After granting KVzip the same 64-token pin VarRate inherits from SnapKV (a correction that erases an earlier apparent advantage), the two are accuracy-equivalent in three of four model×regime cells; KVzip leads the fourth (Qwen reuse) by 4.63 points and the pooled mean by 0.77 (per-example paired bootstrap, $c$2/cell). The paper claims no accuracy advantage over KVzip anywhere; its claim is cost: comparable robustness at roughly one-eighth of the prefill overhead (+44.0% vs. +372.2%), placing VarRate on the accuracy–cost frontier while cheaper methods sit ten-plus points below both.

Cost and overhead

VarRate's measured prefill overhead is +44.0% against +2.0% for selection and +24.4% for flat, though profiling attributes most of this to an unfused fp32 implementation: the codec's FLOP count is only 2.61% of one forward pass, and a bit-identity-controlled bf16 variant cuts overhead to +25.1% with accuracy unchanged (max $c$3). Decode throughput is unchanged (37.4 vs. 37.5 tok/s) because the codec reconstructs a full-length cache; realizing the memory saving at run time requires a fused kernel, following the field's standard reconstruct-in-place convention. End-to-end wall-clock shows the tax concentrated in prefill-heavy retrieval workloads (+41.8%) and diluted to noise on decode-heavy summarization (+2.6%).

Ablations and mechanism

Both components carry weight. Replacing salience with a random signal costs 6.5 retrieval points; replacing water-filling with a binary top-$c$4 split ties on retrieval but falls below even uniform rank on summarization (29.0 vs. 30.1)—starving unselected tokens destroys exactly the distributed context low-rank coding preserves. An oracle allocating by gold answer span reaches 100.0 on passage retrieval against attention salience's 99.0, so the computable signal is within a point of optimal. Negative controls refute a "recency-with-extra-steps" explanation: shuffling real scores across tokens, or keeping only the recency/attention-sink positional shape without content, both underperform—even falling below flat on Qwen—establishing that content alignment, not gradedness alone, does the work. A structurally different second-order Fisher-sensitivity signal ties the default on both models, indicating the heuristic choice is not overfit. Calibration across three independently built bases moves results by at most 1.8 points, inside confidence intervals.

Two mechanistic findings qualify the design narrative. First, per-token rank logs over millions of tokens show 0.0% occupancy at the floor whether $c$5 is 16 or 0, so the floor is slack safety margin, not an active mechanism—refuting the stronger hypothesis that water-filling would rediscover eviction and subsume selection as a special case. Second, allocated rank follows a decile profile (attention-sink bump at document start, dip through the middle, sharp climb toward the query), confirming the signal is dominated by recency and query proximity with content alignment supplying the remainder.

Limitations and open questions

The paper is unusually candid about boundaries. At aggressive budgets the advantage reverses: at keep-8%, SnapKV matches or beats VarRate, and on Qwen VarRate collapses to flat's level (passage 29.6, multi-hop 4.2)—below a viability threshold, a shared low-rank basis cannot represent any token usefully, whereas exact selection always keeps some tokens perfectly. Parity returns by keep-10% (Llama) and keep-12% (Qwen). Quantization retains a structural summarization advantage on distributed information (KIVI reaches ~35 on GovReport vs. VarRate's ~30) that no better salience signal is expected to close, since rank reduction discards coordinate directions regardless of allocation. The stride hyperparameter embodies an unresolved trade-off between exact structural preservation and reconstruction fidelity: wider stride wins at keep-10% but costs 0.71 points on the full suite at keep-20%. On RULER, both compressors give up ~8 points against uncompressed, and the gap between them is within noise at the tested sample size. The 128K needle sweep shows genuine but unexplained non-monotonicity (SnapKV's 64K dip and 128K recovery lack a confirmed mechanism), and the paper discloses a corrected data-aggregation bug affecting an earlier version of its main table. Finally, no direct empirical comparison is made against trained adaptive-rank codecs (MatryoshkaKV, DynaKV, STAR-KV), framed as a difference in kind rather than matched-budget accuracy.

Conclusion

VarRate demonstrates that making a cheap, query-aware importance signal survivable—by allocating variable low-rank budget via water-filling over a shared PCA basis, never dropping a token—yields robustness under cache reuse that otherwise requires either expensive signal repair (KVzip) or training (DynaKV). Its central supported claims are the significant margin over its own uniform-rank ablation on two model families, strict dominance of Palu at every tested budget, graceful degradation where selection loses 11–15 points, and parity with KVzip in three of four reuse settings at roughly one-eighth the cost. The method's operating range is bounded below (~keep-10–12%) and ceded to quantization on summarization, leaving open how a shared-basis codec might extend viability into more aggressive regimes.