GVote: Adaptive KV-Cache Compression for LLMs

• GVote is an adaptive KV-cache compression scheme that forecasts future query demands using Monte Carlo sampling and union-based voting.
• It eliminates the need for fixed budget settings by dynamically retaining keys based on the predicted attention structure for each request.
• Empirical results show GVote achieves a 2× memory reduction with comparable or improved accuracy across diverse benchmarks such as GSM8K, RULER, and Longbench.

GVote is an adaptive KV-cache compression scheme for LLM inference that eliminates manual budget specification by predicting future attention demand and retaining the keys most likely to be needed by later queries. It is motivated by the observation that fixed compression ratios impose a “Procrustes’ bed” on heterogeneous workloads: aggressive compression can damage difficult tasks, while conservative compression wastes memory on easier ones. GVote operationalizes a different criterion—“important keys are the aggregation of keys required by future queries”—by using Monte-Carlo style sampling of potential future queries and a union-based voting rule over the keys selected by those samples. The reported outcome is a superior accuracy-efficiency trade-off, including 2$\times$ memory reduction with higher or comparable accuracy across benchmarks such as GSM8K, RULER, and Longbench (Tang et al., 3 Sep 2025).

1. Problem setting and motivation

KV-cache compression addresses the rapidly growing memory footprint incurred by autoregressive decoding in LLMs. In the formulation underlying GVote, the central deficiency of prior methods is not merely imperfect saliency estimation, but the requirement that users manually specify a fixed cache budget, such as keeping a constant fraction of the cache for all requests. The paper characterizes this as a “Procrustes’ bed” limitation: diverse workloads are forced into a single compression ratio, even though the appropriate budget varies by task and request (Tang et al., 3 Sep 2025).

The practical consequences are explicitly asymmetric. Underprovisioning causes difficult tasks, including mathematical reasoning, to suffer catastrophic accuracy degradation under aggressive compression. Overprovisioning wastes memory on simpler tasks when the configured budget is overly conservative. This suggests that the relevant optimization target is not a global compression ratio, but a request-conditional retained set that adapts to the attention structure of the current context.

GVote is designed around that premise. Rather than fixing a budget first and ranking keys second, it estimates which keys will matter for plausible future queries and lets the resulting keep-set determine the budget. A plausible implication is that GVote reframes KV-cache compression as a forecasting problem over future attention rather than as a static pruning problem.

2. Core principle: future-query aggregation

The method’s guiding principle is stated directly: the optimal cache is the union of keys needed for future queries (Tang et al., 3 Sep 2025). Since those future queries are unavailable at pruning time, GVote approximates them statistically. The paper assumes that the distribution of hidden states from which queries are computed is approximately Gaussian, attributed to layer norm, and uses this approximation to simulate future attention behavior.

The current query is first used to derive a proxy for near-term demand. Let the current attention weights be

$$\mathbf{A}_0 = \mathrm{softmax}\left(\frac{\mathbf{Q}_0 \mathbf{K}_{1:L}^{\top}}{\sqrt{d_k}}\right).$$

From these weights, GVote selects the smallest candidate set $\mathcal{C}_0$ covering a nucleus probability mass $p_{\text{nuc}}$:

$$\mathcal{C}_0 \gets \mathrm{TopP}(\mathbf{A}_0, p_{\text{nuc}}), \qquad B_{\text{step}} = |\mathcal{C}_0|.$$

Here, $B_{\text{step}}$ acts as a single-step proxy budget. The method then estimates the mean $\mu$ and variance $\sigma^2$ of the hidden state and samples synthetic hidden states from $\mathcal{N}(\mu, \operatorname{diag}(\sigma^2))$. These sampled hidden states are projected into query space, with rotary positional embedding applied as in the model, producing plausible future queries (Tang et al., 3 Sep 2025).

The “vote” in GVote is the aggregation mechanism over these sampled queries. Each synthetic query selects a top-$k$ key set, where $$\mathbf{A}_0 = \mathrm{softmax}\left(\frac{\mathbf{Q}_0 \mathbf{K}_{1:L}^{\top}}{\sqrt{d_k}}\right).$$0, and the final retained set is the union across all such selections:

$$\mathbf{A}_0 = \mathrm{softmax}\left(\frac{\mathbf{Q}_0 \mathbf{K}_{1:L}^{\top}}{\sqrt{d_k}}\right).$$1

This is described as bottom-up aggregation: a key is kept if it is selected by at least one plausible future query. The resulting cache size is therefore variable and data-dependent rather than manually imposed.

3. Algorithmic workflow

The paper presents GVote as a multi-stage procedure. The first stage is current query analysis, where the model computes the present-step attention distribution and extracts $$\mathbf{A}_0 = \mathrm{softmax}\left(\frac{\mathbf{Q}_0 \mathbf{K}_{1:L}^{\top}}{\sqrt{d_k}}\right).$$2 via nucleus selection. This stage supplies a request-local estimate of how many keys are immediately relevant.

The second stage is hidden-state statistics extraction. GVote computes the hidden-state mean and variance, ignoring sink positions, and uses those statistics to define the Gaussian sampling distribution for future-query simulation. This statistical compression of the local hidden-state geometry is what makes the subsequent Monte Carlo step tractable (Tang et al., 3 Sep 2025).

The third stage is Monte Carlo sampling of future queries. For each of $$\mathbf{A}_0 = \mathrm{softmax}\left(\frac{\mathbf{Q}_0 \mathbf{K}_{1:L}^{\top}}{\sqrt{d_k}}\right).$$3 samples, the method draws a synthetic hidden state, projects it through $$\mathbf{A}_0 = \mathrm{softmax}\left(\frac{\mathbf{Q}_0 \mathbf{K}_{1:L}^{\top}}{\sqrt{d_k}}\right).$$4, applies RoPE, computes attention scores over all keys, and performs top-$$\mathbf{A}_0 = \mathrm{softmax}\left(\frac{\mathbf{Q}_0 \mathbf{K}_{1:L}^{\top}}{\sqrt{d_k}}\right).$$5 selection with $$\mathbf{A}_0 = \mathrm{softmax}\left(\frac{\mathbf{Q}_0 \mathbf{K}_{1:L}^{\top}}{\sqrt{d_k}}\right).$$6. In compact form:

$$\mathbf{A}_0 = \mathrm{softmax}\left(\frac{\mathbf{Q}_0 \mathbf{K}_{1:L}^{\top}}{\sqrt{d_k}}\right).$$7

followed by top-$$\mathbf{A}_0 = \mathrm{softmax}\left(\frac{\mathbf{Q}_0 \mathbf{K}_{1:L}^{\top}}{\sqrt{d_k}}\right).$$8 selection on the resulting attention distribution. The fourth stage is the union operation over sampled keep-sets, which returns the pruned cache $$\mathbf{A}_0 = \mathrm{softmax}\left(\frac{\mathbf{Q}_0 \mathbf{K}_{1:L}^{\top}}{\sqrt{d_k}}\right).$$9.

The paper’s algorithmic summary emphasizes that the output budget is not separately tuned. Instead, the final retained budget is

$\mathcal{C}_0$0

which emerges from the sampled future attention structure. This makes GVote a budgeting mechanism and a pruning mechanism simultaneously.

4. Automatic budget determination and heterogeneity handling

A distinguishing property of GVote is that it does not require a fixed memory usage ratio. The budget is determined automatically from the aggregation of sampled future-query selections, and the paper describes this adaptation as operating per request and per head (Tang et al., 3 Sep 2025).

This design directly targets workload heterogeneity. For each new prompt, GVote assesses the current attention context, simulates plausible future attention demands using synthetic queries, and retains all keys deemed important in at least one sampled scenario. Easy workloads may therefore produce a small retained set, while harder workloads may preserve substantially more keys. The method is intended to avoid the paired failure modes that fixed-budget methods exhibit across heterogeneous tasks.

A common misconception is that “adaptive” KV-cache compression simply means redistributing a fixed total budget across heads. The comparison in the paper differentiates GVote from that regime. AdaKV is described as adaptive per head, but still dependent on a fixed global budget; GVote, by contrast, eliminates manual budget specification altogether. Another misconception is that GVote claims to know future queries exactly. The method does not do so. It uses a Gaussian approximation to hidden states and Monte Carlo sampling to estimate plausible future query demands, which is a probabilistic surrogate rather than an oracle.

The ablation results further indicate that the nucleus threshold $\mathcal{C}_0$1 and the sampling number $\mathcal{C}_0$2 provide a controllable trade-off: higher values improve accuracy at some increase in cache usage. This suggests that GVote replaces direct manual memory budgeting with higher-level statistical sensitivity controls.

5. Empirical evaluation

The reported evaluation covers multiple benchmarks and model families. The benchmark suite includes GSM8K, RULER-4K, Longbench, and additional settings such as Multi-Doc QA and Single-Doc QA extending to >50k tokens. The models tested include Llama3.1-8B-Instruct, Llama3.2-3B-Instruct, Qwen2.5-7B-Instruct, and Qwen2.5-14B-Instruct (Tang et al., 3 Sep 2025).

The paper compares GVote to StreamLLM, SnapKV, and AdaKV. StreamLLM is characterized as a sliding-window, fixed-window attention method; SnapKV prunes based on recent tokens; AdaKV adaptively allocates per head but still requires a fixed total budget. Against these baselines, GVote is reported to achieve 2$\mathcal{C}_0$3 memory reduction while maintaining higher or comparable accuracy.

The summary highlights task-level behavior as well. On Multi-Doc QA, GVote yields approximately 0.35 accuracy with only 10% average memory usage, while competing methods require at least twice as much memory for worse accuracy. Across eight datasets, the paper states that GVote’s points in the accuracy/cache-usage plane lie on or near the best accuracy-memory trade-off, and that the same qualitative pattern holds across multiple architectures and model sizes.

The method’s internal approximation quality is also evaluated. The Monte Carlo synthetic queries exhibit mean attention overlap 0.929 with ground-truth top-0.95 tokens and a correlation coefficient $\mathcal{C}_0$4. These measurements support the paper’s claim that the sampled-query mechanism captures reuse-relevant KV pairs with high fidelity (Tang et al., 3 Sep 2025).

6. Significance, interpretation, and boundaries

GVote’s significance lies in how it changes the interface between model deployment and KV-cache compression. Instead of requiring workload-specific tuning of a compression ratio, it computes a retained set from predicted attention demand. This is important in serving environments where prompt difficulty, context length, and reasoning structure vary substantially across requests.

The method is also conceptually notable for using a union of sampled top-$\mathcal{C}_0$5 selections rather than a single deterministic saliency score. That choice privileges recall of potentially useful keys across plausible futures. A plausible implication is that GVote is implicitly conservative with respect to diverse downstream attention trajectories: any key selected by one plausible future query is preserved.

At the same time, the paper does not describe GVote as budget-free in an absolute sense. It introduces statistical control parameters such as the nucleus threshold and number of samples, and these affect the accuracy-memory trade-off. The method therefore removes manual budget specification in the sense of fixed retention ratios, but it still relies on configurable sampling and coverage parameters. That distinction is important for interpreting the claim of adaptivity precisely.

Within the scope reported, GVote is best understood as a principled, request-adaptive KV-cache compression framework that forecasts future query requirements via Monte Carlo sampling from a Gaussian hidden-state approximation and converts those forecasts into a variable keep-set through voting aggregation. Its empirical profile, as summarized in the paper, positions it as an alternative to fixed-budget compression schemes for heterogeneous long-context inference workloads (Tang et al., 3 Sep 2025).