VATP: Value-Aware Token Pruning

• The paper introduces VATP as a method that combines attention scores with value vector norms to compute token importance, ensuring essential tokens are retained.
• VATP computes token importance via the product of accumulated attention and λ1-norm of value vectors, outperforming traditional attention-only pruning.
• Empirical results demonstrate that VATP improves throughput and task accuracy in LLMs without requiring offline calibration or adding significant computational overhead.

Value-Aware Token Pruning (VATP) is a family of structured token pruning strategies developed to accelerate LLM inference by reducing the size of the key-value (KV) cache during generation. VATP methods augment standard attention-score–based pruning by leveraging both attention weights and the explicit content of value vectors maintained in each token’s cache slot. Empirical analyses demonstrate that traditional attention-based criteria are insufficient for identifying non-contributory tokens, motivating the integration of value vector norms and, in some variants, joint similarity metrics and variance-aware fusion. VATP consistently yields improved retention of task-relevant information at aggressive pruning ratios, with negligible or no runtime overhead and without requiring offline calibration.

1. Motivation and Problem Setting

Autoregressive LLMs process sequences by incrementally appending tokens and, for each step $t$, recomputing the attention output: $$\text{Attention}(Q, K, V)_t = \sum_{i \leq t} a_i^t \cdot v_i$$ where $a_i^t$ is the attention score from the query at position $t$ to the key at position $i$, and $v_i$ is the value vector stored in the KV cache for token $i$. As model context sizes and batch sizes scale, the memory and computational overheads from maintaining ever-growing KV caches become bottlenecks. Token pruning strategies aim to selectively remove low-importance tokens from the cache, thereby improving throughput and reducing system memory requirements.

Prevailing methods (e.g., H$_2$O, Scissorhands, StreamLLM) use accumulated or sliding-window attention scores as proxies for token importance. However, empirical evidence reveals that this criterion is unreliable: some tokens (“attention sinks”) attract large aggregate attention yet have near-zero value norms, contributing negligibly to the attention output. Conversely, tokens with moderate attention but large value norm can play a critical role in model performance. These observations directly motivate value-aware approaches (Guo et al., 2024).

2. Formalization and Methodological Advances

VATP extends conventional token pruning by incorporating value norm information into token ranking:

• Token Importance Metric: For token $i$ at decoding step $t$,

$$\text{Attention}(Q, K, V)_t = \sum_{i \leq t} a_i^t \cdot v_i$$0

where $$\text{Attention}(Q, K, V)_t = \sum_{i \leq t} a_i^t \cdot v_i$$1 is the accumulated or windowed attention score, and $$\text{Attention}(Q, K, V)_t = \sum_{i \leq t} a_i^t \cdot v_i$$2 is the $$\text{Attention}(Q, K, V)_t = \sum_{i \leq t} a_i^t \cdot v_i$$3-norm of the value vector. Pruning keeps the $$\text{Attention}(Q, K, V)_t = \sum_{i \leq t} a_i^t \cdot v_i$$4 highest-scoring tokens as determined by $$\text{Attention}(Q, K, V)_t = \sum_{i \leq t} a_i^t \cdot v_i$$5 (Guo et al., 2024).

• Hybrid formulations: Convex combinations of attention scores and value norms are possible,

$$\text{Attention}(Q, K, V)_t = \sum_{i \leq t} a_i^t \cdot v_i$$6

but the product form $$\text{Attention}(Q, K, V)_t = \sum_{i \leq t} a_i^t \cdot v_i$$7 provides the best empirical performance without additional tuning.

Advanced VATP variants, such as Token Filtering, further refine token redundancy estimation by:

• Computing per-head cosine similarities between the current key/value vectors and anchor vectors maintained via exponential smoothing across past tokens;
• Aggregating redundancy signals across heads via variance-aware fusion, where head means are weighted inversely by their variability, yielding a robust $$\text{Attention}(Q, K, V)_t = \sum_{i \leq t} a_i^t \cdot v_i$$8 redundancy score per layer (Lee et al., 8 Dec 2025).

3. Algorithmic Workflow

A typical VATP-enhanced pipeline proceeds as follows:

1. Preprocessing and initialization:
   • For each token, initialize running attention scores and precompute value norms per head.
   • Unconditionally retain the first $$\text{Attention}(Q, K, V)_t = \sum_{i \leq t} a_i^t \cdot v_i$$9 “sink”/safety tokens in the cache.
2. Incremental decoding:
   • At each step $a_i^t$0, update $a_i^t$1 for all $a_i^t$2 using current attention weights.
   • When the pruning trigger fires (usually after reaching a budget limit), compute joint importance scores $a_i^t$3 for all tokens outside the sink set and/or sliding window, then select the top-ranked tokens to retain.
3. Layer- and head-wise execution:
   • All computations can be performed per-head and per-layer to maintain alignment with the transformer architecture.
   • For Token Filtering, only a configurable “tail set” (e.g., last $a_i^t$4 fraction) of layers is pruned in order to maximize computational benefit while minimizing impact on model fidelity.
4. Online adaptation:
   • Pruning thresholds for the redundancy metric are adapted at each layer via a low-rate update to match the target global prune ratio, supporting stable per-layer skip rates.
5. Forward integration:
   • Within each transformer block, after QKV projection, compute pruning scores, apply cache modifications, and output either a zero vector (for skipped tokens) or the standard attention output.

Pseudocode for this procedure in the context of Token Filtering is explicitly provided in (Lee et al., 8 Dec 2025).

4. Experimental Results and Empirical Behavior

Comparative assessments on LLaMA2-7B-chat, Vicuna-v1.5-7B, LLaMA-3 (8B), and Mistral (7B) demonstrate the empirical effectiveness of VATP:

Model	Method	Prune Ratio	Commonsense (%)	MMLU (%)	Throughput (tokens/s)	Notes
LLaMA-2-13B	TF (VATP)	50%	65.9	46.68	5.2	Stable across tasks
LLaMA-2-13B	FLAP	50%	57.41	30.81	5.8	Relies on offline calibration
SlimGPT/Probe Pruning	SlimGPT / PP	50%	51–57	22–31	4.4	Accuracy collapse at lower ratios
LLaMA-2-7B	TF (VATP)	50%	N/A	35.31	N/A	Substantially outperforms baselines
LLaMA-3-8B	TF (VATP)	50%	≥ 59.3	N/A	N/A	Baseline drops to 42–47%

For LongBench tasks on LLaMA2-7B-chat, VATP variants (H₂O w/ VATP and Scissorhands w/ VATP) outperform or match their respective baselines in at least 12 out of 16 tasks at the 50% retention setting, with especially pronounced improvements as budgets decrease below 30% (Guo et al., 2024). The overhead of value-norm computation is negligible, and inference latency is unaffected at matched budgets.

5. Theoretical Justification and Interpretive Analysis

The mechanism underpinning VATP rests on the observation that the actual contribution of a token to subsequent attention outputs is not solely a function of its attention score but also the magnitude of its value encoding. The aggregate attention output $a_i^t$5 highlights that tokens with large $a_i^t$6 but vanishing $a_i^t$7—the “attention sinks”—impose unnecessary cache costs without meaningful benefit. Conversely, tokens with substantial $a_i^t$8 but moderate $a_i^t$9 can be pivotal. Ignoring value-norm information thus either wastes cache (by retaining sinks) or discards useful context.

Variance-aware and multi-head fusion approaches introduced in Token Filtering yield resilience to momentary noise in either key or value modalities by adaptively down-weighting volatile signals. This structure further stabilizes pruning decisions in the presence of non-uniform head behavior (Lee et al., 8 Dec 2025).

6. Comparative Analysis

VATP distinguishes itself from several prevalent pruning ecosystems:

• SlimGPT, FLAP (OBS-based head pruning): These require offline calibration on held-out data and experience severe accuracy collapse at pruning ratios above 20%, especially in large LLMs.
• Probe Pruning (PP): Employs online probes but remains batch-oriented and calibration-dependent, with lower accuracy and lower throughput.
• Token Filtering (VATP, (Lee et al., 8 Dec 2025)):
  • Fully online and calibration-free;
  • Utilizes joint key-value similarity and variance weighting as redundancy indicators;
  • Adapts thresholding per batch;
  • Applies pruning only to select tail layers to optimize the accuracy-throughput frontier;
  • Empirically demonstrates dominant performance at fixed pruning budgets on reasoning, factual recall, and long-context tasks.

A summary table:

Method	Calibration Required	Pruning Decision	Pruning Granularity	Empirical Robustness
VATP / Token Filtering	No	Online	Per-token, per-layer	High
FLAP, SlimGPT	Yes	Offline	Head/channel	Low at high ratios
Probe Pruning	Partial	Probe-driven	Channel, batchwise	Moderate

7. Limitations and Future Research

• Architectural constraints: Current FlashAttention libraries do not efficiently expose per-head attention weights, limiting practical speedups during prompt prefilling for certain configurations (Guo et al., 2024).
• Generalization: VATP has not been directly extended to grouped-query attention or adaptive-budget scenarios, and optimal trade-offs between attention and value-norm contributions (e.g., learnable λ) remain unexplored.
• Head-wise extensions: Exploring head-wise or layer-wise weighting and adaptation of importance metrics is highlighted as a future direction.

A plausible implication is that as models and contexts continue to scale, the need for principled, content-aware KV cache management will become central to practical and efficient LLM deployment. VATP provides a foundation for such methods and benchmarks their empirical impact across current LLM architectures.

• "Leveraging KV Similarity for Online Structured Pruning in LLMs" (Lee et al., 8 Dec 2025)
• "Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters" (Guo et al., 2024)