KVSink: Enhancing KV Quantization in LLMs

• KVSink is a plug-and-play methodology for KV cache quantization that preserves critical attention sinks—token positions absorbing disproportionate attention mass.
• The approach dynamically identifies sink tokens at a fixed emergence layer, excluding them from quantization to recover over 95% of FP16 perplexity.
• Empirical evaluations on models like LLaMA2 show that KVSink outperforms PFN, offering robust performance and minimal computational overhead for aggressive model compression.

KVSink is a plug-and-play methodology for key-value (KV) cache quantization in LLM inference that targets the preservation of “attention sinks”—token positions that disproportionately absorb attention mass and introduce quantization sensitivity. It advances prior quantization pipelines by algorithmically predicting sink tokens during inference, permitting targeted exclusion from quantization and thereby maximizing perplexity (PPL) recovery with minimal overhead (Su et al., 6 Aug 2025).

1. Definition and Role of Attention Sinks

An attention sink is formally a token position $i$ that consistently aggregates an outsized fraction of attention mass in multiple heads and layers. For a single head at layer $l$, attention scores are defined by

$$e^{l,k}_{t,i} = \frac{\langle q^{l,k}_t, k^{l,k}_i \rangle}{\sqrt{d_k}},\quad p^{l,k}_{t,i} = \mathrm{Softmax}_i(e^{l,k}_{t,i}),$$

and the set of sinks for that head is

$$S^{l,k} = \left\{ i~|~\forall t,\; p^{l,k}_{t,i} \gg \frac{1}{t} \right\}.$$

Aggregating across all heads and layers,

$S = \bigcup_{l,k} S^{l,k}.$

Sink tokens $i \in S$ frequently show numerically small $v^{l,k}_i$ due to QKV suppression, so quantization error in $V$-cache at these indices yields disproportionately large errors in attention outputs. Formally, the attention output can be decomposed as

$$\mathrm{Attention}(Q,K,V)^t = \underbrace{\sum_{i \notin S} p_i^t v_i}_{\text{ordinary}} + \underbrace{\sum_{i \in S} p_i^t v_i}_{\text{sink bias}},$$

where the sink bias term is nearly constant across $t$ and highly sensitive to quantization of the corresponding $l$0.

2. Cross-Layer Dynamics of Extreme Activation Outliers

The emergence and propagation of extreme channel-wise outliers underlie the formation of attention sinks. Tracking the tensors per block $l$1: $l$2

$l$3

$l$4

one observes five distinct stages in outlier magnitude and position:

• Initial: Absence of extreme outliers.
• Emergence (layer $l$5): Spikes in $l$6 propagate to $l$7, creating stable outliers in $l$8.
• Stabilization: Intermediate layers sustain large, consistent outliers in $l$9 and $$e^{l,k}_{t,i} = \frac{\langle q^{l,k}_t, k^{l,k}_i \rangle}{\sqrt{d_k}},\quad p^{l,k}_{t,i} = \mathrm{Softmax}_i(e^{l,k}_{t,i}),$$0, while $$e^{l,k}_{t,i} = \frac{\langle q^{l,k}_t, k^{l,k}_i \rangle}{\sqrt{d_k}},\quad p^{l,k}_{t,i} = \mathrm{Softmax}_i(e^{l,k}_{t,i}),$$1 and $$e^{l,k}_{t,i} = \frac{\langle q^{l,k}_t, k^{l,k}_i \rangle}{\sqrt{d_k}},\quad p^{l,k}_{t,i} = \mathrm{Softmax}_i(e^{l,k}_{t,i}),$$2 subside.
• Dissipation: Counter-spikes in $$e^{l,k}_{t,i} = \frac{\langle q^{l,k}_t, k^{l,k}_i \rangle}{\sqrt{d_k}},\quad p^{l,k}_{t,i} = \mathrm{Softmax}_i(e^{l,k}_{t,i}),$$3 nullify prior stable outliers.
• Final: All activations return to baseline magnitudes.

These persistent, stable outliers correspond to emerging and sustained attention sinks, which significantly influence downstream attention and the sensitivity to quantization.

3. Limitations of the Preserve-First-N Strategy

The Preserve-First-N (PFN) strategy excludes the first $$e^{l,k}_{t,i} = \frac{\langle q^{l,k}_t, k^{l,k}_i \rangle}{\sqrt{d_k}},\quad p^{l,k}_{t,i} = \mathrm{Softmax}_i(e^{l,k}_{t,i}),$$4 tokens’ KVs from quantization. There are two core limitations:

• Attention sinks may occur at positions beyond $$e^{l,k}_{t,i} = \frac{\langle q^{l,k}_t, k^{l,k}_i \rangle}{\sqrt{d_k}},\quad p^{l,k}_{t,i} = \mathrm{Softmax}_i(e^{l,k}_{t,i}),$$5 (e.g., token 14 in LLaMA2-7B prefill). PFN cannot detect or exclude such outliers.
• Empirical evidence demonstrates sudden PPL degradation if a non-excluded sink emerges after $$e^{l,k}_{t,i} = \frac{\langle q^{l,k}_t, k^{l,k}_i \rangle}{\sqrt{d_k}},\quad p^{l,k}_{t,i} = \mathrm{Softmax}_i(e^{l,k}_{t,i}),$$6, regardless of how many initial tokens are preserved. Thus, PFN is not robust against atypical sink locations and cannot guarantee PPL stability (Su et al., 6 Aug 2025).

4. KVSink: Algorithmic Sink Token Identification and Preservation

KVSink mitigates these limitations by dynamically predicting sink positions in the prefill phase through activation outlier identification at a fixed emergence layer $$e^{l,k}_{t,i} = \frac{\langle q^{l,k}_t, k^{l,k}_i \rangle}{\sqrt{d_k}},\quad p^{l,k}_{t,i} = \mathrm{Softmax}_i(e^{l,k}_{t,i}),$$7. The algorithm:

• Selects a pre-identified outlier channel $$e^{l,k}_{t,i} = \frac{\langle q^{l,k}_t, k^{l,k}_i \rangle}{\sqrt{d_k}},\quad p^{l,k}_{t,i} = \mathrm{Softmax}_i(e^{l,k}_{t,i}),$$8 at layer $$e^{l,k}_{t,i} = \frac{\langle q^{l,k}_t, k^{l,k}_i \rangle}{\sqrt{d_k}},\quad p^{l,k}_{t,i} = \mathrm{Softmax}_i(e^{l,k}_{t,i}),$$9.
• For the hidden-state tensor $$S^{l,k} = \left\{ i~|~\forall t,\; p^{l,k}_{t,i} \gg \frac{1}{t} \right\}.$$0, determines the threshold $$S^{l,k} = \left\{ i~|~\forall t,\; p^{l,k}_{t,i} \gg \frac{1}{t} \right\}.$$1 as the $$S^{l,k} = \left\{ i~|~\forall t,\; p^{l,k}_{t,i} \gg \frac{1}{t} \right\}.$$2-th largest $$S^{l,k} = \left\{ i~|~\forall t,\; p^{l,k}_{t,i} \gg \frac{1}{t} \right\}.$$3 over token positions.
• The predicted sink set $$S^{l,k} = \left\{ i~|~\forall t,\; p^{l,k}_{t,i} \gg \frac{1}{t} \right\}.$$4 comprises all positions $$S^{l,k} = \left\{ i~|~\forall t,\; p^{l,k}_{t,i} \gg \frac{1}{t} \right\}.$$5 where $$S^{l,k} = \left\{ i~|~\forall t,\; p^{l,k}_{t,i} \gg \frac{1}{t} \right\}.$$6.

Pseudocode: $i \in S$4

The quantization step uses: $$S^{l,k} = \left\{ i~|~\forall t,\; p^{l,k}_{t,i} \gg \frac{1}{t} \right\}.$$7

Computational complexity is dominated by a single top-$$S^{l,k} = \left\{ i~|~\forall t,\; p^{l,k}_{t,i} \gg \frac{1}{t} \right\}.$$8 operation ($$S^{l,k} = \left\{ i~|~\forall t,\; p^{l,k}_{t,i} \gg \frac{1}{t} \right\}.$$9), negligible for contemporary context lengths ($S = \bigcup_{l,k} S^{l,k}.$0, $S = \bigcup_{l,k} S^{l,k}.$1).

5. Experimental Validation and Comparative Performance

Experiments span models including LLaMA2-7B/13B/70B, LLaMA2-7B-chat, Mistral-7B, LLaMA3-8B, and LLaMA3.1-8B-instruct, evaluated on Wikitext-2 and C4 with variable quantization schemes (RTN INT2/INT4, static/dynamic per-token and per-channel). Baselines are PFN($S = \bigcup_{l,k} S^{l,k}.$2) and KVQuant.

Results Overview

Model/Method	FP16 PPL	PFN(5) PPL	KVSink(5) PPL
LLaMA2-70B, 4-bit	2.5	59.5	5.0

Across all tested LLMs, preserving $S = \bigcup_{l,k} S^{l,k}.$3 sinks via KVSink recovers $S = \bigcup_{l,k} S^{l,k}.$4 of FP16 PPL, while PFN requires $S = \bigcup_{l,k} S^{l,k}.$5 and still fails in some cases.

For KVSink + KVQuant integration (Wikitext-2, LLaMA2-7B, 2-bit quantization, 1% outlier isolation):

• KVQuant PPL: 5.53
• KVSink-5 PPL: 5.44

On C4 dataset: 6.94 $S = \bigcup_{l,k} S^{l,k}.$6 6.81 (KVQuant $S = \bigcup_{l,k} S^{l,k}.$7 KVSink).

Importantly, lowering the outlier-isolation budget (to 0.1%) does not hurt PPL with KVSink, enabling more aggressive model compression.

6. Guidelines for Practical Integration

KVSink integration follows static configuration per architecture:

• Identify $S = \bigcup_{l,k} S^{l,k}.$8 once for each model (cf. Table 9).
• Target $S = \bigcup_{l,k} S^{l,k}.$9 sink tokens (empirically sufficient for tested LLMs).
• In static-quant pipelines, also exclude sink set $i \in S$0 from calibration.
• Implementation overhead is minimal: $i \in S$10.04–0.05 ms latency per 4K-token sequence (A100), $i \in S$2 MB memory for storage.
• Recommended practice: wrap quantizer, perform outlier detection at $i \in S$3, and “freeze” sink set for remainder of inference.

KVSink thus provides a methodologically grounded, low-cost approach for attention sink preservation, consistently improving quantized inference accuracy over both PFN and state-of-the-art alternatives (Su et al., 6 Aug 2025).