---
title: Sink-Aware Pruning for Efficient Model Compression
url: https://www.emergentmind.com/topics/sink-aware-pruning
type: topic
---

# Sink-Aware Pruning for Efficient Model Compression

Sink-aware pruning denotes a class of compression methods that uses sink behavior as an explicit pruning signal. In recent work, the underlying sink can be a BOS-directed attention head in autoregressive LLMs, a transient high-mass position in a diffusion language model, a dominant accumulator of probability mass in a Vision Transformer, a spatially persistent high-attention visual token in a Video LLM, an activation-level outlier graph token in a graph language model, or a preserved sink region in long-video KV-cache management. The shared premise is that sink-like behavior is mechanistically structured, but its functional meaning is model- and modality-dependent: in some settings it marks redundancy that is safe to remove, whereas in others it is more useful as an anchor for grouping, routing, or cache preservation than as an object of direct deletion [2601.06787] [2602.17664] [2605.22372] [2604.20937] [2606.03712] [2512.05081].

## 1. Mechanistic basis of sink behavior

The concept builds on the broader attention-sink phenomenon: some components allocate or attract a disproportionate amount of attention even when they are not semantically important. In causal LLMs, the canonical case is the BOS token at position \(k=0\). A head that is not needed for meaningful routing can still absorb leftover probability mass because softmax forces each query to distribute total attention mass \(1\); such heads can therefore act as attention dumping grounds. In the formulation of "Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning," high-BOS sink heads are functionally redundant, especially in deeper layers, where other routes for information integration already exist [2601.06787].

This basic picture does not transfer unchanged across architectures. "Sink-Aware Pruning for Diffusion Language Models" argues that the AR heuristic of treating sinks as stable global anchors does not hold for DLMs, because sink locations shift across denoising timesteps and are often transient rather than structurally essential [2602.17664]. In Vision Transformers, "ASAP: Attention Sink Anchored Pruning" treats the sink as a structurally predictable attractor in a Lazy Random Walk rather than as pathological noise; sink-aware compression then uses the sink as a geometric anchor for separating redundant background tokens from semantically important foreground tokens [2605.22372]. In Video LLMs, "Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs" identifies sink tokens as semantically uninformative tokens that attract excessive attention and distort visual evidence under aggressive token budgets [2604.20937].

A central misconception addressed across these works is that high attention or large internal activation automatically implies semantic importance. The graph-language-model study "When Graph Tokens Sink" makes this point particularly explicitly: graph sink tokens are activation-salient, but they are not necessarily the dominant attention targets of query tokens and are not the primary semantic or structural carriers for downstream prediction [2606.03712].

## 2. Formal definitions and scoring rules

In the BOS-based LLM setting, attention weights are written as \(\alpha_{t,k}^{(\ell,h)}\), where \(t\) is the query position, \(k\) the key position, \(\ell\) the layer, \(h\) the head, and \(T\) the sequence length. The sink score of token position \(k\) is the average attention received from all queries,
\[
\operatorname{sink\text{-}score}_k^{(\ell,h)} = \frac{1}{T}\sum_{t=0}^{T-1} \alpha_{t,k}^{(\ell,h)}.
\]
For BOS, \(k=0\), so the head-wise BOS sink score is
\[
S_{\text{BOS}}^{(\ell,h)} = \operatorname{sink\text{-}score}_0^{(\ell,h)},
\]
and the layer-wise BOS sink score is
\[
S_{\text{BOS}}^{(\ell)} = \frac{1}{H}\sum_{h=1}^{H} S_{\text{BOS}}^{(\ell,h)}.
\]
This score is computed by extracting attention maps during inference and averaging the BOS attention mass across query positions [2601.06787].

For DLMs, sinkness is defined over the generation trajectory. With attention matrix \(\mathbf{A}^{(t)} \in \mathbb{R}^{N \times N}\) at generation step \(t\), the incoming attention mass to token position \(i\) is
\[
m_t(i) = \sum_{j=1}^{N} A^{(t)}_{j,i}.
\]
The trajectory-averaged attention mass is
\[
\bar{m}(i) = \frac{1}{T}\sum_{t=1}^{T} m_t(i),
\]
with spatial variance
\[
\sigma^2_{\text{spatial}} = \mathrm{Var}_{i}\!\left(\bar{m}(i)\right).
\]
To capture sink migration, the paper defines a sink-set centroid
\[
c_t = \frac{\sum_{i \in \mathcal{S}_t} m_t(i) \cdot i}{\sum_{i \in \mathcal{S}_t} m_t(i)},
\]
and temporal variance
\[
\sigma^2_{\text{temporal}} = \mathrm{Var}_{t}(c_t).
\]
A token \(j\) is treated as a sink at step \(t\) if
\[
m_t(j) \;>\; \frac{1}{S-1}\sum_{k \neq j} m_t(k) \;+\; \epsilon,
\]
with a sigmoid relaxation
\[
\phi_t(j) = \sigma\!\left(m_t(j) - \frac{1}{S-1}\sum_{k \neq j} m_t(k) - \epsilon\right),
\]
an averaged sink estimate
\[
\bar{\phi}(k) = \frac{1}{|\mathcal{T}|}\sum_{t \in \mathcal{T}} \phi_t(k),
\]
and a suppression weight
\[
\omega_j = 1 - \bar{\phi}(j).
\]
The formal objective is not to preserve all sinks, but to identify unstable sink candidates and discount them in pruning importance [2602.17664].

In ASAP, each attention matrix is interpreted as a Markov transition matrix. For layer \(l\), with head-averaged attention \(A^l\), the residual connection is incorporated through a Lazy Random Walk,
\[
\tilde{A}^l = \alpha A^l + (1 - \alpha) I,
\]
and the cumulative information flow up to layer \(t\) is
\[
P^{(t)} = \tilde{A}^1 \times \tilde{A}^2 \times \cdots \times \tilde{A}^t.
\]
Sink emergence is detected by
\[
t^* = \min \left\{ t \;\middle|\; \max_{j \geq 1} \sum_{i=1}^{N} P^{(t)}_{i,j} > \tau \right\}, \quad
x_s = \arg\max_{j \geq 1} P^{(t^*)}_{0,j},
\]
and the pruning signal is the diffusion distance to the sink,
\[
D(x_i, x_s) = \left\| P^{(t^*)}_{i,*} - P^{(t^*)}_{s,*} \right\|_2.
\]
Here, small \(D(x_i, x_s)\) indicates sink-proximal, likely redundant background; large \(D(x_i, x_s)\) indicates a distinct trajectory and likely foreground semantics [2605.22372].

In Video LLMs, the sink score is defined from attention persistence across frames:
\[
s_i=\text{MinMax-Norm}(\hat{s}_i^{w}), \qquad \hat{s}_i=\sum_{t=1}^{T}A_i^t.
\]
A token with consistently high attention across many frames receives a larger sink score, and the exponent \(w\) sharpens the sink-score distribution [2604.20937].

The GLM formulation is different again. A graph token is treated as a sink token if its hidden state has a sufficiently large RMS-normalized activation on at least one sink dimension:
\[
\{j \in \mathcal{I}_g \mid \phi(\mathbf{x}_j^{l-1}) \ge T\},
\]
with
\[
\phi(\mathbf{x}) = \max_{d \in D_{\text{sink}}} \mathrm{RMSNorm}_d(\mathbf{x}^{l-1}).
\]
The main experiments use \(T = 15.0\) for both LLaGA and TEA-GLM. This definition is activation-level rather than attention-level, which is precisely why the paper distinguishes graph sink tokens from classical attention sinks [2606.03712].

## 3. Algorithmic forms of sink-aware pruning

In BOS-based head and block pruning, the procedure is explicitly structured. Head-wise pruning runs a calibration or evaluation pass, extracts attention maps, computes \(S_{\text{BOS}}^{(\ell,h)}\) for every head, ranks heads by descending BOS sink score, and prunes the highest-scoring heads by zeroing their output-projection slice,
\[
W_O^{(\ell,h)} \leftarrow \mathbf{0}.
\]
Layer-wise pruning averages head scores into \(S_{\text{BOS}}^{(\ell)}\), ranks layers by descending score, prunes the highest-scoring transformer blocks, and excludes the first and last layers for stability [2601.06787].

In DLMs, sink-aware pruning is not introduced as a new backend but as a wrapper around existing post-training methods. Layer input activations \(\mathbf{X} \in \mathbb{R}^{S \times C_{\mathrm{in}}}\) are sink-masked by
\[
\widetilde{\mathbf{X}}_{j,:} = \omega_j \cdot \mathbf{X}_{j,:}.
\]
For Wanda, the baseline score
\[
S_{ij} = |W_{ij}| \cdot \lVert X_{\cdot j} \rVert_2
\]
becomes
\[
\widetilde{S}_{ij} = |W_{ij}| \cdot \lVert \widetilde{X}_{\cdot j} \rVert_2.
\]
For SparseGPT, the Hessian approximation is recomputed from sink-masked inputs,
\[
\widetilde{H} = \frac{1}{|\mathcal{D}|}\sum_{n} \widetilde{\mathbf{X}}_n^\top \widetilde{\mathbf{X}}_n + \lambda \mathbf{I}.
\]
The method uses WikiText-2, 128 randomly sampled sequences, each truncated to 2048 tokens, to estimate sink statistics and create pruning masks offline [2602.17664].

ASAP performs a single-shot token partitioning rather than iterative local pruning. After identifying the sink \(x_s\) and computing diffusion distances, tokens are assigned to radial semantic levels by
\[
C_k = \left\{ x_i \;\middle|\; k-1 \leq K \cdot \tilde{D}(x_i, x_s) < k \right\}, \quad k = 1, \dots, K,
\]
and the sink-proximal background cluster is compressed with Transition Weight Pooling,
\[
\mathbf{m}_1 = \sum_{x_i \in C_1} w_i \mathbf{f}_i, \qquad
w_i = \frac{\exp(D(x_i, x_s))}{\sum_{x_j \in C_1} \exp(D(x_j, x_s))}.
\]
The core pipeline detects the layer where the sink emerges, computes a cumulative transition matrix, measures token-to-sink diffusion distance, clusters tokens radially, compresses the background cluster, and optionally applies a plug-in pruning stage for extreme compression [2605.22372].

SToP modifies both spatial and temporal pruning in Video LLMs. In sink-aware spatial pruning, raw attention is replaced by a sink-penalized score,
\[
\tilde{A}_i^t = A_i^t - \mu_s \cdot s_i.
\]
In sink-aware temporal pruning, the pruning set becomes
\[
P_t = \{i \mid \text{sim}(H_v^{i,t}, H_v^{i,t+1}) + \mu_t \cdot s_i > \tau,\; i \in [1,n_v] \}.
\]
The method is plug-and-play: STSP is applied to VisionZip and FastVid, while Holitom uses both STTP and STSP [2604.20937].

Deep Forcing presents a sink-aware KV-cache variant for autoregressive video diffusion. Participative Compression divides the cache into Sink, Recent, and Candidate regions, scores candidate keys by recent-query participation,
\[
\phi_j = \sum_{r=1}^{R} \mathbf{q}_r^\top \mathbf{k}_j,
\]
selects the top-\(C\) candidate tokens, and constructs
\[
K_{\text{compressed}} = [K_{\text{sink}} \Vert K_{\text{top}} \Vert K_{\text{rct}}],
\qquad
V_{\text{compressed}} = [V_{\text{sink}} \Vert V_{\text{top}} \Vert V_{\text{rct}}].
\]
This is described as sink-aware and importance-aware KV-cache pruning rather than token pruning in the ViT or LLM sense [2512.05081].

## 4. Empirical results across domains

In autoregressive LLMs, BOS-based pruning preserves downstream performance better than weight- or activation-based criteria. On Gemma-3-4B at 12.5% pruning, the dense average is **0.648**, BOS pruning **0.641**, Wanda-SP **0.594**, and Mag-SP **0.543**. On Qwen3-4B at 12.5%, the dense average is **0.649**, BOS pruning **0.643**, Wanda-SP **0.618**, and Mag-SP **0.502**. On Llama-3.1-8B at 25.0%, the dense average is **0.673**, BOS pruning **0.631**, Wanda-SP **0.572**, and Mag-SP **0.403**. Layer-level pruning also performs strongly: on Gemma-3-4B at 12.5%, BOS averages **0.589** versus ShortGPT **0.456** and Top-Down **0.439**; on Qwen3-4B at 12.5%, BOS averages **0.570** versus ShortGPT **0.544** and Top-Down **0.541**. These experiments use Gemma-3-4B, Llama-3.1-8B, and Qwen3-4B, a maximum sequence length of **4096 tokens**, and **lm-evaluation-harness** with WikiText-2, ARC-Easy, BoolQ, OpenbookQA, PIQA, Winogrande, ARC-Challenge, HellaSwag, and MMLU [2601.06787].

In DLMs, sink-aware masking improves the quality-efficiency trade-off under matched compute. On LLaDA at **50% sparsity**, Wanda averages **52.70** and Sink-aware Wanda **53.18**, while SparseGPT averages **52.34** and Sink-aware SparseGPT **52.36**. On Dream at **50% sparsity**, SparseGPT improves from **54.40** to **54.58** with the sink-aware variant. On LLaDA-1.5 at **50% sparsity**, Sink-aware Wanda improves from **53.78** to **54.02**, and Sink-aware SparseGPT from **53.91** to **54.10**. The paper also reports structured pruning on LLaDA: at pruning ratio **0.5**, the structured baseline gives **0.5898** PIQA, **0.5572** WinoG, **0.2039** ARC-C, while the sink-aware version gives **0.6037** PIQA, **0.5724** WinoG, **0.2362** ARC-C [2602.17664].

ASAP reports gains across image, video, and vision-language tasks. On ImageNet-1K, **DeiT-Base** with ASAP pooling reaches **81.68%** versus baseline **81.80%** with **+48.21% throughput**, while ASAP pruning reaches **81.33%** with **+48.45% throughput**. On Kinetics-400 with CLIP ViT-Large, ASAP pruning achieves **80.13%**, only \(-0.65\) drop, with **+24.5% throughput**, and ASAP pooling reaches **80.57%**, only \(-0.21\) drop, with **+23.7% throughput**. On LLaVA-1.5-7B, ASAP pooling at **128 tokens** gives **71.5 avg** = **98.5% relative**, and ASAP hybrid at **64 tokens** gives **70.2 avg** = **96.6% relative**; on POPE it matches or exceeds the full baseline with **86.2** at 128 tokens versus **85.9** for the full model [2605.22372].

In Video LLMs, SToP is most beneficial under aggressive retention. On fine-grained tasks at **10% retention**, **FastVid** drops by **15.69\%** and **FastVid + SToP** by **6.32\%**; **VisionZip** drops by **16.79\%** and **VisionZip + SToP** by **6.87\%**; **Holitom** drops by **6.22\%** and **Holitom + SToP** by **4.80\%**. On MCQA at **10% retention**, **VisionZip** drops by **7.36\%** and **VisionZip + SToP** by **3.34\%**; **FastVid** drops by **5.12\%** and **FastVid + SToP** by **2.90\%**; **Holitom** drops by **3.21\%** and **Holitom + SToP** by **2.02\%**. The evaluation explicitly includes EventHallusion, VideoComp, VCG-Bench, MVBench, VideoMME, NextQA, LongVideoBench, and MLVU [2604.20937].

Deep Forcing shows that sink-aware cache pruning can improve long-video generation without sacrificing real-time throughput. In the ablation table, **Self Forcing** has image quality **68.58**, **Self Forcing + Deep Sink** **68.54**, and **Self Forcing + Deep Sink + PC** **69.31**. Dynamic degree rises from **36.62** to **48.58** and then **57.56**. On a single H100 GPU, throughput is **15.78 FPS** for Self Forcing and **15.75 FPS** for Deep Forcing, which the paper describes as essentially unchanged [2512.05081].

## 5. Domain dependence, misconceptions, and limitations

The strongest general lesson is that sink saliency is not a universal indicator of importance. In causal LLMs, high BOS sink scores correlate with redundancy and are especially concentrated in deeper layers, so pruning the highest-scoring heads or blocks is often safe. The same paper reports that average BOS sink scores rise in later layers while the average performance impact of removing heads in those layers is low, which gives a structural explanation for why later blocks are often easier to prune [2601.06787].

In DLMs, however, the key limitation of AR-derived heuristics is temporal instability. The paper explicitly concludes that the “always keep sinks” heuristic is an AR-specific property, not a universal pruning rule. AR models show high spatial concentration but near-zero temporal variance, whereas DLMs exhibit much larger temporal variance, so preserving all sinks can waste capacity on positions that are not consistently important [2602.17664].

The GLM case makes the saliency-versus-utility distinction even sharper. Graph sink tokens are identified by activation magnitude, not by incoming attention mass, and the paper states that they are not necessarily the dominant attention targets of query tokens. In TEA-GLM, sink tokens are mostly at positions **0 and 1**, but the highest attention often goes to later graph tokens, especially positions **2–4**. In LLaGA, the top-2 graph sink tokens by activation are always \([PAD]\) tokens across all datasets and both tasks, while the center node at index 0 is never identified as a sink token. Pruning the top-2 sink tokens causes only a small drop, whereas random non-sink pruning can be more harmful. The paper’s interpretive lesson is explicit: activation-level saliency does not imply graph-semantic utility [2606.03712].

Video LLM results expose a different misconception: high-attention persistence across frames can be actively harmful under a limited token budget. The paper argues that if sink tokens survive pruning, they distort the model’s visual evidence and hinder fine-grained understanding, especially in hallucination evaluation and compositional reasoning. A related but distinct lesson appears in ASAP: local attention-based heuristics are vulnerable because sink tokens can appear important under one-layer metrics, so pruning should not be based on a token’s raw local attention alone [2604.20937] [2605.22372].

## 6. Adjacent developments and likely directions

Sink-aware pruning has already begun to blur into adjacent forms of sink-aware compression. "SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models" does not primarily evict tokens; instead, it detects when a KV group is likely to be in the sink regime and skips historical KV-cache loading, using a zero surrogate because sink-dominant heads are empirically low-impact. Its core thresholded routing rule is built around the sink condition \(\alpha_0 > \gamma\) with \(\gamma = 0.65\), a cosine-similarity proxy, and a length-dependent threshold calibrated to about a **60% skip ratio**; the paper reports a maximum speedup of **2.03× at 512K context** [2604.16883].

Several papers also suggest that sink-aware methods will remain paradigm-dependent rather than converging to a single universal rule. The GLM study argues that future pruning methods should consider structural role in the graph-token construction, semantic contribution to prediction, attention behavior, and downstream sensitivity rather than activation magnitude alone. The Video LLM study similarly shows that MCQA alone can hide pruning failures, implying that evaluation design affects what sink-aware pruning is optimized to preserve. A plausible implication is that future work will increasingly combine sink detection with structural grouping, routing, and calibration rather than treating pruning as a purely magnitude-based selection problem [2606.03712] [2604.20937].

Taken together, recent work supports a narrow but robust conclusion: sink-aware pruning is not a single algorithm but a mechanistic design pattern. Its technical content lies in identifying what kind of sink a model exhibits, determining whether that sink signals redundancy, instability, or useful geometric structure, and then incorporating that diagnosis into pruning, pooling, or cache-selection rules. Where sink behavior marks functional redundancy, it supports aggressive compression; where it marks a stable attractor, it can serve as an anchor for globally informed reduction; and where it is merely an activation-salient artifact, it should not be conflated with semantic importance.

Source: https://www.emergentmind.com/topics/sink-aware-pruning