Sink-Aware Pruning for Efficient Model Compression

• Sink-Aware Pruning is a set of compression techniques that identify inattention-absorbing tokens or activations (sinks) to differentiate redundant parts from structurally important components.
• It spans transformer, diffusion, vision, video, and graph models, using metrics like BOS sink scores, diffusion distances, and activation thresholds to inform pruning decisions.
• Empirical evaluations show that sink-aware methods maintain performance at various pruning ratios, ensuring efficient model compression across diverse applications.

Sink-aware pruning denotes a class of compression methods that uses sink behavior as an explicit pruning signal. In recent work, the underlying sink can be a BOS-directed attention head in autoregressive LLMs, a transient high-mass position in a diffusion LLM, a dominant accumulator of probability mass in a Vision Transformer, a spatially persistent high-attention visual token in a Video LLM, an activation-level outlier graph token in a graph LLM, or a preserved sink region in long-video KV-cache management. The shared premise is that sink-like behavior is mechanistically structured, but its functional meaning is model- and modality-dependent: in some settings it marks redundancy that is safe to remove, whereas in others it is more useful as an anchor for grouping, routing, or cache preservation than as an object of direct deletion (Sok et al., 11 Jan 2026, Myrzakhan et al., 19 Feb 2026, Lee et al., 21 May 2026, Kim et al., 22 Apr 2026, Zhang et al., 2 Jun 2026, Yi et al., 4 Dec 2025).

1. Mechanistic basis of sink behavior

The concept builds on the broader attention-sink phenomenon: some components allocate or attract a disproportionate amount of attention even when they are not semantically important. In causal LLMs, the canonical case is the BOS token at position $k=0$. A head that is not needed for meaningful routing can still absorb leftover probability mass because softmax forces each query to distribute total attention mass $1$; such heads can therefore act as attention dumping grounds. In the formulation of "Garbage Attention in LLMs: BOS Sink Heads and Sink-aware Pruning," high-BOS sink heads are functionally redundant, especially in deeper layers, where other routes for information integration already exist (Sok et al., 11 Jan 2026).

This basic picture does not transfer unchanged across architectures. "Sink-Aware Pruning for Diffusion LLMs" argues that the AR heuristic of treating sinks as stable global anchors does not hold for DLMs, because sink locations shift across denoising timesteps and are often transient rather than structurally essential (Myrzakhan et al., 19 Feb 2026). In Vision Transformers, "ASAP: Attention Sink Anchored Pruning" treats the sink as a structurally predictable attractor in a Lazy Random Walk rather than as pathological noise; sink-aware compression then uses the sink as a geometric anchor for separating redundant background tokens from semantically important foreground tokens (Lee et al., 21 May 2026). In Video LLMs, "Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs" identifies sink tokens as semantically uninformative tokens that attract excessive attention and distort visual evidence under aggressive token budgets (Kim et al., 22 Apr 2026).

A central misconception addressed across these works is that high attention or large internal activation automatically implies semantic importance. The graph-language-model study "When Graph Tokens Sink" makes this point particularly explicitly: graph sink tokens are activation-salient, but they are not necessarily the dominant attention targets of query tokens and are not the primary semantic or structural carriers for downstream prediction (Zhang et al., 2 Jun 2026).

2. Formal definitions and scoring rules

In the BOS-based LLM setting, attention weights are written as $\alpha_{t,k}^{(\ell,h)}$, where $t$ is the query position, $k$ the key position, $\ell$ the layer, $h$ the head, and $T$ the sequence length. The sink score of token position $k$ is the average attention received from all queries,

$$\operatorname{sink\text{-}score}_k^{(\ell,h)} = \frac{1}{T}\sum_{t=0}^{T-1} \alpha_{t,k}^{(\ell,h)}.$$

For BOS, $1$0, so the head-wise BOS sink score is

$1$1

and the layer-wise BOS sink score is

$1$2

This score is computed by extracting attention maps during inference and averaging the BOS attention mass across query positions (Sok et al., 11 Jan 2026).

For DLMs, sinkness is defined over the generation trajectory. With attention matrix $1$3 at generation step $1$4, the incoming attention mass to token position $1$5 is

$1$6

The trajectory-averaged attention mass is

$1$7

with spatial variance

$1$8

To capture sink migration, the paper defines a sink-set centroid

$1$9

and temporal variance

$\alpha_{t,k}^{(\ell,h)}$0

A token $\alpha_{t,k}^{(\ell,h)}$1 is treated as a sink at step $\alpha_{t,k}^{(\ell,h)}$2 if

$\alpha_{t,k}^{(\ell,h)}$3

with a sigmoid relaxation

$\alpha_{t,k}^{(\ell,h)}$4

an averaged sink estimate

$\alpha_{t,k}^{(\ell,h)}$5

and a suppression weight

$\alpha_{t,k}^{(\ell,h)}$6

The formal objective is not to preserve all sinks, but to identify unstable sink candidates and discount them in pruning importance (Myrzakhan et al., 19 Feb 2026).

In ASAP, each attention matrix is interpreted as a Markov transition matrix. For layer $\alpha_{t,k}^{(\ell,h)}$7, with head-averaged attention $\alpha_{t,k}^{(\ell,h)}$8, the residual connection is incorporated through a Lazy Random Walk,

$\alpha_{t,k}^{(\ell,h)}$9

and the cumulative information flow up to layer $t$0 is

$t$1

Sink emergence is detected by

$t$2

and the pruning signal is the diffusion distance to the sink,

$t$3

Here, small $t$4 indicates sink-proximal, likely redundant background; large $t$5 indicates a distinct trajectory and likely foreground semantics (Lee et al., 21 May 2026).

In Video LLMs, the sink score is defined from attention persistence across frames: $t$6 A token with consistently high attention across many frames receives a larger sink score, and the exponent $t$7 sharpens the sink-score distribution (Kim et al., 22 Apr 2026).

The GLM formulation is different again. A graph token is treated as a sink token if its hidden state has a sufficiently large RMS-normalized activation on at least one sink dimension: $t$8 with

$t$9

The main experiments use $k$0 for both LLaGA and TEA-GLM. This definition is activation-level rather than attention-level, which is precisely why the paper distinguishes graph sink tokens from classical attention sinks (Zhang et al., 2 Jun 2026).

3. Algorithmic forms of sink-aware pruning

In BOS-based head and block pruning, the procedure is explicitly structured. Head-wise pruning runs a calibration or evaluation pass, extracts attention maps, computes $k$1 for every head, ranks heads by descending BOS sink score, and prunes the highest-scoring heads by zeroing their output-projection slice,

$k$2

Layer-wise pruning averages head scores into $k$3, ranks layers by descending score, prunes the highest-scoring transformer blocks, and excludes the first and last layers for stability (Sok et al., 11 Jan 2026).

In DLMs, sink-aware pruning is not introduced as a new backend but as a wrapper around existing post-training methods. Layer input activations $k$4 are sink-masked by

$k$5

For Wanda, the baseline score

$k$6

becomes

$k$7

For SparseGPT, the Hessian approximation is recomputed from sink-masked inputs,

$k$8

The method uses WikiText-2, 128 randomly sampled sequences, each truncated to 2048 tokens, to estimate sink statistics and create pruning masks offline (Myrzakhan et al., 19 Feb 2026).

ASAP performs a single-shot token partitioning rather than iterative local pruning. After identifying the sink $k$9 and computing diffusion distances, tokens are assigned to radial semantic levels by

$\ell$0

and the sink-proximal background cluster is compressed with Transition Weight Pooling,

$\ell$1

The core pipeline detects the layer where the sink emerges, computes a cumulative transition matrix, measures token-to-sink diffusion distance, clusters tokens radially, compresses the background cluster, and optionally applies a plug-in pruning stage for extreme compression (Lee et al., 21 May 2026).

SToP modifies both spatial and temporal pruning in Video LLMs. In sink-aware spatial pruning, raw attention is replaced by a sink-penalized score,

$\ell$2

In sink-aware temporal pruning, the pruning set becomes

$\ell$3

The method is plug-and-play: STSP is applied to VisionZip and FastVid, while Holitom uses both STTP and STSP (Kim et al., 22 Apr 2026).

Deep Forcing presents a sink-aware KV-cache variant for autoregressive video diffusion. Participative Compression divides the cache into Sink, Recent, and Candidate regions, scores candidate keys by recent-query participation,

$\ell$4

selects the top-$\ell$5 candidate tokens, and constructs

$\ell$6

This is described as sink-aware and importance-aware KV-cache pruning rather than token pruning in the ViT or LLM sense (Yi et al., 4 Dec 2025).

4. Empirical results across domains

In autoregressive LLMs, BOS-based pruning preserves downstream performance better than weight- or activation-based criteria. On Gemma-3-4B at 12.5% pruning, the dense average is 0.648, BOS pruning 0.641, Wanda-SP 0.594, and Mag-SP 0.543. On Qwen3-4B at 12.5%, the dense average is 0.649, BOS pruning 0.643, Wanda-SP 0.618, and Mag-SP 0.502. On Llama-3.1-8B at 25.0%, the dense average is 0.673, BOS pruning 0.631, Wanda-SP 0.572, and Mag-SP 0.403. Layer-level pruning also performs strongly: on Gemma-3-4B at 12.5%, BOS averages 0.589 versus ShortGPT 0.456 and Top-Down 0.439; on Qwen3-4B at 12.5%, BOS averages 0.570 versus ShortGPT 0.544 and Top-Down 0.541. These experiments use Gemma-3-4B, Llama-3.1-8B, and Qwen3-4B, a maximum sequence length of 4096 tokens, and lm-evaluation-harness with WikiText-2, ARC-Easy, BoolQ, OpenbookQA, PIQA, Winogrande, ARC-Challenge, HellaSwag, and MMLU (Sok et al., 11 Jan 2026).

In DLMs, sink-aware masking improves the quality-efficiency trade-off under matched compute. On LLaDA at 50% sparsity, Wanda averages 52.70 and Sink-aware Wanda 53.18, while SparseGPT averages 52.34 and Sink-aware SparseGPT 52.36. On Dream at 50% sparsity, SparseGPT improves from 54.40 to 54.58 with the sink-aware variant. On LLaDA-1.5 at 50% sparsity, Sink-aware Wanda improves from 53.78 to 54.02, and Sink-aware SparseGPT from 53.91 to 54.10. The paper also reports structured pruning on LLaDA: at pruning ratio 0.5, the structured baseline gives 0.5898 PIQA, 0.5572 WinoG, 0.2039 ARC-C, while the sink-aware version gives 0.6037 PIQA, 0.5724 WinoG, 0.2362 ARC-C (Myrzakhan et al., 19 Feb 2026).

ASAP reports gains across image, video, and vision-language tasks. On ImageNet-1K, DeiT-Base with ASAP pooling reaches 81.68% versus baseline 81.80% with +48.21% throughput, while ASAP pruning reaches 81.33% with +48.45% throughput. On Kinetics-400 with CLIP ViT-Large, ASAP pruning achieves 80.13%, only $\ell$7 drop, with +24.5% throughput, and ASAP pooling reaches 80.57%, only $\ell$8 drop, with +23.7% throughput. On LLaVA-1.5-7B, ASAP pooling at 128 tokens gives 71.5 avg = 98.5% relative, and ASAP hybrid at 64 tokens gives 70.2 avg = 96.6% relative; on POPE it matches or exceeds the full baseline with 86.2 at 128 tokens versus 85.9 for the full model (Lee et al., 21 May 2026).

In Video LLMs, SToP is most beneficial under aggressive retention. On fine-grained tasks at 10% retention, FastVid drops by 15.69\% and FastVid + SToP by 6.32\%; VisionZip drops by 16.79\% and VisionZip + SToP by 6.87\%; Holitom drops by 6.22\% and Holitom + SToP by 4.80\%. On MCQA at 10% retention, VisionZip drops by 7.36\% and VisionZip + SToP by 3.34\%; FastVid drops by 5.12\% and FastVid + SToP by 2.90\%; Holitom drops by 3.21\% and Holitom + SToP by 2.02\%. The evaluation explicitly includes EventHallusion, VideoComp, VCG-Bench, MVBench, VideoMME, NextQA, LongVideoBench, and MLVU (Kim et al., 22 Apr 2026).

Deep Forcing shows that sink-aware cache pruning can improve long-video generation without sacrificing real-time throughput. In the ablation table, Self Forcing has image quality 68.58, Self Forcing + Deep Sink 68.54, and Self Forcing + Deep Sink + PC 69.31. Dynamic degree rises from 36.62 to 48.58 and then 57.56. On a single H100 GPU, throughput is 15.78 FPS for Self Forcing and 15.75 FPS for Deep Forcing, which the paper describes as essentially unchanged (Yi et al., 4 Dec 2025).

5. Domain dependence, misconceptions, and limitations

The strongest general lesson is that sink saliency is not a universal indicator of importance. In causal LLMs, high BOS sink scores correlate with redundancy and are especially concentrated in deeper layers, so pruning the highest-scoring heads or blocks is often safe. The same paper reports that average BOS sink scores rise in later layers while the average performance impact of removing heads in those layers is low, which gives a structural explanation for why later blocks are often easier to prune (Sok et al., 11 Jan 2026).

In DLMs, however, the key limitation of AR-derived heuristics is temporal instability. The paper explicitly concludes that the “always keep sinks” heuristic is an AR-specific property, not a universal pruning rule. AR models show high spatial concentration but near-zero temporal variance, whereas DLMs exhibit much larger temporal variance, so preserving all sinks can waste capacity on positions that are not consistently important (Myrzakhan et al., 19 Feb 2026).

The GLM case makes the saliency-versus-utility distinction even sharper. Graph sink tokens are identified by activation magnitude, not by incoming attention mass, and the paper states that they are not necessarily the dominant attention targets of query tokens. In TEA-GLM, sink tokens are mostly at positions 0 and 1, but the highest attention often goes to later graph tokens, especially positions 2–4. In LLaGA, the top-2 graph sink tokens by activation are always $\ell$9 tokens across all datasets and both tasks, while the center node at index 0 is never identified as a sink token. Pruning the top-2 sink tokens causes only a small drop, whereas random non-sink pruning can be more harmful. The paper’s interpretive lesson is explicit: activation-level saliency does not imply graph-semantic utility (Zhang et al., 2 Jun 2026).

Video LLM results expose a different misconception: high-attention persistence across frames can be actively harmful under a limited token budget. The paper argues that if sink tokens survive pruning, they distort the model’s visual evidence and hinder fine-grained understanding, especially in hallucination evaluation and compositional reasoning. A related but distinct lesson appears in ASAP: local attention-based heuristics are vulnerable because sink tokens can appear important under one-layer metrics, so pruning should not be based on a token’s raw local attention alone (Kim et al., 22 Apr 2026, Lee et al., 21 May 2026).

6. Adjacent developments and likely directions

Sink-aware pruning has already begun to blur into adjacent forms of sink-aware compression. "SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models" does not primarily evict tokens; instead, it detects when a KV group is likely to be in the sink regime and skips historical KV-cache loading, using a zero surrogate because sink-dominant heads are empirically low-impact. Its core thresholded routing rule is built around the sink condition $h$0 with $h$1, a cosine-similarity proxy, and a length-dependent threshold calibrated to about a 60% skip ratio; the paper reports a maximum speedup of 2.03× at 512K context (Liu et al., 18 Apr 2026).

Several papers also suggest that sink-aware methods will remain paradigm-dependent rather than converging to a single universal rule. The GLM study argues that future pruning methods should consider structural role in the graph-token construction, semantic contribution to prediction, attention behavior, and downstream sensitivity rather than activation magnitude alone. The Video LLM study similarly shows that MCQA alone can hide pruning failures, implying that evaluation design affects what sink-aware pruning is optimized to preserve. A plausible implication is that future work will increasingly combine sink detection with structural grouping, routing, and calibration rather than treating pruning as a purely magnitude-based selection problem (Zhang et al., 2 Jun 2026, Kim et al., 22 Apr 2026).

Taken together, recent work supports a narrow but robust conclusion: sink-aware pruning is not a single algorithm but a mechanistic design pattern. Its technical content lies in identifying what kind of sink a model exhibits, determining whether that sink signals redundancy, instability, or useful geometric structure, and then incorporating that diagnosis into pruning, pooling, or cache-selection rules. Where sink behavior marks functional redundancy, it supports aggressive compression; where it marks a stable attractor, it can serve as an anchor for globally informed reduction; and where it is merely an activation-salient artifact, it should not be conflated with semantic importance.