Global Attention Sinks in Transformers

• Global Attention Sinks (GAS) are tokens or positions that attract extreme cumulative attention, defined when their attention mass far exceeds the mean across tokens.
• They manifest as first-position tokens, [CLS]/[SEP] anchors, stop words, or similar elements across models, influencing computational routing and interpretability.
• Recent studies show GAS can serve distinct roles—such as adaptive nop, broadcast, or grouping mechanisms—highlighting both mitigation challenges and design opportunities.

Searching arXiv for papers on attention sinks and Global Attention Sinks. Global Attention Sinks (GAS) are tokens or positions that attract disproportionately large attention from many queries across layers and heads, often with limited direct semantic relevance. The exact acronym appears in some recent work, while many other papers use the broader term “attention sink” for the same family of effects. Across current transformer research, GAS encompass first-position or BOS sinks in decoder-only LLMs, [CLS]/[SEP]-like anchors, EOS and language-tag sinks in multilingual NMT, stop-word and register sinks in diffusion and vision models, and visual or multimodal sink tokens that act as scene-level priors. Recent work increasingly treats GAS not as a single anomaly but as a recurrent structural pattern with mechanistic, algorithmic, and systems-level consequences (Su et al., 11 Apr 2026, Li et al., 7 May 2026, Kukleva et al., 26 Sep 2025).

1. Definitions and taxonomy

A standard formalization treats a sink as a token whose cumulative received attention is an extreme outlier. In the survey literature, for an attention matrix $A \in \mathbb{R}^{L \times L}$, cumulative attention to token $j$ is

$\hat A_j = \sum_{i=1}^L A_{ij},$

with mean cumulative attention

$\mu_A = \frac{1}{L}\sum_{k=1}^L \hat A_k,$

and a token is classified as an attention sink when $\hat A_j > \tau \mu_A$ for a large threshold $\tau$ (Su et al., 11 Apr 2026). In diffusion transformers for zero-shot referral segmentation, a token is identified as GAS when its average incoming text-to-text attention mass exceeds $10\times$ the global mean across all layers and tokens; those GAS are absent in early layers, appear in later blocks, and are typically stable in deeper blocks (Kukleva et al., 26 Sep 2025).

Several papers further separate sink types. “Primary” sinks are the classic BOS-like sinks that emerge early and persist through the remaining network, while “secondary” sinks arise mainly in middle layers, attract a smaller but still significant amount of attention mass, and persist for a variable number of layers (Wong et al., 22 Dec 2025). In this depth-aware view, GAS are not limited to a single always-on BOS phenomenon; they can also form a hierarchy of sink levels indexed by emergence layer and lifetime.

A useful operational distinction is between “global” and merely “strong” sinks. A GAS is typically fixed or recurrent across inputs, visible to many or all later positions, and influential across many heads or layers. This is why BOS, [CLS], EOS, register tokens, and certain punctuation or stop-word tokens are repeatedly singled out in the literature: they are not only high-attention targets, but reusable anchors of model computation (Su et al., 11 Apr 2026).

Setting	Typical GAS	Representative signature
Decoder-only LLMs	BOS / first token, later hidden sinks	persistent first-column dominance
Diffusion transformers	stop words, </s>, occasional color tokens	$m_j > 10\mu$ criterion
Multilingual NMT	</s>, language tags, punctuation	80–91% mass on non-content tokens

2. Structural and mechanistic accounts

One mechanistic line of work traces GAS to the value aggregation structure of causal self-attention. In decoder-only transformers, for

$$O_{i,k} = \sum_{j=0}^{i} A_{i,j} V_{j,k}, \qquad \sum_{j=0}^{i} A_{i,j}=1,$$

token $0$ is exempt from averaging, whereas later positions aggregate over progressively larger prefixes. The result is a positional variance discrepancy: token $j$0 remains a high-variance outlier, output projection preserves that discrepancy, FFN “super neurons” selectively amplify it, sparse down-projection concentrates energy into a few coordinates, and RMSNorm plus $j$1 projections lock the representation into a direction that later heads strongly favor (Li et al., 7 May 2026). The paper quantifies the resulting concentration by the dominance ratio

$j$2

and shows that this rises sharply for the first token in early layers.

A distinct GPT-2 account identifies a different circuit: a learned query bias $j$3, the first-layer MLP transformation of the positional encoding, and structure in the key projection $j$4. In that setting, a source-agnostic term

$j$5

gives position $j$6 a broad score advantage, and the first-layer MLP turns the first positional embedding into an Effective Positional Encoding with a few massive coordinates that are specifically amplified by $j$7 (Ran-Milo et al., 16 Apr 2026). The paper also emphasizes that each component of this circuit is individually dispensable across architectures, which indicates that attention sinks can arise through distinct circuits rather than one universal implementation.

A geometric account recasts sinks as reference-frame tokens. On that view, centralized, distributed, and bidirectional reference frames correspond to different sink organizations: a single BOS-like anchor, multiple syntactic anchors, or start/end boundary anchors, respectively (Ruscio et al., 4 Aug 2025). This suggests that GAS can be understood as coordinate-system devices in representation space, not merely as optimization accidents.

A complementary anatomy of spikes and sinks argues that massive activations and attention sinks often co-occur because pre-norm residual dynamics let a few channels become global implicit parameters while attention heads use those global directions locally as sinks (Sun et al., 5 Mar 2026). Taken together, these mechanistic accounts suggest that GAS are structurally favored whenever an architecture combines asymmetric visibility, a normalized competition over keys, and learned pathways that create privileged outlier directions.

3. Functional interpretations

Recent work argues that visually similar sink patterns can implement different algorithms. One unifying account distinguishes two cases. In adaptive nop, a head routes attention to a sink whose value norm is negligible, so the head effectively suppresses its update. In broadcast, a sink aggregates and redistributes global information, yielding a low-rank output that adds a shared component to many tokens (Fesser et al., 6 Jun 2026). This directly challenges the common assumption that all GAS are either useless or uniformly beneficial.

A separate line describes attention sinks and outlier features as a “catch, tag, and release” mechanism. In this picture, a sink catches a subset of tokens, tags them by broadcasting a shared outlier feature into their representations, and releases them into the residual stream so later layers can retrieve or manipulate that tagged subset (Zhang et al., 2 Feb 2025). The paper proves that even simple tasks such as averaging can naturally give rise to this motif under low-rank parameterizations, which suggests that GAS may support grouping, segmentation, and retrieval computations rather than merely absorbing surplus softmax mass.

Another theoretical treatment identifies sinks and diagonal self-attention patterns as alternative ways to realize an attention switch and to prevent oversmoothing. Under the paper’s assumptions, a hard attention switch—where the head output is identically zero for a subset of tokens—is equivalent to a sink with zero BOS value, while diagonal patterns implement a softer switch that preserves self-communication (Súkeník et al., 8 May 2026). The same work also specifies conditions under which denser attention provably smooths more than sparser attention and reports that those conditions are often satisfied in practice. This suggests that GAS can function as anti-oversmoothing devices by limiting inter-token mixing.

In multimodal vision-LLMs, visual sinks have also been interpreted as global priors. V-sinks from the vision encoder and L-sinks that emerge inside the LLM preserve scene-level information such as count and size better than ordinary tokens, yet their dominance can suppress the fine-grained evidence required for local perception (Choi et al., 1 Apr 2026). This reinforces a broader point: GAS can be both a routing convenience and an information bottleneck.

4. Empirical manifestations across domains

In decoder-only LLMs, first-position sink behavior is highly robust. StreamingLLM showed that on sequences of length 4096, the 4096-th token’s attention to the first token often exceeds 50% of total attention in many layers except the first two, and that preserving a few initial tokens is enough to recover long-stream performance (Xiao et al., 2023). Structural work on LLaMA-2-7B and LLaMA-3-8B further reports that the first token’s attention share is small in early layers but spikes sharply around layer 2, with the spike invariant across inputs and datasets, including random-token sequences (Li et al., 7 May 2026).

Sink phenomena also extend beyond BOS. In KV-cache quantization, tokens 0 and 14 can act as sinks in LLaMA2-7B, and the sink-induced term

$j$8

is described as an attention bias that is nearly the same for all tokens $j$9 within a head (Su et al., 6 Aug 2025). In the same work, stable activation outliers at sink tokens persist through a broad band of intermediate layers, and the layers where emergence and dissipation occur are almost fixed per model.

In diffusion transformers used for referring image and video segmentation, GAS appear in late text blocks rather than at the start of processing. The paper reports that later layers typically contain 1–3 GAS per sequence, often stop words such as "_a" or the end-of-sequence token "</s>", and that 77% of GAS tokens correspond to stop words, with about 10% color tokens and about 10% other content words (Kukleva et al., 26 Sep 2025). These GAS allocate disproportionately high and nearly uniform attention across text and image tokens and are treated as noise-like for grounding.

In multilingual NMT cross-attention, the phenomenon changes form but remains global. In NLLB-200 (600M), non-content tokens dominated by </s> absorb 83% to 91% of total cross-attention mass, with </s> alone absorbing 78% to 87%; raw teacher-forcing similarity of 36.7% rises to 70.7% after filtering those sink tokens, which shows how severely sinks can distort interpretability metrics (Mutisya et al., 2 May 2026).

In multimodal speech recognition, the BOS sink coexists with intermediate low-semantic sinks such as <audio>, </audio>, <video>, and prompt tokens. These intermediate sinks emerge after about layer 2, share massive activation indices with BOS, and become highly cosine-aligned with BOS in mid-layers; a decorrelation loss then reduces those intermediate sinks and improves WER under strong compression (Anand et al., 26 Oct 2025).

In large vision-LLMs, V-sinks and L-sinks are quantitatively sparse but influential. In LLaVA-1.5 with 576 visual tokens, the paper reports about 2.6 V-sinks per image and about 7 L-sinks per layer, while ordinary tokens account for roughly 98.4% of visual positions (Choi et al., 1 Apr 2026). This suggests that GAS-like behavior can be a low-cardinality, high-impact organizing principle even in heavily multimodal settings.

5. Control, mitigation, and exploitation

The most direct exploitation of GAS appears in long-context inference. StreamingLLM preserves a small fixed set of initial sink tokens together with a rolling recent window, enabling stable language modeling up to 4 million tokens and yielding up to 22.2x speedup over sliding-window recomputation in streaming settings (Xiao et al., 2023). In the same paper, pretraining with a dedicated sink token allows a single learned sink to stabilize streaming behavior.

Architectural mitigation can instead target the upstream statistics that create sinks. “Head-wise RMSNorm,” inserted immediately after value aggregation and before $\hat A_j = \sum_{i=1}^L A_{ij},$0, is designed to restore variance parity across positions and heads. In a 152M model, the head-normalized system improved training loss from 2.748 to 2.707, validation loss from 2.781 to 2.742, layer-wise average effective rank from 344 to 446, and reduced layer-wise mean dimension disparity from 82.7 to 33.7 (Li et al., 7 May 2026). This is a direct attempt to break the variance-discrepancy pathway that forms BOS-like GAS.

In KV-cache quantization, the operative question is not how to remove sinks but how to preserve the right ones. KVSink predicts sink tokens from stable outlier channels at a known emergence layer and keeps only those tokens in high precision, which the paper describes as a plug-and-play method with negligible overhead that outperforms static Preserve-First-N strategies (Su et al., 6 Aug 2025).

OASIS takes a different route by introducing a Softmax1-based null space at token and depth levels and coupling token-level null evidence to depth routing through an inter-layer null signal. The abstract reports average reduction of 9.26% in maximum infinity norm and 2.60% in average kurtosis across evaluated settings, while lowering perplexity by 75.85% under W8A8 and improving GSM8K Pass@1 by 12.42% under W4A4 (Luo et al., 18 May 2026). This mitigation is especially notable because it reframes sink behavior as a consequence of softmax normalization without an explicit null option.

Other interventions exploit GAS rather than suppressing them wholesale. RefAM appends attention magnets—stop words and an auxiliary color word—and filters them out after attention aggregation, effectively redirecting late-layer GAS and background attention away from content tokens (Kukleva et al., 26 Sep 2025). Layer-wise Sink Gating in LVLMs scales the attention contributions of V-sinks versus the rest of the visual tokens; in most layers it improves multimodal benchmarks by balancing global scene priors against local evidence (Choi et al., 1 Apr 2026). These approaches imply that practical control of GAS is often selective: preserve when they act as useful anchors, suppress when they overshadow fine-grained content.

6. Open problems and broader significance

Current work converges on two points. First, GAS are ubiquitous across model families and modalities. Second, the same qualitative pattern can arise from distinct circuits and serve distinct computational roles. The survey literature therefore treats sink research as spanning utilization, mechanistic interpretation, and mitigation rather than a single pathology class (Su et al., 11 Apr 2026).

Several open questions remain unresolved. One concerns architectural universality: GPT-2 identifies a concrete sink circuit involving query bias, first-layer MLP-transformed positional encoding, and key structure, yet the same paper stresses that each of those ingredients is individually dispensable across architectures (Ran-Milo et al., 16 Apr 2026). Another concerns depth structure: large reasoning-oriented models appear to organize sinks into discrete levels, with three levels reported in QwQ-32B and six in Qwen3-14B (Wong et al., 22 Dec 2025). This suggests that GAS may evolve from a single BOS effect into a staged routing system as models scale and specialize.

A further open question is whether a given GAS is implementing nop, broadcast, or some mixed behavior. The unifying view that combines value-norm diagnostics with low-rank output diagnostics implies that effective intervention should begin by asking what the model is computing, not merely where its attention concentrates (Fesser et al., 6 Jun 2026). A plausible implication is that future sink-aware architectures will need explicit mechanisms for both abstention and global workspace formation, rather than a one-size-fits-all sink suppression rule.

The geometric literature adds another unresolved dimension. If sinks are manifestations of reference frames that anchor representational space, then eliminating them indiscriminately may remove coordinate systems that the model relies on for stable computation (Ruscio et al., 4 Aug 2025). This suggests that the most productive direction is not the blanket removal of GAS, but the design of explicit, interpretable, and controllable global anchors that separate beneficial global routing from pathological sink dominance.

Global Attention Sinks have therefore become a central concept for understanding how transformers allocate attention when content alone does not determine routing. They are at once an interpretability challenge, a systems consideration for streaming and quantization, a mechanistic clue about residual-space geometry and FFN amplification, and a design variable for future architectures.