---
title: Stable Sink Token in Transformers
url: https://www.emergentmind.com/topics/stable-sink-token
type: topic
---

# Stable Sink Token in Transformers

A stable sink token is a persistent, structurally-induced attractor in the attention computations of transformer networks, characterized by receiving an outsized, layer-wise attention allocation from a substantial subset of query positions—often irrespective of semantic content. In autoregressive language models, this role is almost always played by the first token (commonly a [BOS], <s>, or similar initialization symbol), but the phenomenon generalizes to other architectural variants, modalities, and pretraining regimes. The stability of the sink token is fundamental to information flow, optimization, and the computational geometry of deep transformer networks.

## 1. Mathematical Definition and Geometric Properties

Let $L$ denote the number of layers, and $N$ the context or sequence length in a transformer. At layer $l$ ($1 \leq l \leq L$), the (pre-attention) hidden state for token $i$ is $h_i^l \in \mathbb{R}^d$, with $i = 0$ reserved as the canonical sink token (typically the first position). The unit-normalized hidden state is defined as $\hat{h}_i^l = h_i^l/\Vert h_i^l \Vert_2$.

The canonical indicator of the sink token's stability is the layer-wise drift:
$$
\Delta_s^l = \Vert \hat{h}_s^{l+1} - \hat{h}_s^l \Vert_2 \ll 1
$$
Empirical studies across Llama-2, Mistral, Llama-3, and related architectures demonstrate that for all $l > l_{\text{sink}}$ (the onset of the sink phenomenon), $\hat{h}_s^l$ remains essentially constant, with $\cos(\hat{h}_s^{l_1}, \hat{h}_s^{l_2}) \approx 0.95$–$0.99$ even among distant layers [2507.03865].

The cosine similarity between any other token and the sink,
$$
\cos(\hat{h}_i^l, \hat{h}_s^l) = (\hat{h}_i^l)^\top \hat{h}_s^l,
$$
increases monotonically with depth: tokens are geometrically "attracted" to the static direction defined by the sink, converging toward its point on the representation hypersphere.

This geometric attractor property is robust: token trajectories in the normalized hidden-state space point toward the fixed sink direction, forming a reference axis that fundamentally organizes the coordinate system of deep transformer representations [2507.03865, 2508.02546].

## 2. Mechanistic Origins and Architectural Determinants

The emergence and stability of sink tokens arise from several mechanistic and architectural sources:

- **Positional encoding and masking:** In standard RoPE (rotary positional encoding), the first token (index 0) is always unrotated, conferring a unique, unperturbed reference in the key space [2508.02546]. Absolute positional encodings used in encoders (e.g., BERT, RoBERTa) can induce dual (bidirectional) stable sinks at both ends [2508.02546].

- **LayerNorm, residual connections, and feature pinning:** The first token’s features, propagated through residual pathways and LayerNorm/RMSNorm, tend to maintain large and stable magnitudes. These are insufficiently affected by stochastic or input-dependent noise, producing a hidden state with persistent directionality across depth [2507.03865].

- **Softmax normalization constraint:** The probability simplex enforced by softmax compresses attention such that any residual or non-informative "budget" is persistently absorbed by the reference (sink) key, acting as a mathematically-optimal key-bias to soak up unused mass [2410.10781].

- **Training and data distribution:** Sinks meed sufficient data and optimization before stabilizing. Their strength is modulated by learning rate, weight decay, and context packing, but they almost always arise in large-scale LM pretraining [2410.10781, 2504.02732].

This structural origin is not idiosyncratic to decoder-only LMs or NLP—variants with different position encoding schemes (e.g., scaled RoPE in Qwen/Phi-2) yield distributed reference frames and hence distributed (multi-token) sinks [2508.02546]. In vision, improper token mixing (e.g., in ViTs) results in the [CLS] token becoming a sink unless carefully architected [2504.06738].

## 3. Functional Role in Information Propagation and Model Stability

Stable sink tokens implement a regularizing constraint on sequence-wide mixing:

- **Avoidance of rank and representational collapse:** Without constrained mixing, repeated attention across layers drives the hidden-state matrix toward low-rank (or even rank-1) subspaces, resulting in over-mixing and representational collapse for late-position tokens [2504.02732]. The stable sink, by absorbing a large fraction of attention, attenuates this process by "turning off" mixing in a subset of heads/layers—a defense against over-squashing [2504.02732].

- **No-op update zones:** Tokens highly aligned with the sink receive minimal further updates, effectively skipping full computation while preserving residual contributions, as exploited in token selection and dynamic sparsification [2507.03865].

- **Reference-frame anchoring:** The sink provides a geometric reference axis for self-attention alignment, enabling transformers to order and coordinate information in high-dimensional space [2508.02546].

These roles generalize to different modalities and architectures. In diffusion language models (DLMs), the instability of moving sinks (due to dynamic masking) disrupts robustness, motivating explicit insertion of a static sink token to regularize the residual stream [2601.19657].

## 4. Practical Exploitation: Efficient Inference, Sparsity, and Compression

Recent methods leverage the stable sink to optimize inference and resource utilization:

- **OrthoRank dynamic selection:** By measuring token importance as orthogonality to the sink
$$
w_i^l = 1 - \cos(\hat{h}_i^l, \hat{h}_s^l),
$$
and updating only the most orthogonal tokens each layer (those farthest from the sink on the hypersphere), OrthoRank achieves lower perplexity and improved zero-shot accuracy at the same or lower computational cost versus standard layer-pruning approaches [2507.03865]. Importantly, this method yields superior performance on long-context tasks such as LongBench.

- **KV cache quantization and storage:** The key-bias nature of the sink token underlies strategies for selective precision preservation. KVSink, for instance, identifies tokens with stable outlier activations in a designated layer/channel and preserves their key/value representations at full precision during quantization, attaining substantial improvements over naive first-N preservation strategies [2508.04257].

- **Token selection in structured sequences:** In click-through rate prediction, CTR-Sink strategically inserts trainable sink tokens between behaviors, carrying external correlation signals. These guide attention to meaningful boundaries, mitigate semantic fragmentation, and systematically improve empirical performance [2508.03668].

- **Video diffusion and long-context sequence models:** Deep Sink, in autoregressive video diffusion, dedicates half of the sliding attention window to persistent sink tokens, using temporal RoPE alignment to anchor global context and prevent drift during out-of-distribution, long-range generation [2512.05081].

## 5. Deviations, Modifications, and Countermeasures

The presence of stable sink tokens is not inevitable; specific modifications disrupt or entirely eliminate the phenomenon:

- **Alternative normalization:** Replacing softmax with non-sum-to-one or rectified-normalization operations (e.g., Softpick), disallows the concentration of mass and hence eliminates attention sinks. Softpick's formula
$$
\text{Softpick}(x)_i = \frac{\max\{0, e^{x_i - m} - e^{-m} \}}{\sum_{j=1}^n |e^{x_j - m} - e^{-m}| + \epsilon}
$$
guarantees 0% sink rate in both empirical and theoretical analyses. The result is elimination of massive activations and more robust quantization at ultra-low bitwidths [2504.20966].

- **Architectural realignment:** Architectures such as EDIT for vision maintain attention stability by reengineering the encoder–decoder coupling, preventing [CLS] from collapsing to a sink and ensuring distributed, layer-wise information flow. This preserves richer token representations and yields consistently better segmentation and transfer performance compared to vanilla ViTs [2504.06738].

- **Dynamic sink relocation or suppression:** Randomizing the initial token position, or controlling key bias via small additive logit modifications, produces weaker or distributed sinks at the cost of interpretability or mixing [2410.10781, 2504.02732].

- **Explicit sink tokens in diffusion:** In DLMs, attention sinks migrate unpredictably due to dynamic masking. Inserting a dedicated token constrained by an attention mask to attend only to itself provides a static, low-norm, globally visible sink, restoring stability and regularization [2601.19657].

## 6. Experimental and Empirical Characterization

Empirical analysis demonstrates that stable sink tokens:

- **Arise universally in transformer models trained with softmax and sufficient data,** even in small models [2410.10781].
- **Emerge early during pretraining,** forming within several thousand training steps and before task-loss convergence [2508.02546, 2410.10781].
- **Exhibit context- and input-agnostic stability:** The specific slot acting as sink is determined by pretraining choices (e.g., first position, special marker), but the effect is highly robust to data type and packing [2504.02732, 2410.10781].
- **Produce measurable topological and spectral signatures** in attention matrices: persistent Betti numbers, dominant eigenvalue gaps, and low key-norm in the sink slot [2508.02546].
- **Drive downstream behavior:** Disabling or quantizing sink tokens directly degrades performance and increases perplexity or accuracy loss [2508.04257, 2601.19657, 2507.03865].

A selection of empirical observations:

| Empirical Observation      | Paper/Source        | Finding                                   |
|---------------------------|---------------------|--------------------------------------------|
| $\Delta_s^l \approx 0$    | [2507.03865]        | Sink token direction static after $l_\text{sink}$  |
| Sink rate $\sim$60–80%    | [2410.10781, 2504.02732] | LLMs (70B+) exhibit high sink rate after pretraining      |
| 0% sink rate with Softpick| [2504.20966]        | Rectified normalization eliminates sinks   |
| OrthoRank $\Delta$PPL $=-0.4...-0.8$ vs pruning| [2507.03865]  | At 20% compute, superior to standard layer-pruning |

These findings demonstrate the universality, impact, and practical leverage of stable sink tokens across scales, modalities, and domains.

## 7. Broader Implications and Future Directions

The existence and functionality of stable sink tokens have influential consequences for model design, interpretability, and efficient inference:

- **Model compression, quantization, and mixed-precision inference** benefit from precise recognition and tailored treatment of sink tokens, as these are often the largest sources of activation outliers [2508.04257, 2504.20966].
- **Architectural explorations**—such as encoder–decoder realignment (EDIT), adaptive attention normalization (Softpick), and explicit no-op token insertion in diffusion models—are directly motivated by insights into the role of stable sink tokens [2504.06738, 2504.20966, 2601.19657].
- **Emerging downstream tasks,** such as extremely long context retention (video generation, streaming, hierarchical summarization), leverage persistent sink mechanisms for robust context anchoring without catastrophic forgetting [2512.05081].
- **Theoretical understanding of transformer geometry and information propagation** is fundamentally shaped by the reference-frame and attractor dynamics induced by stable sinks [2508.02546, 2504.02732, 2507.03865].

A plausible implication is that future work will increasingly focus on adaptive or data-driven formation of sink tokens, alignment with domain structure, and hybrid approaches that blend sink-aware sparsification with dynamic reference-frame selection. The theoretical insight that attention sinks are not quirk artifacts, but optimal solutions for imposing high-dimensional coordinate systems on transformer representations, reframes both the analysis and practical exploitation of the stable sink token phenomenon.

Source: https://www.emergentmind.com/topics/stable-sink-token