---
title: BOS Sink Phenomenon in Transformer Models
url: https://www.emergentmind.com/topics/bos-sink-phenomenon
type: topic
---

# BOS Sink Phenomenon in Transformer Models

The BOS sink phenomenon—often called the "attention sink" or simply "sink"—is a pervasive and theoretically profound pattern in modern sequence models employing attention mechanisms, particularly large transformer-based language models (LLMs). In the canonical case, the beginning-of-sequence (BOS) token, which typically carries negligible semantic content, consistently attracts a disproportionate fraction of self-attention from other tokens across layers and heads. This behavior roots in fundamental architectural, statistical, and geometric principles, and has significant consequences for model efficiency, compression, representational dynamics, and even security.

## 1. Defining the BOS Sink Phenomenon

In transformer models, multi-head self-attention routes each query token's representation across all keys in the input sequence. Empirically, starting from the shallow layers and persisting or accelerating through depth, the BOS token (usually position 0 or 1) acts as a universal "sink": most tokens send a large portion of their attention mass there, often exceeding 50–80% by mid to late layers [2507.03865][2601.06787]. Formally, for an attention map $A^{(\ell)}$, the BOS sink behavior can be characterized as:
$$
A^{(\ell)}_{i \rightarrow 0} \gg A^{(\ell)}_{i \rightarrow j}, \quad j \geq 1, \quad \forall i, \forall \ell \geq \ell_{\text{sink}}
$$
with the layer-wise average
$$
S_{\text{BOS}}^{(\ell)} = \frac{1}{H} \sum_{h=1}^{H} \left( \frac{1}{T} \sum_{t=1}^{T} \alpha^{(\ell, h)}_{t,0} \right).
$$
This "sink score" is robust across architectures, context lengths, model scales, optimization regimes, and even input randomness [2410.10781]. Removal, masking, or disturbance of the BOS token typically degrades performance [2508.02546].

## 2. Geometric and Representational Foundations

The sink token is not merely an attention-matrix artifact, but is deeply tied to the geometry of deep representation learning. Cosine similarity analysis between layer-normalized hidden states shows that, as depth increases, all tokens' representations $\hat h_i^{(\ell)}$ monotonically approach the BOS direction $\hat h_0^{(\ell)}$, which itself remains nearly fixed across depth:
- $\cos \theta_i^{(\ell)} = \langle \hat h_i^{(\ell)}, \hat h_0^{(\ell)} \rangle$ increases from $\sim$0.1–0.3 to $\sim$0.6–0.8 through the stack.
- $\cos \theta_0^{(\ell_1, \ell_2)}$ remains $>0.9$ across $\ell_1, \ell_2$.

This implies a representational "collapse" or attractor phenomenon, with all token embeddings spiraling toward the static BOS vector. The geometric explanation ties the sink to the establishment of a reference frame: in high-dimensional space, transformers naturally assign a stable axis or anchor—embodied by the BOS token—around which other representations are organized [2508.02546]. The softmax operation, by enforcing the probability-simplex constraint, further promotes sparsity and centralization of attention on such anchors.

## 3. Functional Consequences: Redundancy, Compression, and Over-Mixing Control

The BOS sink effect has several core functional implications:
- **Redundancy:** Heads (and even entire layers) with high BOS sink scores serve as "dumping grounds" for attention mass but contribute little to functional routing or mixing, becoming superfluous in downstream computation [2601.06787]. Pruning heads or layers ranked by $S_{\text{BOS}}^{(\ell, h)}$ preserves predictive accuracy substantially better than magnitude- or activation-based criteria, especially in deeper layers.
- **Over-Mixing Mitigation:** Unchecked global attention promotes rank and representational collapse (over-mixing), destroying distinctions among tokens and degrading model expressivity. The BOS sink acts as a throttle, providing a "controlled no-op": heads saturated on BOS pass previous representations unchanged, slowing down mixing and preserving diversity [2504.02732].
- **Information Partitioning:** The degree of orthogonality to the BOS direction can be leveraged to identify tokens carrying genuinely novel information versus those collapsing toward redundancy, a principle exploited in orthogonality-based token selection for inference speedup [2507.03865].

## 4. Quantification, Induction, and Universality

Quantitative metrics are well specified for sink detection:
- **Per-head Sink Score:** $ \mathrm{SinkScore}(\ell, h) = \frac{1}{T} \sum_{i=1}^T \alpha_{i,0}^{(\ell, h)} $
- **Model-level Sink Rate:** $ \mathrm{Sink}_0^{\epsilon} = \frac{1}{LH} \sum_{\ell, h} \mathbf{1}[\mathrm{SinkScore}(\ell, h) > \epsilon] $
Sink rates exceeding 30–40% at $\epsilon=0.3$ are ubiquitous in autoregressive models beyond the smallest scales (routinely $\sim$70–99%) [2410.10781].

**Emergence:** Sink behavior develops rapidly once sufficient data, optimization, and weight decay are applied during pretraining, saturating at large scale. The effect is stable across input domains, model scales, and even under strong variations in token content.

**Cause:** The underlying mechanism is the combination of softmax normalization and key/query parameterization. If any $q_i$·$k_0$ consistently dominates the logits, $A_{i,0}$ concentrates, leading to a global "attention bias register." Removing softmax normalization—e.g., by using sigmoid attention without L1 normalization—completely abolishes the sink effect, even in large models, confirming it is not an inherent necessity but a byproduct of attention competition [2410.10781].

## 5. Mitigation, Exploitation, and Model Design

A range of strategies are available, either to suppress, utilize, or redistribute the BOS sink effect.

- **Mitigation via Regularization:**
  - Penalizing large $\alpha_{i,0}^{(\ell, h)}$ during training suppresses emergent sinks [2510.17021].
  - Dropout or randomization of BOS embeddings prevents the model from over-relying on a fixed anchor.
  - Modifying positional encoding (e.g., NTK-aware scaling, ALiBi) distributes attention mass over multiple reference points, softening the centralized sink [2508.02546].

- **Utilization:**
  - The stability and universality of the BOS vector make it an effective global representational readout or coordinate anchor for downstream tasks.
  - Orthogonality to BOS—measured as $1 - \cos^2 \theta_i^{(\ell)}$—serves as a criterion for dynamic token selection (OrthoRank), enhancing inference efficiency by routing computation only through informative tokens [2507.03865].

- **Exploitation for Compression:**
  - Reliable pruning of high-sink heads or layers identifies structurally redundant elements without major loss in perplexity or downstream accuracy, outperforming magnitude- or activation-based methods [2601.06787].

- **Security and Adversarial Risks:**
  - Backdoor attacks targeting model unlearning leverage the BOS sink as a transmission gateway, with prefix triggers amplifying and propagating backdoor signals more efficiently due to the sink's "global amplifier" role. Mitigation requires attention sink auditing for adversarial risk [2510.17021].

## 6. Extension Beyond Language Models

BOS sink and related attention sink phenomena extend to audio-visual transformers and other multimodal architectures. In audio-visual speech recognition, the BOS and other intermediate tokens (e.g., prompt or modality markers) emerge as sinks, attracting not only attention but also massive activations in hidden state features, amplified by the MLP pathway. These sinks align in hidden space (cos-sim $\gtrsim$ 0.9) and amplify correspondence between attention and representation collapse [2510.22603]. Decorrelation objectives penalizing cosine similarity to BOS suppress sink behaviors and yield measurable improvement in downstream metrics such as word error rate.

## 7. Summary Table: Sink-Related Metrics and Implications

| Metric / Phenomenon        | Formal Definition / Quantification                             | Implication                                  |
|----------------------------|---------------------------------------------------------------|----------------------------------------------|
| Sink Score $S_{\text{BOS}}$ | $\frac{1}{T} \sum_{i=1}^T \alpha_{i,0}$                      | Quantifies degree to which head/layer is a BOS sink; enables pruning [2601.06787] |
| Cosine alignment to BOS    | $\langle \hat h_i^{(\ell)}, \hat h_0^{(\ell)} \rangle$        | Orthogonality indicates token information value [2507.03865] |
| Representational collapse  | $||V^{(L)} - \tfrac{1}{n}\mathbf{1}\mathbf{1}^\top V^{(L)}||_F \to 0$ | Warning for over-mixing; BOS sink as mitigation [2504.02732] |
| Massive activations        | $|H^{\ell}[i,j]| \geq \tau \cdot \text{median}(|H^{\ell}|)$   | Feature amplification in sinks, pointer to representational dominance [2510.22603] |
| Sink mitigation loss       | $\ell_{\text{sink}} = \sum_{\ell=1}^{L_s} \sum_h \frac{1}{T} \sum_i (\alpha_{i,0}^{(\ell, h)})^2$ | Mitigation of BOS sink via targeted regularization [2510.17021] |

## References

- [2507.03865] OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
- [2601.06787] Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning
- [2504.02732] Why do LLMs attend to the first token?
- [2508.02546] What are you sinking? A geometric approach on attention sink
- [2410.10781] When Attention Sink Emerges in Language Models: An Empirical View
- [2510.22603] Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMS
- [2510.17021] Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearning

Source: https://www.emergentmind.com/topics/bos-sink-phenomenon