Sink Neurons in Transformer Models

• Sink neurons are specialized components in transformer models that isolate and aggregate targeted functional behaviors, such as memorization and attention modulation.
• They function by creating dedicated representations—through mechanisms like attention sinks and super-neuron amplification—that balance global information broadcast and localized processing.
• Intervention strategies like head-wise RMSNorm, gated attention, and layer-wise gating effectively manage sink behavior to improve model stability and performance.

Sink neurons are specialized units or representational mechanisms within neural network models—predominantly transformers—that exhibit the property of concentrating, isolating, or monopolizing targeted functional behaviors. The “sink” concept arises in multiple forms: as individual neurons whose activity is structurally dedicated to specific tasks (such as the isolation of memorized content), as tokens or representations in attention layers that dominate the attention weights (attention sinks), or as dedicated mechanisms for suppressing or aggregating distributed information. This article surveys the mechanistic definitions, formation, and function of sink neurons and attention sinks, with emphasis on their role in transformer architectures, practical interventions, functional trade-offs, and empirical diagnostics (Li et al., 7 May 2026, Fesser et al., 6 Jun 2026, Súkeník et al., 8 May 2026, Ghosal et al., 14 Jul 2025).

1. Mechanistic Definitions of Sink Neurons and Attention Sinks

Sink neurons, as introduced in the MemSinks paradigm, are a reserved subset of hidden units in transformer feed-forward networks selectively activated on a per-sequence basis to accumulate and isolate memorization gradients (Ghosal et al., 14 Jul 2025). In parallel, attention sinks denote tokens—typically the first-token (BOS) in autoregressive models or other “hub” tokens—whose attention scores approach unity across all queries within a head or layer (Li et al., 7 May 2026, Fesser et al., 6 Jun 2026). Formally, for the output $o_i$ of an attention head:

$$A_{ij} = \mathrm{softmax}_j\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right), \qquad o_i = \sum_j A_{ij} v_j$$

A token $s$ is an $\epsilon$-sink if $A_{is} \geq 1-\epsilon$ for all $i$ (Fesser et al., 6 Jun 2026). In practice, sink behavior is usually diagnosed via vertical stripes in attention matrices, or by analyzing the value norms and activation statistics of individual neurons or dimensions associated with the sink token(s) (Choi et al., 1 Apr 2026, Li et al., 7 May 2026).

2. Structural Origins and Amplification Mechanisms

The formation of attention sinks in decoder-only transformers can be traced to the interplay of self-attention value aggregation, feed-forward “super-neuron” amplifications, and dimension disparity effects (Li et al., 7 May 2026). Causal masking induces a systematic positional variance discrepancy in the value aggregation step:

$$V'_{i,k} = \sum_{j=0}^{i} \alpha_{ij} V_{j,k}, \qquad \operatorname{Var}[V'_{i,k}] = \sum_{j=0}^i \alpha_{ij}^2 \operatorname{Var}[V_{j,k}]$$

Here, the variance for the initial token $i=0$ is maximal, decaying sharply for $i>0$. Feed-forward layers amplify this outlier variance: “super-neurons”—columns in the MLP weight matrices with large $\ell_2$-norms—become highly active only on the high-variance outlier, funneling activation into a sparse set of output dimensions (dimension disparity). This produces a highly anisotropic representation at the sink position, locking the query-key geometry to favor persistent sink behavior in subsequent layers (Li et al., 7 May 2026).

In vision and vision-LLMs, attention sinks also emerge from structural properties of the Vision Transformer encoder (V-sinks) or are constructed within the language module (L-sinks) via massive, dimension-specific activations, with each class playing distinct roles in encoding global scene priors and fine-grained features (Choi et al., 1 Apr 2026).

3. Functional Roles and Computational Interpretations

In transformer models, attention sinks serve two primary computational functions (Fesser et al., 6 Jun 2026):

• Adaptive nop (null-operation) sinks: These route attention to a special token (usually with near-zero value), effectively suppressing the update and leaving the residual stream unchanged.
• Broadcast sinks: These allocate attention to tokens whose value vectors contain globally relevant information. All tokens attend to the broadcast hub to retrieve aggregate signals.

Mathematically, the adaptive nop mechanism is implemented when $$A_{ij} = \mathrm{softmax}_j\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right), \qquad o_i = \sum_j A_{ij} v_j$$0 and $$A_{ij} = \mathrm{softmax}_j\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right), \qquad o_i = \sum_j A_{ij} v_j$$1, while broadcast mechanisms manifest as rank-1 residual updates due to all $$A_{ij} = \mathrm{softmax}_j\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right), \qquad o_i = \sum_j A_{ij} v_j$$2 and $$A_{ij} = \mathrm{softmax}_j\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right), \qquad o_i = \sum_j A_{ij} v_j$$3. Diagnostic metrics—value-norm traces $$A_{ij} = \mathrm{softmax}_j\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right), \qquad o_i = \sum_j A_{ij} v_j$$4, and stable-rank $$A_{ij} = \mathrm{softmax}_j\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right), \qquad o_i = \sum_j A_{ij} v_j$$5 of the residual update matrix—separate these regimes (Fesser et al., 6 Jun 2026).

Sink neurons in architectures like MemSinks serve orthogonal purposes: they localize the capacity for memorization to a dedicated, sequence-specific subspace, facilitating complete post-hoc removal of memorized content without affecting generalization (Ghosal et al., 14 Jul 2025).

4. Diagnostic Criteria, Geometric Conditions, and Empirical Signatures

The existence and implementability of sink patterns in attention depend on geometric-algebraic separability. For a perfect sink on token $$A_{ij} = \mathrm{softmax}_j\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right), \qquad o_i = \sum_j A_{ij} v_j$$6 attending to token $$A_{ij} = \mathrm{softmax}_j\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right), \qquad o_i = \sum_j A_{ij} v_j$$7 (BOS), it is both necessary and sufficient that all key embeddings $$A_{ij} = \mathrm{softmax}_j\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right), \qquad o_i = \sum_j A_{ij} v_j$$8 and $$A_{ij} = \mathrm{softmax}_j\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right), \qquad o_i = \sum_j A_{ij} v_j$$9 lie in a common half-space (Súkeník et al., 8 May 2026). This is almost always satisfied in practice for transformer models (LLaMA, GPT-2, Gemma, Mistral). Conceptually, sink heads create attention switches—hard switches when the output is exactly zero for dormant queries, with soft switches allowing self-communication via diagonal patterns.

Empirically, dense (nearly uniform) attention induces oversmoothing by raising the average cosine similarity among token representations more than sparse (sink-dominated) patterns, unless the $s$0 map is specifically anti-aligned with token differences. Sinks are observed far more commonly than diagonal heads, due to their lower representational and regularization cost, particularly for large context lengths (Súkeník et al., 8 May 2026).

In vision transformers, sink behavior is layer- and head-specialized, with the [CLS] token as a sink early, diffuse or patch-specific sinks in intermediate layers, and deep specialization in later layers (Fesser et al., 6 Jun 2026). Histogram analyses reveal head-level specialization, often with “vertical stripe” patterns corresponding to frequent sink usage.

5. Intervention Strategies and Architectures

Multiple algorithmic interventions have been developed to control, mitigate, or exploit sink neuron and attention sink behavior:

• Mask-based and variance-amplification interventions: Direct manipulation of attention masks or value variance at specific positions can force arbitrary tokens to become sinks, confirming causality in sink formation (Li et al., 7 May 2026).
• Head-wise RMSNorm: Applying normalization independently to each head after aggregation stabilizes per-position variance and eliminates high-variance sink outliers, restoring uniformity and accelerating convergence (Li et al., 7 May 2026).
• Gated attention: Insertion of per-token, per-dimension gates suppresses attention head output adaptively, highly effective in controlling adaptive nop sinks (Fesser et al., 6 Jun 2026).
• Register tokens: Adding learnable auxiliary tokens provides a persistent broadcast workspace for heads that require global aggregation functionality (Fesser et al., 6 Jun 2026).
• Layer-wise Sink Gating (LSG): Dynamic, input- and layer-adaptive scaling of sink versus ordinary token keys via small MLP modules, trained with next-token prediction, effectively balances global reasoning (sink-dominated) and local evidence (non-sink) in vision-LLMs (Choi et al., 1 Apr 2026).
• Sink-neuron masking in MemSinks: Sequence-tied dropout masks on dedicated sink neurons route memorization capacity into an isolated subspace, allowing precise post-hoc deletion (Ghosal et al., 14 Jul 2025).

Empirical results indicate intervention efficacy is context and task dependent—gating and register tokens yield complementary gains for stability and spatial performance in vision models, while architectural normalization such as head-wise RMSNorm eliminates pathological dimension disparity and accelerates pre-training in LLMs.

6. Functional Trade-offs and Practical Implications

Attention sinks facilitate efficient implementation of key functions in transformers: attention “switching,” suppression of redundant computation, aggregation of global priors, and prevention of oversmoothing. However, dominance of sink tokens may suppress the propagation of fine-grained evidence, which is detrimental for local or spatially sensitive tasks (Choi et al., 1 Apr 2026). The trade-off is especially pronounced in large vision-LLMs, where sink modulation must be task- and layer-specific to balance global and local requirements.

Architecturally, sinks are favored over diagonal (self-referential) attention patterns due to their lower cost under regularization constraints, especially as sequence length grows. In memorization, isolating content in sink neurons enables privacy and selective forgetting without degrading generalization, a property not achievable with post-hoc pruning or conventional expert mixtures (Ghosal et al., 14 Jul 2025).

7. Comparative Table: Approaches to Sink Formation and Control

Intervention/Mechanism	Functional Role	Key Outcome
Head-wise RMSNorm	Variance parity	Eliminates anomalous sink variance, speeds convergence
Gated attention	nop-sink control	Suppresses null-operation heads, limits oversmoothing
Register tokens	Broadcast sinks	Dedicated workspace for global aggregation
Layer-wise Sink Gating	Dynamic modulation	Balances global/local, improves VLM and OCR tasks
MemSinks sink neurons	Memorization isolation	Enables post-hoc forgetting, preserves generalization

This table summarizes distinct mechanisms for addressing, exploiting, or mitigating sink neuron phenomena across language, vision, and multimodal transformer models. Each approach is tailored to the functional requirements and failure modes revealed by mechanistic analyses (Li et al., 7 May 2026, Fesser et al., 6 Jun 2026, Súkeník et al., 8 May 2026, Choi et al., 1 Apr 2026, Ghosal et al., 14 Jul 2025).