Attention Sinking Phenomenon

• Attention sinking phenomenon is defined as the disproportionate absorption of attention in physical, granular, and transformer systems, evidenced by collective sinking and low-rank spectral behavior.
• Studies link this effect to experimental observations in colloidal particle systems and granular media, with theoretical results supported by spectral and geometric analyses in transformer architectures.
• Understanding attention sinks informs calibration methods, quantization strategies, and adaptive inference, enhancing model performance and stability across various computational frameworks.

The attention sinking phenomenon, across its diverse scientific and machine learning contexts, denotes the disproportionate allocation or absorption of system “attention,” energy, or interaction into particular states, tokens, or physical locations. Originally observed as collective vertical displacement in complex particle systems, it has become central to the analysis of high-dimensional signal propagation, transformer attention maps, and even emergent behavior in granular and geotechnical systems.

1. Physical Origin: Collective Sinking in Interfacial Particle Systems

The earliest rigorous formulation of attention sinking derives from colloidal particles at fluid interfaces (Lee et al., 2016). Charged particles confined to an air–water interface experience long-range dipolar repulsion stabilized by capillary effects. When many such particles assemble, their individual meniscus deformations overlap, which forces the interface to “sink” further to maintain vertical force balance—termed collective sinking. For $N$ particles, this increases both the depth and the interaction energy per particle, which scales approximately as $U_N/N \sim \frac{1}{4} N W^2$ for weight-per-length parameter $W$ and dimensionless number $N$. The many-body effect dominates over simple pairwise energy estimates, and cluster formation is driven by the amplified capillary field:

Parameter	Definition	Role
$W$	$mg/\gamma$	weight per interfacial tension
$C$	$q^2/(2\pi\varepsilon_0\gamma)$	dimensionless charge for repulsion
$d$	Mean particle separation	determined by $C^2/W^2$ and $N$

Theoretical and experimental findings converge: as concentration increases, particles in the center of a raft are more deeply sunk—a physical analog for “focal attention” in interactive fields or networks.

2. Sinking Dynamics and Extreme Events in Granular and Geological Systems

Granular droplets suspended in vibro-fluidized beds display a related attention sinking phenomenon by virtue of their density contrast, frictional inter-particle contacts, and minimal droplet diameter (Metzger et al., 2021). Instead of pure vertical motion, sinking droplets encounter an immobilized zone of bulk particles underneath, forcing lateral spread and sequential binary splitting—akin to fragmentation in Rayleigh-Taylor instabilities, but driven by discrete force networks rather than capillarity. The process critically depends on high-density droplets generating strong, localized contact networks; repeated splitting events are governed by the geometry and evolution of immobilized regions.

Liquefaction in saturated soils, traditionally attributed to shear-driven compaction and pore pressure elevation, can occur via a buoyancy-controlled mechanism where seismic acceleration reduces effective friction at grain contacts, resulting in sinking or “liquefaction” even under drained or well-compacted conditions (Clément et al., 2018). The dimensionless threshold for triggering sliding (and thus sinking) is

$I_L = \mu \frac{P_g - P_w}{P_g}$

with $\mu$ the friction coefficient, $P_g$, $P_w$ grain and water densities. Experiments and simulations underscore this alternative route to sinking phenomena in geomaterials.

3. Emergence and Structure of Attention Sinks in Transformer Models

In transformer architectures, the attention sinking phenomenon arises as a fundamental geometric and spectral property of attention layers and residual streams. Certain tokens (often special anchors such as [CLS] or BOS) attract a disproportionate share of cumulative attention due to intrinsic architecture and training dynamics (Gu et al., 14 Oct 2024, Ruscio et al., 4 Aug 2025). Softmax-based attention maps enforce a normalization over the probability simplex $\Delta^{n-1}$, which, under strong dot product alignment, consistently concentrates mass on reference tokens.

Geometric Perspective

The formation of attention sinks is explained via reference frames—centralized (e.g., BOS anchor in RoPE decoders), distributed (multiple intermediate anchors in NTK-aware models), or bidirectional (dual [CLS]/[SEP] anchors in absolute PE encoders) (Ruscio et al., 4 Aug 2025). These emerge as solutions for establishing stable coordinate systems in high-dimensional representational spaces. The mathematical signature is

$$\text{sink}(j) = \left( \frac{1}{n} \sum_i \mathbf{1}\{\alpha_{ij} \geq \tau\} \right) \geq \gamma$$

with $\alpha_{ij}$ the attention weight from $i$ to $j$, threshold $\tau$, and frequency parameter $\gamma$.

Spectral and Low-Rank Analysis

Spectral analysis reveals that dominant attention sinks correspond to the formation of massive activation outliers in the residual stream (Queipo-de-Llano et al., 7 Oct 2025, Wang et al., 23 Aug 2025). As a result, the representation matrix $X$ exhibits near rank-1 collapse (“compression valley”), quantified by the top singular value

$\sigma_1^2 \geq \Vert x_0 \Vert^2 + \alpha R$

where $x_0$ is the sink token, $R$ the residual norm, and $\alpha$ the alignment term. This reduces entropy and projects attention outputs into a low-dimensional active subspace, with about 60% of dimensions capturing 99% of the variance (Wang et al., 23 Aug 2025). If initialization of feature directions ignores this anisotropy, “dead features” (inactive sparse dictionary elements) proliferate.

Softmax-induced spectral gaps in the attention matrix lead both to collapse in depth (tokens converging with increasing layers) and in width (large context limits reducing effective rank per layer) (Saada et al., 10 Oct 2024). The dominant eigenvalue quantifies the absorbed signal, and removing this outlier improves representational diversity.

4. Detection, Calibration, and Preservation Strategies

Attention sinks play crucial operational roles in transformers and require careful handling in compression and optimization (Yu et al., 22 Jun 2024, Su et al., 6 Aug 2025). For example, in quantized KV cache implementations, sink tokens correspond to extreme stable outliers. Failure to preserve these tokens during quantization propagates substantial error throughout the network. The KVSink algorithm leverages cross-layer tracking of activation outliers to efficiently select which tokens need higher-precision preservation; empirical results demonstrate marked improvement in perplexity under memory-optimized deployment (Su et al., 6 Aug 2025).

Training-free calibration methods—such as ACT—identify and modulate undue attention concentration on sinks (Yu et al., 22 Jun 2024). By adaptively redistributing excessive attention throughout inference, significant accuracy gains can be achieved across many LLM tasks, suggesting that not all attention sinks are useful and that targeted correction improves overall model performance.

5. Robustness and Dynamics in Emerging Architectures

Masked Diffusion LLMs (DLMs) present a distinctive variation of the attention sinking phenomenon (Rulli et al., 17 Oct 2025). Attention sinks in DLMs are bidirectional, dynamic, and migrate across denoising steps, often aligning with punctuation or structurally salient tokens rather than static anchors. Unlike autoregressive models, which are catastrophically sensitive to sink removal, DLMs remain robust, suffering only minor degradation upon sink masking due to parallel, iterative unmasking and distributed attention dynamics. This suggests potentially superior resilience to context truncation and efficient attention routing, with promising implications for future architectural design.

6. Unified Frameworks and Implications

Recent works synthesize the theory and practice into unified frameworks, connecting attention sinking with representational compression, massive activation formation, and phase-wise information processing (Queipo-de-Llano et al., 7 Oct 2025). The proposed Mix-Compress-Refine theory posits three phases: broad mixing, compressed attention-sink-driven computation, and final selective refinement. These phases map to observed behavior across LLM families (410M–120B parameters):

Phase	Layer Depth	Dominant Mechanism	Functional Role
Mix	Early	Diffuse attention mixing	Context integration
Compress	Middle	Activation outlier/sinking	Semantic abstraction
Refine	Late	Norm equalization, pattern switch	Specific generation

Embeddings benefit from the compressed middle layers, while full-depth refinement is critical for accurate generative modeling.

7. Broader Implications and Future Directions

The attention sinking phenomenon—manifested through collective physical sinking, outlier formation, reference framing, and low-rank spectral collapse—represents a fundamental route by which systems achieve stable, compressed, and robust computation in high-dimensional contexts. For machine learning, this clarifies the intrinsic trade-offs between efficient attention allocation and representational flexibility, informing architecture, initialization, and spectrum-conscious parameterization. Future research may focus on adaptive phase-aware inference, dynamic attention routing, and further integration of geometric and spectral diagnostics for interpretable and efficient model design.