Dual Attention Sinks: Theory & Applications

• Dual attention sinks phenomenon is defined as specific positions in a model's input that disproportionately capture attention across channels, tokens, or modalities.
• It appears in diverse fields such as topological photonics and deep learning, where both physical energy and computational focus accumulate in designated system nodes.
• Researchers can mitigate negative impacts by applying calibration techniques, pruning dormant heads, and redistributing attention, thereby enhancing model efficiency and robustness.

The dual attention sinks phenomenon describes the presence and implications of attention sinks—positions in a model's input sequence that receive disproportionate attention weights—from a variety of topological, algorithmic, and empirical perspectives. While the term "dual" takes on distinct technical meanings across fields, a unifying theme is the existence, transfer, or co-occurrence of sink behavior across multiple channels (e.g., reciprocal and nonreciprocal photonic systems), positions (e.g., initial and intermediate tokens in LLMs), or modalities (e.g., text and vision tokens in multimodal transformers). The phenomenon is robust across disciplines: topological photonics, deep learning models for vision and language, and even video diffusion transformers. It is both a theoretically grounded and practically influential aspect of modern AI and physics.

1. Topological Energy Sinks and Duality in Electromagnetic Continua

In topological photonic systems, energy sinks are physical loci where electromagnetic energy accumulates with ultra-enhanced field intensity. The formation of these energy sinks is rooted in a breakdown of the bulk-edge correspondence—a principle tying the number of protected edge modes to topological invariants (Chern numbers) of the system’s band structure. In electromagnetic continua without a spatial cut-off, interfaces between distinct topological phases can fail to support edge modes, resulting in one-way (nonreciprocal) waves terminating at a point where energy becomes trapped and can diverge in the lossless limit: $$\frac{d\mathcal{E}}{dt} + \gamma\mathcal{E} = P_\mathrm{in},$$ with energy diverging as $\gamma \to 0$.

Through parity-time-duality ($\mathcal{P}\mathcal{T}\mathcal{D}$) symmetry, this sink behavior can be mapped from nonreciprocal (magnetically biased) to reciprocal platforms. Duality transformations transfer the energy sink property onto reciprocal meta-materials with engineered symmetries, preserving the singular, field-concentrating behavior. Thus, dual topological attention sinks occur in both nonreciprocal and reciprocal systems, each hosting modes that accumulate and focus energy similarly. The dual nature here reflects the ability of topological symmetry to guarantee sink phenomena in multiple regimes.

2. Dual-Path and Distributed Attention Sinks in Artificial Neural Architectures

In machine learning, the term "dual attention" appears in architectures designed to process both local and global dependencies in parallel. The DualFormer model, for example, integrates both a convolutional branch (local attention) and a transformer-based global attention branch. Rather than manifesting as a failure mode, the dual attention mechanism is an architectural strategy that yields discriminative, hierarchical representations. However, the "sink" phenomenon proper, as characterized in LLMs, refers to emergent behaviors where certain tokens (often the initial token or other semantically weak tokens) become the central focus of attention :

• In streaming LLMs, initial tokens act as attention sinks essential for stabilizing attention scores, anchoring the softmax normalization, and enabling efficient handling of infinite-length sequences. Removal of these sink tokens (e.g., through windowed KV caching without sink preservation) leads to catastrophic performance degradation, underscoring their functional role as computational anchors.
• Empirical studies show that such sinks are not always limited to the sequence start: attention sinks can emerge throughout the input, with distributed or "dual" (multiple) tokens attracting disproportionately high attention.

This distributed sink phenomenon impacts both performance and model optimization, as attention focused on non-informative tokens can hinder accuracy, but proper calibration (e.g., via on-the-fly Attention Calibration Techniques) frequently improves results. The duality here consists in the possibility of multiple, contextually-determined sink positions, rather than a single static anchor.

3. Spectral, Subspace, and Mutual Reinforcement Perspectives

A spectral decomposition of model embedding and unembedding matrices reveals that attention sinking is facilitated by signals in the tail (dark) end of the spectrum. Tokens acting as sinks are associated with large projections onto these subspaces, constituting a "hidden channel" enabling surplus attention to be funneled away harmlessly. This spectral framework supports the presence of primary and secondary (dual) attention sinks, where multiple tokens with similar spectral signatures absorb attention with minimal impact on output.

Mechanistically, in simplified models and large pretrained transformers, a mutual reinforcement mechanism is observed: as attention concentrates on a sink token, the output values for that token shrink, further reinforcing the collapse of attention and driving heads into dormant states. Heads can transition between active (meaningful) and dormant (sink-dominated, near-zero output) depending on data and training phase.

4. Practical Applications, Model Design, and Mitigation Strategies

Attention sinks have both positive and negative implications:

• Positive: In LLMs, they mitigate over-mixing (rank collapse) in deep transformer stacks, serving as controllable "no-op" heads to preserve representation diversity as context lengths and depth grow. In vision transformers, dual attention structures facilitate scalability and representation richness.
• Negative: Redundant, sink-dominated (dormant) heads consume resources without contributing to output. Identifying and zeroing out dormant heads (assessed by head output average norms) can significantly compress models and speed up inference with negligible accuracy loss. In continual learning, attention sinks, if unmanaged, cause over-smoothing and cross-task interference, requiring training protocols that enforce attention diversity.
• Mitigation: Interventions—such as replacing softmax with unnormalized kernels (sigmoid, ReLU), introducing explicit key biases, retraining sink-prone layers, or applying targeted patches to sink neurons—can remove or control sink phenomena as needed.

These methods are equally relevant for streaming, compression, and robust deployment, and the practical significance extends to multimodal models and video diffusion transformers, where attention sinks confined to specific layers or heads can be reinitialized and retrained to unlock sparsity and efficiency gains.

5. Multimodal and Vision-Enhanced Models: Cross-Modal Attention Sinks

In large multimodal models (LMMs), attention sink behavior is observed in both textual and visual domains. Visual tokens associated with background content can become visual attention sinks, evidenced by massive activations in specific hidden state dimensions. Removing these tokens or redistributing attention away from visual sinks (e.g., using Visual Attention Redistribution) enhances multimodal understanding and reduces hallucination without retraining.

A summary table illustrates typical forms:

Aspect	LLMs (LMs)	Multimodal Models (LMMs)
Typical Sink Token	BOS, ".", ",", "\n"	Background patches
Hidden State Activation	Massive in $\mathcal{D}_{sink}$	Massive in $\mathcal{D}_{sink}$
Effect if Removed	Negligible	Negligible
Mitigation	Removal, reallocation, patch	Redistribution (VAR)

This cross-modal consistency reinforces the interpretation of attention sinks as architecture- and training-induced, rather than functionally meaningful, features.

6. Implications for Theory, Practice, and Future Research

The dual attention sinks phenomenon is increasingly recognized as integral to understanding, optimizing, and deploying deep transformer architectures. The presence of multiple sink tokens or dormant heads provides both a source of practical computational savings and a theoretical lens for interpreting over-squashing, stability, and redundancy in deep networks.

The duality concept, whether realized via physical symmetry (in topological photonics), architectural fusion (dual-path models), spectral subspace construction (embedding analysis), or algorithmic redundancy (dormant heads), underpins model robustness and adaptability across domains. Control and management of attention sinks—via calibration, pruning, spectral filtering, or retraining—are likely to remain central in scaling, compressing, and robustifying next-generation models.

Future research may focus on characterizing the formation and transition of sinks in pretraining, generalizing intervention strategies, and clarifying the interplay of architectural and data factors that set their location, multiplicity, and effect. The phenomenon also suggests new pathways for efficient model pruning, interpretability, and continual learning strategies, emphasizing the enduring importance of attention mechanisms and their emergent properties.