Mechanisms underlying attention sinks in alternative Transformer architectures

Investigate which mechanisms form attention sinks in Transformer architectures that omit learned query biases or use alternative positional schemes such as RoPE or ALiBi.

Background

The paper’s mechanistic analysis focuses on a GPT-2–style Transformer whose attention sinks arise through the interaction of learned query biases, the first-layer MLP-transformed positional encoding, and structure in the key projection. The authors emphasize that these components are individually dispensable because attention sinks also occur in architectures lacking one or more of them.

Consequently, the specific GPT-2 circuit cannot be assumed to generalize to Transformers without query biases or positional encodings, or to models using alternative positional representations. The unresolved problem is to identify the distinct circuits or mechanisms that generate attention sinks in those settings.

References

The broader Transformer ecosystem includes architectures that omit such biases or use alternative positional schemes (e.g., RoPE, ALiBi). While we do show that the specific components constituting the GPT-2 circuit are absent or altered in those settings, we leave the investigation of what mechanisms do form in those models to future work.

— A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation  (2604.14722 - Ran-Milo et al., 16 Apr 2026) in Section 5, Limitations, subsection “Scope across architectures and scales”

Our interventions reduce the sink substantially but do not eliminate it entirely: nullifying $b_Q$ or zeroing the massive-activation rows of $W_k$ leaves a residual sink. This indicates that secondary contributors beyond the three main components we isolate also play a role. Identifying them is a natural extension of our work which we leave to future work.

— A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation  (2604.14722 - Ran-Milo et al., 16 Apr 2026) in Section 5, Limitations, subsection “Secondary contributors”

We do not claim a definitive functional rationale for why such a sink is beneficial or harmful across tasks. Establishing the downstream utility or cost of the sink, and the conditions under which it is selected by optimization, is left for future work.

— A Mechanistic Account of Attention Sinks in GPT-2: One Circuit, Broader Implications for Mitigation  (2604.14722 - Ran-Milo et al., 16 Apr 2026) in Section 5, Limitations, subsection “Mechanism vs. function”