---
title: Self-Attention Heads in Transformers
url: https://www.emergentmind.com/topics/self-attention-heads
type: topic
---

# Self-Attention Heads in Transformers

Self-attention heads are the fundamental computational units within the multi-head self-attention mechanism of transformer architectures. Each head independently parameterizes projections of input feature sequences into subspace representations, computes pairwise contextual dependencies via scaled dot-product attention, and produces output streams that are subsequently fused across heads. This architecture enables simultaneous modeling of diverse patterns—including syntactic, semantic, positional, and recency cues—within deep neural models employed in domains such as language modeling, machine translation, speech recognition, and computer vision. Recent research has illuminated both the remarkable specialization and redundancy of self-attention heads, inspiring systematic advances in pruning, guided masking, head diversity maximization, efficiency-improving variants, and direct head manipulation for feature injection.

## 1. Mathematical Definition and Functional Role

In a standard multi-head self-attention block, for an input sequence $X \in \mathbb R^{T \times D}$, each of $H$ heads computes:

- Queries: $Q_h = X W_h^{\text{query}} \in \mathbb R^{T \times d_k}$
- Keys: $K_h = X W_h^{\text{key}} \in \mathbb R^{T \times d_k}$
- Values: $V_h = X W_h^{\text{value}} \in \mathbb R^{T \times d_v}$

The head output is given by:

\[
H_h = \mathrm{Attention}(Q_h, K_h, V_h) = \mathrm{softmax}\left(\frac{Q_h K_h^T}{\sqrt{d_k}}\right) V_h
\]

The multi-head output is then concatenated and projected:

\[
\mathrm{MultiHead}(X) = \mathrm{Concat}(H_1, \ldots, H_H) W^O
\]

Each head thus learns an independent contextualization subspace, determining what relationships to emphasize in the incoming sequence. Queries define which elements require information, keys serve as content-addressable memory, and values provide the data to be aggregated via the dynamically computed attention coefficients [2209.06096].

## 2. Specialization, Redundancy, and Pruning

Self-attention heads, though structurally uniform, develop highly heterogeneous functional roles in practice. Empirical analyses in translation and language modeling reveal three broad classes of heads:

- **Positional heads**: Attend with high probability to adjacent or fixed-offset tokens (e.g., next/previous token, diagonal identity) [1905.09418, 2002.10260].
- **Syntactic/semantic heads**: Specialize in tracking relationships such as subject–object links, modifiers, or rare tokens [1911.12246, 1905.09418, 2012.12366].
- **Rare/anchor heads**: Point toward tokens salient by infrequency (anchor words) or separators ([SEP], punctuation) [1905.09418, 2512.21956].

Quantitative head importance can be estimated via confidence metrics (max attention weight mean), variance metrics (spread of attended positions), or relevance propagation; direct pruning experiments reveal that the majority of heads, particularly in the encoder, can be ablated with negligible degradation (e.g., 38 out of 48 encoder heads pruned for a 0.15 BLEU drop in EN-RU MT) [1905.09418, 2105.14940]. This supports the principle that the "heavy lifting" is performed by a residue of well-specialized heads, while the remainder can be pruned or reassigned for auxiliary functions (feature injection, coreference augmentation) [2312.09541].

A striking finding is that—in many-to-one multilingual models—even the core set of most important heads is largely language-agnostic and stable across diverse source languages, indicating an absence of per-language head partitioning [2105.14940].

## 3. Diversity Across Heads: Measurement and Regularization

Despite the intent for heads to capture diverse information, empirical studies show that learned heads tend to collapse toward redundancy over the course of training. Cosine similarity metrics applied to head outputs, attention maps, and parameter gradients consistently find high inter-head correlation, especially in later layers [2209.06096, 2512.21956, 2402.17507]. To quantify this, the inter-head context correlation is defined as:

\[
d^Y(m,n) = \frac{1}{T} \sum_{t=1}^T \langle \tilde{y}_{m,t}, \tilde{y}_{n,t} \rangle
\quad , \quad \tilde{y}_{m,t} = \frac{y_{m,t}}{||y_{m,t}||}
\]

Imposing auxiliary diversity-promoting losses—such as cosine-orthogonality on head outputs or query projections—during training can meaningfully decorrelate heads and yield Word Error Rate improvements (up to 6% relative) in ASR systems [2209.06096]. Structured sparsity schemes like Fibottention explicitly differentiate head-level attention patterns (via Wythoff-dilated Fibonacci windows), reducing redundancy and maximizing feature diversity [2406.19391]. Head diversity also conditions optimization and generalization bounds; increased head count weakens the nonconvexity of the loss landscape (more convex-like), speeding convergence and tightening the generalization gap [2310.12680].

## 4. Interpretability: Linguistic and Task-Level Analysis

Rigorous probing has established that individual heads in transformer LMs encode identifiable linguistic abstractions:

- Certain BERT/RoBERTa heads reliably recover specific dependency arcs (e.g., $\text{obj}$, $\text{amod}$) with accuracy far above random and baselines (e.g., up to 86% UAS for object dependencies) [1911.12246].
- Heads often emerge that attend exclusively to rare tokens, topic anchors, or sentence separators—segregating discourse boundaries or binding repetitions [1905.09418, 2512.21956].
- In speech models, attention heads correlate with specific phoneme classes (e.g., fricatives, sibilants, or silence), representing a linguistically plausible division of labor [1803.09519].
- In role-guided or mask-constrained variants, explicit assignment of heads to targeted syntactic or lexical relations improves downstream task accuracy and interpretability, with attention matrices for MajRel (major relations) and DepSyn (dependency syntax) heads forming well-localized, functionally-aligned maps [2012.12366].

Automated constituency parsing via aggregation and scoring of self-attention heads (e.g., cost-rank knee-point selection, CKY tree induction) demonstrates that a handful of high-quality heads suffice to recover nontrivial treebank structure in an unsupervised, cross-linguistic manner [2010.09517].

## 5. Manipulating and Guiding Head Behavior

Several methods have been devised for inducing head-level specialization or guiding attention patterns:

- **Role-guided masks**: Non-learnable binary masks enforce per-head constraints (e.g., positional, syntactic, rare-word-oriented), assigning distinct linguistic roles [2012.12366].
- **Structure-injection by head manipulation**: Underused heads (low loss-sensitivity, as per $\partial\mathcal{L}/\partial\xi$) are selected for targeted feature biasing—e.g., coreference-aware matrices in dialogue summarization—yielding measurable gains (e.g., +2.0 ROUGE-2) and parameter savings [2312.09541].
- **Pruning and dynamic allocation**: Stochastic gating and hard-concrete L0 relaxations facilitate adaptive culling of redundant heads with minimal loss, confirming that fielded models frequently overparameterize at the head level [1905.09418].

Additionally, in large input settings (e.g., LVLMs), enhancing the prominence of vision-sink heads and broadcasting their focused attention maps to all heads in shallow layers can significantly reduce hallucination in image-text models without retraining (e.g., –3.9 CHAIR-I, +8.8 HallusionBench accuracy) [2411.09968].

## 6. Architectural Innovations: Efficiency and Interaction

New architectures explicitly reconsider head count, connectivity, and computational complexity.

- **Hydra Attention**: Sets head count to the feature dimension ($H=d$) under a linear attention kernel, yielding $O(nd)$ rather than $O(n^2d)$ complexity, with heads acting as single-dimension attention filters [2209.07484].
- **Fibottention**: Assigns to each head a distinct sparse mask constructed from Fibonacci/Wythoff dilations, achieving $O(N\log N)$ scaling and maximizing head mask diversity [2406.19391].
- **Overlapping Heads (MOHSA)**: Heads are made to overlap partially in their Q/K/V subspaces with neighbors, blending “hard” disjointness into “soft” local communication between heads; this delivers clear accuracy gains for vision tasks [2410.14874].
- **Interactive Cross-Head Attention**: Non-independence is enforced via decomposed attention matrices and lightweight head-mixing layers ($W_{1,2}$), breaking the linear independence between heads and mitigating redundancy [2402.17507].
- **Hybrid Role Replacement**: In models like RecurFormer, recency-aware heads are automatically detected (high recency ratio), then replaced with efficient linear RNNs (e.g., Mamba), slashing memory and computation while preserving long-range modeling via unaltered heads [2410.12850].

Finally, studies have shown that many learned heads, especially in the encoder of NMT models, can be replaced with non-learnable, position-centric patterns—e.g., next-token, identity, start/end-of-sequence biases—without loss of performance, especially in low-resource regimes [2002.10260]. This suggests that the majority of head capacity is leveraged for trivial structural priors rather than complex content-based reasoning.

## 7. Empirical Insights and Broader Implications

Evidence from a range of tasks and domains consistently demonstrates the overparameterization, redundancy, and yet critical specialization afforded by self-attention heads:

- Redundant heads can be pruned (≥70% in encoder) with ≤0.2 BLEU loss [1905.09418, 2105.14940].
- Core, high-confidence heads are consistently the first few per layer and exhibit linguistic role alignment and stability across languages [2105.14940].
- Head-level diversity (low inter-head similarity) correlates with model robustness and generalization; maximizing this via regularization or design (e.g., role-masks, Wythoff schemes) yields improved downstream results [2209.06096, 2406.19391].
- The absence of "generalist" heads capable of full syntactic parsing from raw attention weights points to the distributed, collective encoding of higher-order structures [1911.12246].

A plausible implication is that future architectures may benefit from further modularization at the head level—allocating, guiding, or specializing heads in a principled and task-adaptive manner to optimize both sample efficiency and inference scalability without sacrificing the compositional abstraction power that underpins the success of the Transformer family.

Source: https://www.emergentmind.com/topics/self-attention-heads