Multi-Head Self-Attention Dynamics

• Multi-Head Self-Attention dynamics are the study of how transformer heads independently partition and interact, enabling diverse information routing and aggregation.
• Techniques like overlapping head schemas (MOHSA) and landmark-based compression (MHLA) demonstrate improved performance across tasks such as ImageNet classification and NLP.
• Recent work extends analysis to optimization and mean-field perspectives, offering insights into stability, convergence, and task-specific head specialization.

Multi-head self-attention dynamics concerns the way a Transformer’s heads partition, share, route, and refine information during attention computation, across depth, across tokens, and over training. In the canonical formulation, attention is defined by

$$Attention(Q,K,V)=Softmax\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$

with head-wise computation

$$Head_i=Attention(Q_i,K_i,V_i),\qquad MHSA(Q,K,V)=Concate(Head_1,\ldots,Head_n)W.$$

This standard decomposition makes heads diverse but computationally isolated until concatenation. Recent work has expanded the notion of MHSA dynamics beyond this independent-head picture, examining cross-head coupling, token-level head organization, locality-constrained alternatives, cache compression, head specialization in applied systems, and continuous-time or mean-field descriptions of head evolution under training (Zhang et al., 2024, Kang et al., 2024, Deora et al., 2023, Pendharkar, 5 May 2026).

1. Canonical head independence and the classical dynamic picture

In standard MHSA, the query, key, and value tensors are hard-partitioned across heads. Each head operates on its own $Q_i,K_i,V_i$, computes its own attention map, and only mixes with other heads after concatenation and output projection. A central implication is that there is no exchange of information between heads while attention is being formed; head interaction is deferred to the post-attention projection (Zhang et al., 2024, Kang et al., 2024).

Several later analyses take this independence as the baseline condition to be altered. The interactive multi-head self-attention literature explicitly treats each head’s attention matrix as a feature map and argues that independent computation limits information flow across heads. The optimization literature likewise formalizes a multi-head model as an average over head outputs scaled by $1/\sqrt{H}$, then studies how increasing $H$ changes curvature and stability. In application papers, head independence is often retained structurally but reinterpreted functionally: different heads become different “views” of the same sequence, different temporal subsegments, or different linguistic cues rather than strictly separate computational channels (Kang et al., 2024, Deora et al., 2023, N et al., 2021, India et al., 2019).

A separate but related line of work emphasizes that “multi-head” need not always mean feature-space partitioning. In MHLA, the decomposition is performed along the token dimension rather than the feature dimension: each block of tokens acts as a head-like unit, local key-value summaries are formed per block, and a learned mixing matrix recombines these summaries in a query-block-dependent manner. This reframes head dynamics from “multiple projections over the same token set” to “multiple token-group summaries with learned routing” (Zhang et al., 12 Jan 2026).

2. From isolated heads to coupled heads

A major development in the study of MHSA dynamics is the replacement of hard head separation with explicitly coupled head computation. In MOHSA, each head is overlapped with its two adjacent heads for queries, keys, and values. Instead of using only $Q_i,K_i,V_i$, head $i$ uses

$Q_i' = Concate(part(Q_{i-1}), Q_i, part(Q_{i+1})),$

and analogously for $K_i'$ and $V_i'$, with zero padding for the first and last heads. Attention is then computed on these overlapped neighborhood representations. This changes the dynamics from independent head processing to partially coupled head processing, because head $$Head_i=Attention(Q_i,K_i,V_i),\qquad MHSA(Q,K,V)=Concate(Head_1,\ldots,Head_n)W.$$0 is influenced by information that also appears in adjacent heads during attention computation itself (Zhang et al., 2024).

The empirical effects of this coupling are nontrivial rather than monotone. The paper studies fixed and depth-varying overlap paradigms, including fixed 1, fixed half, inc, dec, 0-indexed, and 1-indexed schedules. Its main conclusion is that small fixed overlap often helps, increasing or decreasing overlap with depth often works better than a fixed rule, and more overlap is not always better. On ImageNet, for example, CaiT-xxs24 improves from 70.56 to 74.26, while on Tiny-ImageNet the same model improves from 42.46 to 49.87; in an ablation on ViT-Tiny, overlapping all of $$Head_i=Attention(Q_i,K_i,V_i),\qquad MHSA(Q,K,V)=Concate(Head_1,\ldots,Head_n)W.$$1 gives 69.96 versus 69.15 for $$Head_i=Attention(Q_i,K_i,V_i),\qquad MHSA(Q,K,V)=Concate(Head_1,\ldots,Head_n)W.$$2 only and 69.62 for $$Head_i=Attention(Q_i,K_i,V_i),\qquad MHSA(Q,K,V)=Concate(Head_1,\ldots,Head_n)W.$$3 only (Zhang et al., 2024).

A second route to coupling appears in iMHSA, which introduces cross-head interaction through a decomposition of attention into compact query- and key-less factors. Query and key are downsampled into landmarks $$Head_i=Attention(Q_i,K_i,V_i),\qquad MHSA(Q,K,V)=Concate(Head_1,\ldots,Head_n)W.$$4, the attention matrix is factorized as $$Head_i=Attention(Q_i,K_i,V_i),\qquad MHSA(Q,K,V)=Concate(Head_1,\ldots,Head_n)W.$$5, and fully connected layers $$Head_i=Attention(Q_i,K_i,V_i),\qquad MHSA(Q,K,V)=Concate(Head_1,\ldots,Head_n)W.$$6 mix information across the head dimension. Direct cross-head interaction on full $$Head_i=Attention(Q_i,K_i,V_i),\qquad MHSA(Q,K,V)=Concate(Head_1,\ldots,Head_n)W.$$7 attention maps would cost $$Head_i=Attention(Q_i,K_i,V_i),\qquad MHSA(Q,K,V)=Concate(Head_1,\ldots,Head_n)W.$$8; after decomposition the added interaction cost becomes $$Head_i=Attention(Q_i,K_i,V_i),\qquad MHSA(Q,K,V)=Concate(Head_1,\ldots,Head_n)W.$$9, and the overall method has linear complexity $Q_i,K_i,V_i$0. The head-number analysis in that work shows that without interaction, performance plateaus quickly as head count increases, whereas with interaction, performance keeps improving with more heads (Kang et al., 2024).

MHLA addresses a different failure mode: global context collapse in linear attention. Standard kernelized linear attention compresses all tokens into one shared summary $Q_i,K_i,V_i$1, which all queries reuse. MHLA instead partitions the sequence into $Q_i,K_i,V_i$2 non-overlapping token blocks, builds local summaries $Q_i,K_i,V_i$3 and $Q_i,K_i,V_i$4 for each block, and uses a learned mixing matrix $Q_i,K_i,V_i$5 so that each query block receives its own mixture of block summaries. The paper argues that this restores representational diversity and query-conditioned selectivity while preserving linear complexity. Its analysis shows that the attainable rank can grow roughly additively with the number of blocks rather than being capped by the head dimension $Q_i,K_i,V_i$6, and its experiments report a 3.6\% improvement on ImageNet classification, a 6.3\% gain on NLP, a 12.6\% improvement on image generation, and a 41\% enhancement on video generation under the same time complexity (Zhang et al., 12 Jan 2026).

3. Locality, receptive field, and memory compression

MHSA dynamics has also been studied through mechanisms that restrict or restructure the effective receptive field. A strong example is the multi-head neural $Q_i,K_i,V_i$7-gram model, which replaces global self-attention with multiple independent nonlinear projections over local concatenated windows: $Q_i,K_i,V_i$8 This model is local and feed-forward rather than globally content-routed, yet on sequence-to-sequence tasks it is often comparable to or better than self-attention. On IWSLT DE-EN it achieves 35.49 BLEU versus 35.34 for self-attention, and hybridization with self-attention improves BLEU by up to 0.50 when multi-head neural $Q_i,K_i,V_i$9-gram is placed in lower layers. The paper’s interpretation is that local aggregation and global dependency modeling are complementary rather than redundant (Loem et al., 2022).

Temporal restructuring is pursued from an inference perspective in MTLA. Standard multi-head attention stores a full key and value vector per head per past token; MTLA extends multi-head latent attention by compressing the key-value cache in the time dimension as well as the feature dimension. Latent vectors are dynamically merged by a hyper-network, reducing cache length from $1/\sqrt{H}$0 to $1/\sqrt{H}$1, and a stride-aware causal mask is introduced to align training with incremental inference. The paper states that the self-attention component’s per-token decoding complexity is reduced from $1/\sqrt{H}$2 to $1/\sqrt{H}$3. On MuST-C En-De speech translation, MHA takes 281.3 s and 18,646 MiB, while MTLA with default $1/\sqrt{H}$4 takes 65.6 s and 2,835 MiB; with $1/\sqrt{H}$5, speed rises to 5.34× and memory reduction to 8.28×, while BLEU remains 23.25 (2505.13544).

Axis-wise decomposition yields another notion of receptive-field dynamics. In U-Former, MHSA is applied separately along the time and frequency axes of an encoded spectrogram, producing time and frequency sub-attention maps that are fused by

$1/\sqrt{H}$6

In the same architecture, multi-head cross-attention in skip connections gates encoder features by decoder queries and suppresses uncorrelated or noisy components through

$1/\sqrt{H}$7

At 0 dB, the full model reaches STOI 91.69 and PESQ 2.78, compared with STOI 86.79 and PESQ 2.65 without MHSA (Xu et al., 2022).

4. Specialization, pooling, and task-conditioned head behavior

In many applications, MHSA dynamics is observed not primarily as inter-head communication but as head specialization. In counterfactual detection, self-attention weights are extracted from the last three layers of pre-trained transformers as tensors $1/\sqrt{H}$8, processed by convolutional and linear blocks, and concatenated with pooled contextual embeddings. Qualitative analysis of final-layer BERT attention reports that heads 1 and 6 attend strongly to punctuation, heads 2, 4, and 12 focus on auxiliary verbs, heads 3 and 11 attend to conjunctions and verbs functioning as causal connectives, head 5 attends to entities, and head 7 attends to numerical values. This is presented as evidence that different heads become linguistically selective detectors of cues relevant to counterfactual structure (Patil et al., 2020).

Pooling-based speech systems show an analogous temporal specialization. In speaker recognition, self multi-head attention pooling assigns each head its own distribution over time,

$1/\sqrt{H}$9

and concatenates the resulting head summaries. Visualization shows that different heads attend to different regions of the same utterance, producing multiple alignments from different subsegments. On VoxCeleb1, CNN + MHA pooling yields 4.0% EER versus 4.90% for CNN + statistical pooling and 9.54% for i-vector + PLDA (India et al., 2019).

A related but deeper variant is serialized multi-layer multi-head attention for speaker embedding. Here attentive statistics are not computed once; instead, each layer produces weighted mean and weighted standard deviation, converts them to an utterance-level vector, and propagates that information to the next layer. The final embedding is obtained by summing the utterance-level vectors from all heads. The best results occur at $H$0 serialized layers, giving 3.99 EER on VoxCeleb1-H and 2.36 on VoxCeleb1-E, outperforming statistics pooling, attentive statistics pooling, and self-attentive pooling baselines (Zhu et al., 2021).

Other application papers emphasize how head count interacts with the burden assigned to each head. In end-to-end language identification from raw waveform, multi-head self-attention on LSTM outputs improves performance from 92.21% F1 for ResNet-LSTM to 95.90% F1 for ResNet-LSTM-MHA-RAW, an absolute 3.69% improvement; the paper interprets this as frame-level reweighting with multiple heads focusing on different language-discriminative regions (N et al., 2021). In Mandarin ASR, the authors argue that a small head size can create a shortage of learnable aspects, but branch-specific receptive fields can reduce the semantic burden on each head. Their ablation gives CER 8.8 with 2 heads, 6.91 with 4 heads, and 6.67 with 8 heads (Liu et al., 2023).

Compact non-Transformer formulations also exhibit task-conditioned multi-head behavior. LAMA uses a global context vector and low-rank factorization to generate multiple attention distributions efficiently. Validation accuracy improves substantially when moving from $H$1 to multiple heads, peaks around $H$2, and declines at $H$3; disagreement regularization often worsens performance. The paper therefore supports specialization without treating maximal head diversity as universally beneficial (Mehta et al., 2019).

5. Optimization, mean-field, and continuous-time descriptions

A substantial theoretical literature now treats MHSA dynamics as an optimization and dynamical-systems problem rather than only an architectural one. In the gradient-descent analysis of a single-layer multi-head self-attention classifier, the model is written as

$H$4

and the key weak-convexity bound is

$H$5

The central result is that the bad-curvature term shrinks like $H$6, so larger $H$7 makes the loss landscape more convex-like and improves algorithmic stability. Under the paper’s realizability and initialization conditions, polylogarithmically many heads suffice, specifically $H$8, and the training loss becomes $H$9 (Deora et al., 2023).

Mean-field analysis pushes this perspective to the infinite-head limit. In the cross-entropy setting, each head is treated as a particle $Q_i,K_i,V_i$0, the empirical law of the heads is

$Q_i,K_i,V_i$1

and the mean-field risk becomes a functional $Q_i,K_i,V_i$2 on probability measures. Its first variation $Q_i,K_i,V_i$3 generates the nonlinear Wasserstein gradient-flow PDE

$Q_i,K_i,V_i$4

The paper proves a static finite-head approximation bound of order $Q_i,K_i,V_i$5, a quantitative propagation-of-chaos estimate for finite-time SGD, and convergence results that require additional compactness, topological, or Kurdyka–Łojasiewicz assumptions to pass from stationarity to convergence and stability (Huan et al., 9 Jun 2026).

Another line of theory studies the training dynamics of linear attention in in-context learning. For the merged key-query parametrization, the dynamics has two fixed points and the loss shows a single abrupt drop; for the separate $Q_i,K_i,V_i$6 parametrization, the system has $Q_i,K_i,V_i$7 fixed points and follows saddle-to-saddle dynamics. In that separate parametrization, the model implements principal component regression in context, with the number of learned principal components increasing over training time (Zhang et al., 27 Jan 2025).

Continuous-time analyses of the token states themselves expose an additional sense of “dynamics.” In the single-head self-attention ODE on the unit sphere, equilibria are classified into consensus, bipartite consensus, clustering, and polygonal equilibria; consensus at the principal eigenvector is always asymptotically stable, consensus at nonprincipal eigenvectors is unstable, polygonal equilibria are unstable, and bipartite consensus may be stable or unstable depending on $Q_i,K_i,V_i$8 and the sign-splitting structure (Altafini, 14 Nov 2025). In the multi-head gradient-flow treatment, the total multi-head energy

$Q_i,K_i,V_i$9

is non-decreasing along both flat and spherical dynamics under score symmetry and value alignment, but per-head monotonicity is obstructed by radial shadow terms. In a scalar-head equiangular regime, the paper derives a closed-form critical inverse temperature for clustering and shows that heterogeneous heads exhibit super-additive clustering rates; it also proves an entropy production identity under which attention entropy increases monotonically toward equilibrium in that regime (Pendharkar, 5 May 2026).

6. Interpretive themes, misconceptions, and broader extensions

Several recurring themes cut across these lines of work. First, the common identification of MHSA with independent parallel heads is incomplete. MOHSA, iMHSA, and MHLA all alter head organization inside the attention mechanism itself rather than only after concatenation, and they do so for different reasons: adjacent-head information sharing, explicit cross-head mixing, or recovery of token-level representational diversity in linear attention (Zhang et al., 2024, Kang et al., 2024, Zhang et al., 12 Jan 2026).

Second, neither “more heads” nor “more interaction” is universally beneficial. MOHSA reports that more overlap is not always better; MHLA reports that the number of token-level heads matters but very large $i$0 begins to add overhead; LAMA reports a peak around $i$1 and degradation at $i$2; by contrast, the Mandarin ASR paper reports that 8 heads outperform 4 and 2 within its branch-structured design (Zhang et al., 2024, Zhang et al., 12 Jan 2026, Mehta et al., 2019, Liu et al., 2023). This suggests that head utility is architecture- and task-dependent rather than governed by a single monotone law.

Third, full global self-attention is not the only viable dynamic. Multi-head neural $i$3-gram can match or exceed self-attention on several sequence-to-sequence tasks, and its hybridization results indicate that local multi-head neighborhood aggregation may be particularly useful in lower layers or decoder components (Loem et al., 2022). A plausible implication is that MHSA dynamics should be understood as a family of routing and aggregation mechanisms, not as a single canonical pattern of global token-to-token matching.

Fourth, attention weights can be operational rather than merely explanatory. In counterfactual detection they are used as explicit task features, in speaker recognition they function as learned pooling weights over variable-length utterances, and in serialized speaker embedding they become layer-wise statistics propagated across depth (Patil et al., 2020, India et al., 2019, Zhu et al., 2021). At the same time, the interpretability evidence in these papers is predominantly qualitative, based on head visualizations or top-attended tokens rather than formal causal claims.

Finally, the concept of MHSA dynamics has extended beyond standard machine-learning settings. One proposal maps multihead linear self-attention onto cortico-thalamic circuitry: superficial pyramidal cells compute an attention mask from key-query similarity, deep pyramidal cells compute values modulated by that mask, a cortical area corresponds to one head, and higher-order thalamus integrates head outputs. The same work derives formal gradient expressions for a multihead linear self-attention block but states that a complete biologically local implementation of those updates remains unresolved (Granier et al., 8 Apr 2025).

Taken together, these studies depict multi-head self-attention dynamics as a multi-scale phenomenon. At the computational level it concerns whether heads are isolated, overlapping, token-partitioned, or explicitly interactive. At the architectural level it concerns how locality, receptive field, and cache structure constrain or enrich head behavior. At the optimization level it concerns curvature, stability, fixed points, and mean-field limits. At the application level it concerns whether heads act as specialized detectors, pooling operators, or routing channels. The cumulative literature therefore treats head dynamics not as an implementation detail, but as a central design variable in the behavior of Transformer-like systems (Zhang et al., 2024, Deora et al., 2023, Huan et al., 9 Jun 2026).