---
title: InterTwining Attention in Transformers
url: https://www.emergentmind.com/topics/intertwining-attention
type: topic
---

# InterTwining Attention in Transformers

Searching arXiv for the cited attention papers and closely related terminology.
InterTwining Attention denotes a class of attention mechanisms that relax the standard independence of attention heads in Multi-Head Attention (MHA) or Multi-Head Self-Attention (MHSA) by introducing explicit cross-head interaction during attention computation. In this usage, “intertwining” refers to learned coupling among head-wise attention representations rather than mere post hoc concatenation of head outputs. Two concrete realizations in the arXiv literature are **interactive Multi-Head Self-Attention (iMHSA)**, which mixes head-wise attention maps under a decomposed linear-complexity formulation [2402.17507], and **Interleaved Head Attention (IHA)**, which constructs pseudo-heads as learned linear combinations of the original heads and thereby induces many more attention patterns than standard MHA [2602.21371]. A separate use of “intertwining” appears in probability theory, where it describes semigroup relations for Markov processes rather than neural attention; that usage is conceptually distinct despite the shared term [2112.11885].

## 1. Definition and conceptual scope

In standard MHSA, each head computes its own attention map independently and the head outputs are simply concatenated. The core claim made by the attention papers is that this architecture leaves **no information exchange** between heads during attention computation, which can limit information flow, head diversity, and compositional behavior [2402.17507, 2602.21371]. InterTwining Attention addresses that limitation by allowing one head’s attention representation to influence another head’s representation before the final value aggregation.

The concept is not a single canonical architecture. In the current literature, it encompasses at least two distinct design philosophies. The first performs head-to-head mixing directly on attention-map representations, treating attention matrices as features to be connected across the head dimension [2402.17507]. The second mixes queries, keys, and values across heads before the attention operator is applied, creating pseudo-heads whose pairwise interactions expand the set of attainable attention patterns [2602.21371].

This suggests that “InterTwining Attention” is best understood as an architectural principle rather than a fixed operator: heads are no longer isolated computational channels, but coupled components in a shared relational computation.

## 2. The limitation of independent-head attention

The motivation common to both attention papers is structural. Standard MHA produces exactly \(H\) independent attention matrices from \(H\) heads, and those heads do not communicate during attention computation [2602.21371]. In the vision setting, this independence is described as causing attention heads to become redundant and to exhibit performance saturation as head count increases [2402.17507]. In the language-model setting, the same isolation is presented as a bottleneck for multi-step or compositional reasoning, where correct outputs may require combining multiple latent token-to-token relations over a chain of intermediate inferences [2602.21371].

The standard self-attention formulation given for MHSA in the vision paper is
\[
\vb{z} = \{\vb*{z}_{i}|\vb*{z}_{i} \in \mathbb{R}^{c}\text{, and } 1 \leq i \leq N\},
\]
\[
\mathcal{Q}, \mathcal{K}, \mathcal{V} = (W_{Q}, W_{K}, W_{V})\vb{z},
\]
with per-head output
\[
\text{$\vb*{t_i} = \mathcal{A}\mathcal{V}$, where $\mathcal{A} = \mathrm{softmax}(\mathcal{Q} \mathcal{K}^{\top}/d^{\frac{1}{2})$}.
\]
The resulting head outputs are then concatenated. Operationally, each head computes an \(N\times N\) attention matrix and uses it independently [2402.17507].

The language-model paper formalizes the same isolation in standard MHA through
\[
\bm Q^{(h)}=\bm X\bm W_Q^{(h)},\qquad \bm K^{(h)}=\bm X\bm W_K^{(h)},\qquad \bm V^{(h)}=\bm X\bm W_V^{(h)},
\]
\[
\widetilde{\bm X}^{(h)} = \operatorname{softmax}\!\left(\frac{1}{\sqrt d}\bm Q^{(h)}\bm K^{(h)\top}\right)\bm V^{(h)},
\]
followed by concatenation and projection [2602.21371].

A plausible implication is that standard multi-head designs allocate representational capacity across heads, but not compositional interaction among heads at the moment when relational structure is formed.

## 3. Cross-head interaction as the basic intertwining mechanism

The most direct form of InterTwining Attention is explicit head-to-head mixing. In iMHSA, the attention maps from different heads are stacked and processed with learned linear layers across the head dimension:
\[
\text{$\mathcal{A} = W_{2}\mathrm{softmax}(W_{1}\mathcal{S})$, where $\mathcal{S} = \mathcal{Q} \mathcal{K}^{\top}/d^{\frac{1}{2}$,}
\]
with
\[
W_{1}, W_{2} \in \mathbb{R}^{h \times h}.
\]
The stated interpretation is that each head’s attention matrix is treated as a feature, and the learned matrices \(W_1\) and \(W_2\) mix these head-features so that head \(i\)’s attention can influence head \(j\)’s transformed attention representation [2402.17507].

In IHA, the interaction is moved upstream. For each original head \(h\) and pseudo-index \(j\in[P]\), pseudo-queries, pseudo-keys, and pseudo-values are formed as learned linear combinations of all original heads:
\[
\widetilde{\bm Q}_{h,j} = \sum_{m=1}^{H}\alpha^Q_{m,h,j}\,\bm X\bm W_Q^{(m)} \in\mathbb{R}^{N\times d},
\]
\[
\widetilde{\bm K}_{h,j} = \sum_{m=1}^{H}\alpha^K_{m,h,j}\,\bm X\bm W_K^{(m)} \in\mathbb{R}^{N\times d},
\]
\[
\widetilde{\bm V}_{h,j} = \sum_{m=1}^{H}\alpha^V_{m,h,j}\,\bm X\bm W_V^{(m)} \in\mathbb{R}^{N\times d},
\]
where
\[
\alpha^Q,\alpha^K,\alpha^V\in\mathbb{R}^{H\times H\times P}.
\]
These pseudo-heads are then stacked and fed through otherwise standard attention [2602.21371].

The distinction is operationally important. iMHSA intertwines **attention maps**; IHA intertwines **query, key, and value streams**. In both cases, however, cross-head communication is inserted before the final output concatenation. That commonality is the defining feature of the broader idea.

## 4. Two architectural realizations

The two attention papers instantiate the intertwining principle under different complexity constraints and for different application domains.

| Mechanism | Core operation | Stated complexity |
|---|---|---|
| iMHSA | Cross-head mixing on decomposed attention matrices | \(O(2NLh(d+h))\) [2402.17507] |
| IHA | Pseudo-head construction from learned cross-head Q/K/V mixing | Global IHA: \(O(P^2N^2d)\) [2602.21371] |

In iMHSA, a straightforward interaction on full \(N\times N\) attention matrices is said to incur
\[
O(N^2 h^2)
\]
extra computation, along with substantial memory growth because all head-wise attention maps must be stored [2402.17507]. The proposed solution is to decompose attention into reduced query-side and key-side components using spatially downsampled landmarks:
\[
q_{h,w} = \frac{1}{\Omega_{h}\Omega_{w}\sum\limits_{i=-\Omega_h}^{\Omega_h}\sum\limits_{j=-\Omega_w}^{\Omega_w}{Q_{h+i,w+j},
\]
\[
k_{h,w} = \frac{1}{\Omega_{h}\Omega_{w}\sum\limits_{i=-\Omega_h}^{\Omega_h}\sum\limits_{j=-\Omega_w}^{\Omega_w}{K_{h+i,w+j},
\]
followed by
\[
\mathcal{A} = \mathcal{A_{Q}\mathcal{A_{K} \ = \mathrm{softmax}(\mathcal{Q}_{i}k^\top) \mathrm{softmax}(q\mathcal{K}^{\top}).
\]
Here \(q, k \in \mathbb{R}^{L \times d}\), \(L \ll N\), and \(\mathcal{A_Q}, \mathcal{A_K} \in \mathbb{R}^{N \times L}\). The multiplication is reordered so that the full \(N\times N\) attention map is never explicitly formed:
\[
\vb*{o} = (\mathcal{A_{Q}\mathcal{A_{K}^\top)\mathcal{V} = \mathcal{A_{Q}(\mathcal{A_{K}^\top\mathcal{V}).
\]
The stated complexity is then
\[
O(2NLh(d+h)),
\]
which is linear in \(N\) for fixed \(L\), \(d\), and \(h\) [2402.17507].

In IHA, the objective is not linear-complexity self-attention but a richer head interaction pattern within the standard attention operator. Because each original head constructs \(P\) pseudo-queries and \(P\) pseudo-keys, a single head can induce up to
\[
P^2
\]
distinct attention patterns, typically with \(P=H\) [2602.21371]. The pseudo-output collapse is performed by
\[
\bm R\in\mathbb{R}^{H\times HP},
\]
and the paper states an added parameter cost of
\[
3H^2P + H^2P = 4H^2P
\]
beyond the base \(Q\) parameters of MHA [2602.21371].

These designs reflect two different trade-offs. iMHSA seeks efficient cross-head interaction under large token counts in vision. IHA seeks greater pattern capacity and improved compositional reasoning in language models, while explicitly acknowledging increased global attention cost.

## 5. Expressivity, scaling behavior, and theoretical claims

The most explicit expressivity results appear in the IHA paper. It proves
\[
\mathcal{M}\subsetneq \mathcal{P}_P\quad\text{for }P\ge 2,
\]
where \(\mathcal{M}\) is the set of MHA modules and \(\mathcal{P}_P\) is the corresponding IHA family [2602.21371]. Containment is obtained by choosing mixing tensors that effectively recover MHA, while strictness is argued through cases where IHA implements nonlinear behavior on repeated-token inputs for which MHA remains linear.

The same paper further analyzes synthetic tasks in which MHA’s “one head = one pattern” structure becomes a formal limitation. For representing
\[
\big[\bm X,\bm A\bm X,\dots,\bm A^{k-1}\bm X\big],
\]
it states that MHA needs \(k\) heads, whereas IHA needs only \(\lceil \sqrt{k}\rceil\) heads with \(P=H\) [2602.21371]. The key factorization is
\[
i=(h-1)H+(j-1),
\qquad
\bm A^i = \bm A^{(h-1)H}\bm A^{j-1}.
\]
The reported dominant parameter scaling improves from
\[
\Theta(kn^2)
\quad\text{to}\quad
\Theta(\sqrt{k}\,n^2).
\]

For the synthetic Count Permutation Match-3 task, the same quadratic expansion of representable patterns yields another head-efficiency separation. The paper states that MHA needs \(N\) heads in the worst case, whereas IHA uses
\[
H=\lceil\sqrt{N}\rceil
\]
heads with \(P=H\), giving up to \(H^2\) shift patterns from only \(H\) heads [2602.21371].

The iMHSA paper does not frame its contribution in the same formal expressivity language, but its analysis attributes improved behavior to richer information flow and increased feature diversity across heads. It reports that, without interaction, performance saturates quickly as heads increase, while with interaction, accuracy improves more consistently with more heads; it also states that attention maps become more diverse, with higher variance and lower similarity across heads [2402.17507].

Taken together, these results support a common interpretation: head isolation constrains either representational diversity or the efficient realization of multiple relational patterns, and cross-head coupling is a direct attempt to remove that constraint.

## 6. Empirical performance and application domains

The empirical literature represented here spans both vision and language modeling.

For vision, the iMHSA paper states that **interactive Multi-Head Self-Attention (iMHSA)** and the resulting **interactive Vision Transformer (iViT)** outperform prior efficient attention methods on ImageNet under standard token size, larger token size, and high-resolution inputs, specifically naming Performer, Local attention, Scatterbrain, Nyströmformer, LARA, Combiner, Long-Short, and EVA [2402.17507]. It also reports strong performance of iViT on ImageNet classification, COCO detection/instance segmentation, and ADE20K semantic segmentation, with the stated trend of better accuracy, good speed/FLOPs trade-off, and often fewer parameters than competing models [2402.17507]. At high resolutions, the paper notes that softmax attention can run out of memory, whereas iMHSA remains feasible.

For language modeling, IHA is evaluated in a 2.4B-parameter decoder-only transformer with hidden size 2560, 26 layers, \(H=20\) heads, head dim 128, RoPE with \(\theta=500{,}000\), pretraining context 8192, and 240k steps / 240B tokens [2602.21371]. On the RULER benchmark after fine-tuning at 64k, the paper reports Multi-Key Retrieval improvements over Global Attention of **+27%** at 4k, **+32%** at 8k, and **+112%** at 16k, as well as the best average EM across the full RULER suite at **44.0%**, compared with **40.6%** for Global+Local, **37.2%** for Diff Transformer, and **35.0%** for Global Attention [2602.21371].

On reasoning benchmarks, the same paper reports that before supervised fine-tuning, IHA beats Global Attention on GSM8K and MATH-500 in 5-shot evaluation, with **GSM8K: 8.34% EM, 8.42% Maj@5** and **MATH-500: 3.54% EM** [2602.21371]. After supervised fine-tuning on OpenThoughts, it reports **GSM8K Maj@16: 54.2% vs 48.4%**, a **+5.8%** improvement, and **MATH-500 Maj@16: 18.4% vs 15.6%**, a **+2.8%** improvement [2602.21371].

These results indicate that intertwining has been pursued for different downstream reasons: efficient large-scale recognition in vision and improved long-context retrieval and reasoning in language models.

## 7. Limitations, trade-offs, and terminological context

Neither paper presents cross-head interaction as cost-free. In iMHSA, naive head interaction on full attention maps is explicitly described as computationally prohibitive, which motivates decomposition through average-pooled landmarks and reverse-order attention computation [2402.17507]. The method also assumes that attention maps can be meaningfully treated as head-wise features and that the landmark downsampling preserves enough useful information.

In IHA, interleaving expands length from \(N\) to \(NP\), so global attention becomes
\[
O(P^2N^2d),
\]
and the paper therefore uses a hybrid schedule with 4 sliding-window IHA layers and 1 global layer in a 4:1 ratio, with
\[
W=\frac{N}{2P^2},
\]
to keep average compute comparable [2602.21371]. It also notes that the strongest parameter-separation results are for controlled synthetic tasks rather than direct proofs of downstream gains. Additional mixing and collapse parameters, totaling \(4H^2P\), are explicitly acknowledged [2602.21371].

A common misconception would be to treat InterTwining Attention as merely another post-attention fusion or output-projection trick. The defining characteristic in the cited works is earlier coupling: attention heads are connected **during** the formation of relational structure, either through attention-map mixing or through pre-attention mixing of queries, keys, and values [2402.17507, 2602.21371].

A second point of clarification concerns the term “intertwining” itself. In probability theory, “intertwining” denotes relations such as
\[
P_t K = K Q_t
\]
between Markov semigroups, often connected to duality and factorial measures in particle systems [2112.11885]. That literature is mathematically unrelated to neural attention. The overlap is terminological, not methodological.

Within neural sequence modeling and vision transformers, InterTwining Attention therefore refers most precisely to architectures that replace independent-head computation with explicit, learnable cross-head coupling, motivated by the claim that head interaction can improve information flow, increase feature diversity, and enlarge the set of realizable attention patterns [2402.17507, 2602.21371].

Source: https://www.emergentmind.com/topics/intertwining-attention