InterTwining Attention in Transformers

• InterTwining Attention is an architectural principle that enables explicit cross-head interaction to overcome the isolation in standard multi-head attention.
• It employs methods like iMHSA and IHA to mix attention maps or input streams, thereby increasing representational capacity and diversity.
• This approach enhances performance in vision and language tasks by improving information flow, compositional reasoning, and efficient model scaling.

Searching arXiv for the cited attention papers and closely related terminology. InterTwining Attention denotes a class of attention mechanisms that relax the standard independence of attention heads in Multi-Head Attention (MHA) or Multi-Head Self-Attention (MHSA) by introducing explicit cross-head interaction during attention computation. In this usage, “intertwining” refers to learned coupling among head-wise attention representations rather than mere post hoc concatenation of head outputs. Two concrete realizations in the arXiv literature are interactive Multi-Head Self-Attention (iMHSA), which mixes head-wise attention maps under a decomposed linear-complexity formulation (Kang et al., 2024), and Interleaved Head Attention (IHA), which constructs pseudo-heads as learned linear combinations of the original heads and thereby induces many more attention patterns than standard MHA (Duvvuri et al., 24 Feb 2026). A separate use of “intertwining” appears in probability theory, where it describes semigroup relations for Markov processes rather than neural attention; that usage is conceptually distinct despite the shared term (Floreani et al., 2021).

1. Definition and conceptual scope

In standard MHSA, each head computes its own attention map independently and the head outputs are simply concatenated. The core claim made by the attention papers is that this architecture leaves no information exchange between heads during attention computation, which can limit information flow, head diversity, and compositional behavior (Kang et al., 2024, Duvvuri et al., 24 Feb 2026). InterTwining Attention addresses that limitation by allowing one head’s attention representation to influence another head’s representation before the final value aggregation.

The concept is not a single canonical architecture. In the current literature, it encompasses at least two distinct design philosophies. The first performs head-to-head mixing directly on attention-map representations, treating attention matrices as features to be connected across the head dimension (Kang et al., 2024). The second mixes queries, keys, and values across heads before the attention operator is applied, creating pseudo-heads whose pairwise interactions expand the set of attainable attention patterns (Duvvuri et al., 24 Feb 2026).

This suggests that “InterTwining Attention” is best understood as an architectural principle rather than a fixed operator: heads are no longer isolated computational channels, but coupled components in a shared relational computation.

2. The limitation of independent-head attention

The motivation common to both attention papers is structural. Standard MHA produces exactly $H$ independent attention matrices from $H$ heads, and those heads do not communicate during attention computation (Duvvuri et al., 24 Feb 2026). In the vision setting, this independence is described as causing attention heads to become redundant and to exhibit performance saturation as head count increases (Kang et al., 2024). In the language-model setting, the same isolation is presented as a bottleneck for multi-step or compositional reasoning, where correct outputs may require combining multiple latent token-to-token relations over a chain of intermediate inferences (Duvvuri et al., 24 Feb 2026).

The standard self-attention formulation given for MHSA in the vision paper is

$\vb{z} = \{\vb*{z}_{i}|\vb*{z}_{i} \in \mathbb{R}^{c}\text{, and } 1 \leq i \leq N\},$

$\mathcal{Q}, \mathcal{K}, \mathcal{V} = (W_{Q}, W_{K}, W_{V})\vb{z},$

with per-head output

$\text{%%%%0%%%%, where %%%%1%%%%}.$

The resulting head outputs are then concatenated. Operationally, each head computes an $N\times N$ attention matrix and uses it independently (Kang et al., 2024).

The language-model paper formalizes the same isolation in standard MHA through

$$\bm Q^{(h)}=\bm X\bm W_Q^{(h)},\qquad \bm K^{(h)}=\bm X\bm W_K^{(h)},\qquad \bm V^{(h)}=\bm X\bm W_V^{(h)},$$

$$\widetilde{\bm X}^{(h)} = \operatorname{softmax}\!\left(\frac{1}{\sqrt d}\bm Q^{(h)}\bm K^{(h)\top}\right)\bm V^{(h)},$$

followed by concatenation and projection (Duvvuri et al., 24 Feb 2026).

A plausible implication is that standard multi-head designs allocate representational capacity across heads, but not compositional interaction among heads at the moment when relational structure is formed.

3. Cross-head interaction as the basic intertwining mechanism

The most direct form of InterTwining Attention is explicit head-to-head mixing. In iMHSA, the attention maps from different heads are stacked and processed with learned linear layers across the head dimension: $\text{%%%%2%%%%, where %%%%3%%%%,}$ with

$W_{1}, W_{2} \in \mathbb{R}^{h \times h}.$

The stated interpretation is that each head’s attention matrix is treated as a feature, and the learned matrices $H$0 and $H$1 mix these head-features so that head $H$2’s attention can influence head $H$3’s transformed attention representation (Kang et al., 2024).

In IHA, the interaction is moved upstream. For each original head $H$4 and pseudo-index $H$5, pseudo-queries, pseudo-keys, and pseudo-values are formed as learned linear combinations of all original heads: $H$6

$H$7

$H$8

where

$H$9

These pseudo-heads are then stacked and fed through otherwise standard attention (Duvvuri et al., 24 Feb 2026).

The distinction is operationally important. iMHSA intertwines attention maps; IHA intertwines query, key, and value streams. In both cases, however, cross-head communication is inserted before the final output concatenation. That commonality is the defining feature of the broader idea.

4. Two architectural realizations

The two attention papers instantiate the intertwining principle under different complexity constraints and for different application domains.

Mechanism	Core operation	Stated complexity
iMHSA	Cross-head mixing on decomposed attention matrices	$\vb{z} = \{\vb*{z}_{i}|\vb*{z}_{i} \in \mathbb{R}^{c}\text{, and } 1 \leq i \leq N\},$0 (Kang et al., 2024)
IHA	Pseudo-head construction from learned cross-head Q/K/V mixing	Global IHA: $\vb{z} = \{\vb*{z}_{i}|\vb*{z}_{i} \in \mathbb{R}^{c}\text{, and } 1 \leq i \leq N\},$1 (Duvvuri et al., 24 Feb 2026)

In iMHSA, a straightforward interaction on full $\vb{z} = \{\vb*{z}_{i}|\vb*{z}_{i} \in \mathbb{R}^{c}\text{, and } 1 \leq i \leq N\},$2 attention matrices is said to incur

$\vb{z} = \{\vb*{z}_{i}|\vb*{z}_{i} \in \mathbb{R}^{c}\text{, and } 1 \leq i \leq N\},$3

extra computation, along with substantial memory growth because all head-wise attention maps must be stored (Kang et al., 2024). The proposed solution is to decompose attention into reduced query-side and key-side components using spatially downsampled landmarks: $\vb{z} = \{\vb*{z}_{i}|\vb*{z}_{i} \in \mathbb{R}^{c}\text{, and } 1 \leq i \leq N\},$4

$\vb{z} = \{\vb*{z}_{i}|\vb*{z}_{i} \in \mathbb{R}^{c}\text{, and } 1 \leq i \leq N\},$5

followed by

$\vb{z} = \{\vb*{z}_{i}|\vb*{z}_{i} \in \mathbb{R}^{c}\text{, and } 1 \leq i \leq N\},$6

Here $\vb{z} = \{\vb*{z}_{i}|\vb*{z}_{i} \in \mathbb{R}^{c}\text{, and } 1 \leq i \leq N\},$7, $\vb{z} = \{\vb*{z}_{i}|\vb*{z}_{i} \in \mathbb{R}^{c}\text{, and } 1 \leq i \leq N\},$8, and $\vb{z} = \{\vb*{z}_{i}|\vb*{z}_{i} \in \mathbb{R}^{c}\text{, and } 1 \leq i \leq N\},$9. The multiplication is reordered so that the full $\mathcal{Q}, \mathcal{K}, \mathcal{V} = (W_{Q}, W_{K}, W_{V})\vb{z},$0 attention map is never explicitly formed: $\mathcal{Q}, \mathcal{K}, \mathcal{V} = (W_{Q}, W_{K}, W_{V})\vb{z},$1 The stated complexity is then

$\mathcal{Q}, \mathcal{K}, \mathcal{V} = (W_{Q}, W_{K}, W_{V})\vb{z},$2

which is linear in $\mathcal{Q}, \mathcal{K}, \mathcal{V} = (W_{Q}, W_{K}, W_{V})\vb{z},$3 for fixed $\mathcal{Q}, \mathcal{K}, \mathcal{V} = (W_{Q}, W_{K}, W_{V})\vb{z},$4, $\mathcal{Q}, \mathcal{K}, \mathcal{V} = (W_{Q}, W_{K}, W_{V})\vb{z},$5, and $\mathcal{Q}, \mathcal{K}, \mathcal{V} = (W_{Q}, W_{K}, W_{V})\vb{z},$6 (Kang et al., 2024).

In IHA, the objective is not linear-complexity self-attention but a richer head interaction pattern within the standard attention operator. Because each original head constructs $\mathcal{Q}, \mathcal{K}, \mathcal{V} = (W_{Q}, W_{K}, W_{V})\vb{z},$7 pseudo-queries and $\mathcal{Q}, \mathcal{K}, \mathcal{V} = (W_{Q}, W_{K}, W_{V})\vb{z},$8 pseudo-keys, a single head can induce up to

$\mathcal{Q}, \mathcal{K}, \mathcal{V} = (W_{Q}, W_{K}, W_{V})\vb{z},$9

distinct attention patterns, typically with $\text{%%%%0%%%%, where %%%%1%%%%}.$0 (Duvvuri et al., 24 Feb 2026). The pseudo-output collapse is performed by

$\text{%%%%0%%%%, where %%%%1%%%%}.$1

and the paper states an added parameter cost of

$\text{%%%%0%%%%, where %%%%1%%%%}.$2

beyond the base $\text{%%%%0%%%%, where %%%%1%%%%}.$3 parameters of MHA (Duvvuri et al., 24 Feb 2026).

These designs reflect two different trade-offs. iMHSA seeks efficient cross-head interaction under large token counts in vision. IHA seeks greater pattern capacity and improved compositional reasoning in LLMs, while explicitly acknowledging increased global attention cost.

5. Expressivity, scaling behavior, and theoretical claims

The most explicit expressivity results appear in the IHA paper. It proves

$\text{%%%%0%%%%, where %%%%1%%%%}.$4

where $\text{%%%%0%%%%, where %%%%1%%%%}.$5 is the set of MHA modules and $\text{%%%%0%%%%, where %%%%1%%%%}.$6 is the corresponding IHA family (Duvvuri et al., 24 Feb 2026). Containment is obtained by choosing mixing tensors that effectively recover MHA, while strictness is argued through cases where IHA implements nonlinear behavior on repeated-token inputs for which MHA remains linear.

The same paper further analyzes synthetic tasks in which MHA’s “one head = one pattern” structure becomes a formal limitation. For representing

$\text{%%%%0%%%%, where %%%%1%%%%}.$7

it states that MHA needs $\text{%%%%0%%%%, where %%%%1%%%%}.$8 heads, whereas IHA needs only $\text{%%%%0%%%%, where %%%%1%%%%}.$9 heads with $N\times N$0 (Duvvuri et al., 24 Feb 2026). The key factorization is

$N\times N$1

The reported dominant parameter scaling improves from

$N\times N$2

For the synthetic Count Permutation Match-3 task, the same quadratic expansion of representable patterns yields another head-efficiency separation. The paper states that MHA needs $N\times N$3 heads in the worst case, whereas IHA uses

$N\times N$4

heads with $N\times N$5, giving up to $N\times N$6 shift patterns from only $N\times N$7 heads (Duvvuri et al., 24 Feb 2026).

The iMHSA paper does not frame its contribution in the same formal expressivity language, but its analysis attributes improved behavior to richer information flow and increased feature diversity across heads. It reports that, without interaction, performance saturates quickly as heads increase, while with interaction, accuracy improves more consistently with more heads; it also states that attention maps become more diverse, with higher variance and lower similarity across heads (Kang et al., 2024).

Taken together, these results support a common interpretation: head isolation constrains either representational diversity or the efficient realization of multiple relational patterns, and cross-head coupling is a direct attempt to remove that constraint.

6. Empirical performance and application domains

The empirical literature represented here spans both vision and language modeling.

For vision, the iMHSA paper states that interactive Multi-Head Self-Attention (iMHSA) and the resulting interactive Vision Transformer (iViT) outperform prior efficient attention methods on ImageNet under standard token size, larger token size, and high-resolution inputs, specifically naming Performer, Local attention, Scatterbrain, Nyströmformer, LARA, Combiner, Long-Short, and EVA (Kang et al., 2024). It also reports strong performance of iViT on ImageNet classification, COCO detection/instance segmentation, and ADE20K semantic segmentation, with the stated trend of better accuracy, good speed/FLOPs trade-off, and often fewer parameters than competing models (Kang et al., 2024). At high resolutions, the paper notes that softmax attention can run out of memory, whereas iMHSA remains feasible.

For language modeling, IHA is evaluated in a 2.4B-parameter decoder-only transformer with hidden size 2560, 26 layers, $N\times N$8 heads, head dim 128, RoPE with $N\times N$9, pretraining context 8192, and 240k steps / 240B tokens (Duvvuri et al., 24 Feb 2026). On the RULER benchmark after fine-tuning at 64k, the paper reports Multi-Key Retrieval improvements over Global Attention of +27% at 4k, +32% at 8k, and +112% at 16k, as well as the best average EM across the full RULER suite at 44.0%, compared with 40.6% for Global+Local, 37.2% for Diff Transformer, and 35.0% for Global Attention (Duvvuri et al., 24 Feb 2026).

On reasoning benchmarks, the same paper reports that before supervised fine-tuning, IHA beats Global Attention on GSM8K and MATH-500 in 5-shot evaluation, with GSM8K: 8.34% EM, 8.42% Maj@5 and MATH-500: 3.54% EM (Duvvuri et al., 24 Feb 2026). After supervised fine-tuning on OpenThoughts, it reports GSM8K Maj@16: 54.2% vs 48.4%, a +5.8% improvement, and MATH-500 Maj@16: 18.4% vs 15.6%, a +2.8% improvement (Duvvuri et al., 24 Feb 2026).

These results indicate that intertwining has been pursued for different downstream reasons: efficient large-scale recognition in vision and improved long-context retrieval and reasoning in LLMs.

7. Limitations, trade-offs, and terminological context

Neither paper presents cross-head interaction as cost-free. In iMHSA, naive head interaction on full attention maps is explicitly described as computationally prohibitive, which motivates decomposition through average-pooled landmarks and reverse-order attention computation (Kang et al., 2024). The method also assumes that attention maps can be meaningfully treated as head-wise features and that the landmark downsampling preserves enough useful information.

In IHA, interleaving expands length from $$\bm Q^{(h)}=\bm X\bm W_Q^{(h)},\qquad \bm K^{(h)}=\bm X\bm W_K^{(h)},\qquad \bm V^{(h)}=\bm X\bm W_V^{(h)},$$0 to $$\bm Q^{(h)}=\bm X\bm W_Q^{(h)},\qquad \bm K^{(h)}=\bm X\bm W_K^{(h)},\qquad \bm V^{(h)}=\bm X\bm W_V^{(h)},$$1, so global attention becomes

$$\bm Q^{(h)}=\bm X\bm W_Q^{(h)},\qquad \bm K^{(h)}=\bm X\bm W_K^{(h)},\qquad \bm V^{(h)}=\bm X\bm W_V^{(h)},$$2

and the paper therefore uses a hybrid schedule with 4 sliding-window IHA layers and 1 global layer in a 4:1 ratio, with

$$\bm Q^{(h)}=\bm X\bm W_Q^{(h)},\qquad \bm K^{(h)}=\bm X\bm W_K^{(h)},\qquad \bm V^{(h)}=\bm X\bm W_V^{(h)},$$3

to keep average compute comparable (Duvvuri et al., 24 Feb 2026). It also notes that the strongest parameter-separation results are for controlled synthetic tasks rather than direct proofs of downstream gains. Additional mixing and collapse parameters, totaling $$\bm Q^{(h)}=\bm X\bm W_Q^{(h)},\qquad \bm K^{(h)}=\bm X\bm W_K^{(h)},\qquad \bm V^{(h)}=\bm X\bm W_V^{(h)},$$4, are explicitly acknowledged (Duvvuri et al., 24 Feb 2026).

A common misconception would be to treat InterTwining Attention as merely another post-attention fusion or output-projection trick. The defining characteristic in the cited works is earlier coupling: attention heads are connected during the formation of relational structure, either through attention-map mixing or through pre-attention mixing of queries, keys, and values (Kang et al., 2024, Duvvuri et al., 24 Feb 2026).

A second point of clarification concerns the term “intertwining” itself. In probability theory, “intertwining” denotes relations such as

$$\bm Q^{(h)}=\bm X\bm W_Q^{(h)},\qquad \bm K^{(h)}=\bm X\bm W_K^{(h)},\qquad \bm V^{(h)}=\bm X\bm W_V^{(h)},$$5

between Markov semigroups, often connected to duality and factorial measures in particle systems (Floreani et al., 2021). That literature is mathematically unrelated to neural attention. The overlap is terminological, not methodological.

Within neural sequence modeling and vision transformers, InterTwining Attention therefore refers most precisely to architectures that replace independent-head computation with explicit, learnable cross-head coupling, motivated by the claim that head interaction can improve information flow, increase feature diversity, and enlarge the set of realizable attention patterns (Kang et al., 2024, Duvvuri et al., 24 Feb 2026).