External Attention Mechanisms

• External attention is a family of mechanisms that leverages shared learnable memories or external priors to enhance context and reduce computational cost.
• It replaces quadratic self-attention with linear-time memory lookups, effectively lowering complexity from O(N²) to O(Nk) across various applications.
• Applications span computer vision, time-series forecasting, graph learning, and ASR, demonstrating improved efficiency and task-specific performance.

External attention denotes a family of attention mechanisms in which the weighting or modulation applied to a representation is not computed solely from pairwise relations within a single input. In the literature, the term covers several related constructions: shared learnable memory banks that replace or augment self-attention, graph-level external key–value units shared across samples, auxiliary branches that generate spatial attention maps, retrieval-based knowledge memories, and externally controlled gating. The canonical formulation replaces the $N\times N$ affinity matrix of self-attention with attention over a small set of external slots, typically yielding linear rather than quadratic complexity in sequence length while introducing dataset-level shared parameters that can encode recurring patterns across samples (Guo et al., 2021).

1. Terminological scope and major variants

The surveyed literature uses “external attention” for mechanisms that expose a model to information not confined to within-sample token–token interactions. In some works, the “external” object is a pair of learnable memories shared across the entire dataset; in others it is an auxiliary dataset, retrieved textual knowledge, or another subject’s features. This breadth is important because papers sharing the same label do not implement identical operators (Guo et al., 2021).

Form	External source	Representative papers
Shared memory-slot attention	Learnable memories shared across samples	(Guo et al., 2021, Zamanvaziri et al., 30 Apr 2026, Iqbal et al., 2024)
Graph external attention	External node/edge key-value units shared across graphs	(Liang et al., 2024)
Auxiliary-prior attention	External spleen masks or anomaly maps	(Zhou et al., 2022, Katafuchi et al., 2021)
Retrieval-based external attention	ConceptNet, Wiktionary, retrieved QA pairs	(Xu et al., 2021)
Cross-person external attention	Active joints from another person	(Pang et al., 5 Jul 2025)
Externally controlled gating	Task-select signal controlling gates	(Son et al., 2018)

A recurrent theme is that external attention is introduced where purely local or sample-specific attention is considered insufficient. Graph External Attention is motivated by the claim that self-attention over one graph misses inter-graph correlations (Liang et al., 2024). KEAR introduces external knowledge and context through retrieved memory tokens (Xu et al., 2021). In splenic vascular injury segmentation, “external attention” is a spatial prior mined from a separate spleen-labeled dataset rather than an internal feature-map attention block (Zhou et al., 2022). This suggests that the defining property is less a single equation than a modeling stance: attention is informed by a source external to the immediate self-attending set.

2. Canonical memory-bank formulation

The canonical modern formulation is given in “Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks” (Guo et al., 2021). For input $X\in\mathbb{R}^{N\times d}$, external attention introduces two memories, $M_1\in\mathbb{R}^{k\times d}$ and $M_2\in\mathbb{R}^{d\times k}$, and computes

$Z = X M_1^\top \in \mathbb{R}^{N\times k},$

followed by a row-wise softmax and a column-wise $\ell_1$ normalization,

$$\tilde A_{ij} = \exp(Z_{ij})\Big/\sum_{p=1}^{k}\exp(Z_{ip}), \qquad A_{ij} = \tilde A_{ij}\Big/\sum_{q=1}^{N}\tilde A_{qj},$$

and then

$Y = A M_2^\top \in \mathbb{R}^{N\times d}.$

In implementation terms, the mechanism is “two cascaded linear layers and two normalization layers” (Guo et al., 2021).

Subsequent work preserves the same memory-bank idea while altering notation and placement. ITS-Mina defines a refined representation $H\in\mathbb{R}^{L\times C}$, computes affinities $\hat A = H E^\top$ with memory $X\in\mathbb{R}^{N\times d}$0, applies a row-wise softmax over slots, reconstructs a residual $X\in\mathbb{R}^{N\times d}$1 with $X\in\mathbb{R}^{N\times d}$2, and outputs $X\in\mathbb{R}^{N\times d}$3 (Zamanvaziri et al., 30 Apr 2026). EAViT uses key and value memories $X\in\mathbb{R}^{N\times d}$4 and a multi-head external attention operator

$X\in\mathbb{R}^{N\times d}$5

with shared memories across heads (Iqbal et al., 2024). In Graph External Attention, queries are derived from node features but keys and values are external learnable units, giving

$X\in\mathbb{R}^{N\times d}$6

for each head $X\in\mathbb{R}^{N\times d}$7 (Liang et al., 2024).

The common structural pattern is that queries remain input-dependent, whereas keys and values are replaced by a small external basis that is learned globally rather than instantiated afresh for each sample.

3. Relation to self-attention, complexity, and statistical role

The principal formal contrast is between pairwise self-attention over $X\in\mathbb{R}^{N\times d}$8 positions and attention over $X\in\mathbb{R}^{N\times d}$9 or $M_1\in\mathbb{R}^{k\times d}$0 shared slots. In the canonical visual formulation, external attention has complexity $M_1\in\mathbb{R}^{k\times d}$1 rather than the $M_1\in\mathbb{R}^{k\times d}$2 cost of vanilla self-attention (Guo et al., 2021). ITS-Mina states the two dominant products are $M_1\in\mathbb{R}^{k\times d}$3 and $M_1\in\mathbb{R}^{k\times d}$4, each with cost $M_1\in\mathbb{R}^{k\times d}$5, so the total is linear in lookback length $M_1\in\mathbb{R}^{k\times d}$6 when $M_1\in\mathbb{R}^{k\times d}$7 and $M_1\in\mathbb{R}^{k\times d}$8 remain fixed (Zamanvaziri et al., 30 Apr 2026). Graph External Attention reduces per-head complexity from $M_1\in\mathbb{R}^{k\times d}$9 to $M_2\in\mathbb{R}^{d\times k}$0 and memory from $M_2\in\mathbb{R}^{d\times k}$1 to $M_2\in\mathbb{R}^{d\times k}$2 when $M_2\in\mathbb{R}^{d\times k}$3 (Liang et al., 2024). EAViT presents the same comparison as $M_2\in\mathbb{R}^{d\times k}$4 versus $M_2\in\mathbb{R}^{d\times k}$5, with $M_2\in\mathbb{R}^{d\times k}$6 versus $M_2\in\mathbb{R}^{d\times k}$7 complexity (Iqbal et al., 2024).

The statistical interpretation of the memory is equally central. The memories in the original external-attention paper are shared across every sample and every spatial position, so they “effectively accumulate statistics and ‘prototype’ vectors from the entire training set” (Guo et al., 2021). ITS-Mina emphasizes that because $M_2\in\mathbb{R}^{d\times k}$8 and $M_2\in\mathbb{R}^{d\times k}$9 are shared across all windows, they learn dataset-level prototypes rather than overfitting to a single sample (Zamanvaziri et al., 30 Apr 2026). Graph External Attention frames its units as a common memory bank of patterns that implicitly captures inter-graph correlations (Liang et al., 2024). EA-GCL makes the same point in recommendation language: the MLP memory-sharing structure is intended to alleviate the bias interference from the batch-based training scheme by storing global co-occurrence patterns in shared parameters (Wang et al., 2023).

A frequent misconception is that external attention is merely a cheaper approximation to self-attention. The surveyed papers consistently assign it a second role: not only reducing cost, but also introducing a cross-sample inductive bias through shared external memories.

4. Architectural realizations across application domains

In vision, external attention was introduced as a drop-in replacement for self-attention and then extended to an all-MLP architecture, EAMLP. Reported results span image classification, object detection, semantic segmentation, instance segmentation, image generation, and point cloud analysis; for an input of size $Z = X M_1^\top \in \mathbb{R}^{N\times k},$0, the external-attention block uses $Z = X M_1^\top \in \mathbb{R}^{N\times k},$1 M parameters and $Z = X M_1^\top \in \mathbb{R}^{N\times k},$2 G MACs versus $Z = X M_1^\top \in \mathbb{R}^{N\times k},$3 M and $Z = X M_1^\top \in \mathbb{R}^{N\times k},$4 G for vanilla self-attention (Guo et al., 2021).

In multivariate time-series forecasting, ITS-Mina places external attention after iterative mixer refinement. The module projects an $Z = X M_1^\top \in \mathbb{R}^{N\times k},$5 tensor into and out of a memory bank of $Z = X M_1^\top \in \mathbb{R}^{N\times k},$6 learnable slots and adds the resulting residual correction back to the temporal representation. On six benchmark datasets, the model is reported as state-of-the-art or highly competitive against eleven baselines (Zamanvaziri et al., 30 Apr 2026).

In graph representation learning, GEAET interleaves three streams in each layer: local structure encoding via a GNN, global self-attention, and Graph External Attention. The outputs are combined as

$Z = X M_1^\top \in \mathbb{R}^{N\times k},$7

followed by the usual feed-forward sublayer. Reported empirical findings include a ZINC MAE drop from approximately $Z = X M_1^\top \in \mathbb{R}^{N\times k},$8 to approximately $Z = X M_1^\top \in \mathbb{R}^{N\times k},$9, PATTERN/CLUSTER accuracy around $\ell_1$0, and PCQM-Contact MRR improvement from $\ell_1$1 to $\ell_1$2 (Liang et al., 2024).

In audio classification, EAViT inserts multi-head external attention into a ViT encoder operating on $\ell_1$3 spectrogram images with $\ell_1$4 patches, $\ell_1$5 layers, embedding dimension $\ell_1$6, and $\ell_1$7 heads. On GTZAN, EAViT reports $\ell_1$8 overall accuracy versus $\ell_1$9 for the vanilla ViT baseline, a gain of approximately $$\tilde A_{ij} = \exp(Z_{ij})\Big/\sum_{p=1}^{k}\exp(Z_{ip}), \qquad A_{ij} = \tilde A_{ij}\Big/\sum_{q=1}^{N}\tilde A_{qj},$$0 percentage points (Iqbal et al., 2024).

In online network traffic classification, ECM applies external attention to embedded 12-byte IP packet headers and then uses 1D convolution over the attended output. On ISCX and BUPD, ECM reports $$\tilde A_{ij} = \exp(Z_{ij})\Big/\sum_{p=1}^{k}\exp(Z_{ip}), \qquad A_{ij} = \tilde A_{ij}\Big/\sum_{q=1}^{N}\tilde A_{qj},$$1 and $$\tilde A_{ij} = \exp(Z_{ij})\Big/\sum_{p=1}^{k}\exp(Z_{ip}), \qquad A_{ij} = \tilde A_{ij}\Big/\sum_{q=1}^{N}\tilde A_{qj},$$2 accuracy, respectively, with per-packet inference times of $$\tilde A_{ij} = \exp(Z_{ij})\Big/\sum_{p=1}^{k}\exp(Z_{ip}), \qquad A_{ij} = \tilde A_{ij}\Big/\sum_{q=1}^{N}\tilde A_{qj},$$3 ms and $$\tilde A_{ij} = \exp(Z_{ij})\Big/\sum_{p=1}^{k}\exp(Z_{ip}), \qquad A_{ij} = \tilde A_{ij}\Big/\sum_{q=1}^{N}\tilde A_{qj},$$4 ms (Hu et al., 2023).

In contextual spelling correction for E2E neural transducer-based ASR, an improved non-autoregressive model incorporates acoustics information with an external attention as well as text hypotheses, together with semantic aware data augmentation. The abstract reports as much as $$\tilde A_{ij} = \exp(Z_{ij})\Big/\sum_{p=1}^{k}\exp(Z_{ip}), \qquad A_{ij} = \tilde A_{ij}\Big/\sum_{q=1}^{N}\tilde A_{qj},$$5 relative name recall gain over the baseline ASR+Biasing system and stable improvement over different bias list name coverage ratio (Wang et al., 2023).

5. External attention as auxiliary prior, retrieval, or cross-entity interaction

Not all external-attention mechanisms use shared slot memories. In multi-phase splenic vascular injury segmentation, external attention is a pseudo-spleen spatial mask mined from an auxiliary dataset of 30 normal contrast-enhanced abdominal CTs with spleen labels. The attention map is created by applying an external spleen-segmentation model, dilating its soft mask with a Gaussian kernel $$\tilde A_{ij} = \exp(Z_{ij})\Big/\sum_{p=1}^{k}\exp(Z_{ip}), \qquad A_{ij} = \tilde A_{ij}\Big/\sum_{q=1}^{N}\tilde A_{qj},$$6 with $$\tilde A_{ij} = \exp(Z_{ij})\Big/\sum_{p=1}^{k}\exp(Z_{ip}), \qquad A_{ij} = \tilde A_{ij}\Big/\sum_{q=1}^{N}\tilde A_{qj},$$7, thresholding at $$\tilde A_{ij} = \exp(Z_{ij})\Big/\sum_{p=1}^{k}\exp(Z_{ip}), \qquad A_{ij} = \tilde A_{ij}\Big/\sum_{q=1}^{N}\tilde A_{qj},$$8, and using the resulting binary mask as a spatial weight on the loss. The paper emphasizes that external attention “is only applied at training time as a spatial weight on the loss” and reports more than $$\tilde A_{ij} = \exp(Z_{ij})\Big/\sum_{p=1}^{k}\exp(Z_{ip}), \qquad A_{ij} = \tilde A_{ij}\Big/\sum_{q=1}^{N}\tilde A_{qj},$$9 average DSC improvement over DeepLab-v3+ (Zhou et al., 2022).

LEA-Net also uses an auxiliary branch, but here the external signal is an anomaly map generated by unsupervised color reconstruction. A Color Anomaly Attention Network translates the anomaly map into multi-resolution attention maps $Y = A M_2^\top \in \mathbb{R}^{N\times d}.$0, and the anomaly detector updates an intermediate feature map by

$Y = A M_2^\top \in \mathbb{R}^{N\times d}.$1

On five datasets, a ResNet18 base model improves from $Y = A M_2^\top \in \mathbb{R}^{N\times d}.$2 to $Y = A M_2^\top \in \mathbb{R}^{N\times d}.$3 $Y = A M_2^\top \in \mathbb{R}^{N\times d}.$4 on Potato when layer-wise external attention is applied (Katafuchi et al., 2021).

KEAR uses still another meaning of external attention. It retrieves a ConceptNet triple, Wiktionary definitions, and the top $Y = A M_2^\top \in \mathbb{R}^{N\times d}.$5 most similar training question–answer pairs, concatenates them into a knowledge string $Y = A M_2^\top \in \mathbb{R}^{N\times d}.$6, embeds the resulting memory tokens $Y = A M_2^\top \in \mathbb{R}^{N\times d}.$7, and appends them to the question–choice embeddings so that standard transformer self-attention operates over $Y = A M_2^\top \in \mathbb{R}^{N\times d}.$8. The reported result is $Y = A M_2^\top \in \mathbb{R}^{N\times d}.$9 test accuracy on CommonsenseQA, exceeding the cited human accuracy of $H\in\mathbb{R}^{L\times C}$0 (Xu et al., 2021).

ASEA, for human interaction recognition, uses “external attention” to mean cross-person attention restricted to active joints selected by AT-NAC. Within each frame, Person 1 attends to Person 2’s active joints and vice versa: $H\in\mathbb{R}^{L\times C}$1 On NTU-26, the paper reports $H\in\mathbb{R}^{L\times C}$2 for the baseline, $H\in\mathbb{R}^{L\times C}$3 for baseline plus EA, and $H\in\mathbb{R}^{L\times C}$4 for full ASEA (Pang et al., 5 Jul 2025).

These examples show that the term may refer either to an external memory used during forward inference or to externally sourced priors that modulate learning or representation.

6. Benefits, limitations, and points of interpretation

Across domains, the main reported benefits are lower computational complexity, parameter efficiency, and access to global or auxiliary context not available to sample-internal self-attention. External attention is often described as a regularizer because the same memories or external units are reused across samples (Guo et al., 2021). This suggests why the mechanism appears in problems characterized by limited data, asymmetric batch statistics, sparse domains, or weak local evidence.

The limitations are variant-specific. GEAET retains both self-attention and GEA, which indicates that external units do not automatically subsume intra-graph interactions (Liang et al., 2024). ASEA explicitly notes that its EA ignores intra-person attention and relies on the upstream GCN encoder for within-person context (Pang et al., 5 Jul 2025). The splenic vascular injury framework notes that its attention map is fixed at training time and not refined at test time, and that multi-site, multi-vendor trauma CT remains untested (Zhou et al., 2022). KEAR’s formulation inherits the length constraints and retrieval dependence of concatenation-based transformers (Xu et al., 2021).

A second misconception is terminological. “External attention” is not synonymous with “memory-bank attention.” ExGate uses externally controlled sigmoid gates selected by a one-hot task vector of dimension $H\in\mathbb{R}^{L\times C}$5, adding only $H\in\mathbb{R}^{L\times C}$6 bias parameters to a base network of approximately $H\in\mathbb{R}^{L\times C}$7 weights and improving CIFAR-10 accuracy from $H\in\mathbb{R}^{L\times C}$8 to $H\in\mathbb{R}^{L\times C}$9 (Son et al., 2018). EMA-VIO uses external memory attention to transform $\hat A = H E^\top$0 and $\hat A = H E^\top$1 before an attention step in multimodal visual–inertial fusion (Tu et al., 2022). LEA-GCN combines a memory-based channel with a pairwise MLP channel and reports up to $\hat A = H E^\top$2–$\hat A = H E^\top$3 speedups in the sequence encoder portion (Zhang et al., 2023). The common thread is externally shared or externally supplied structure, not one fixed operator.

Taken together, the literature presents external attention as a modular alternative to purely intra-sample attention. Its most canonical form uses shared learnable memories to replace quadratic self-attention with linear-time memory lookup, but the broader research program includes auxiliary spatial priors, retrieval-based textual memory, externally controlled gating, and cross-entity interaction modules. The diversity of implementations indicates that external attention is best understood as a design principle for injecting information that lies outside the immediate self-attending set, whether that information is encoded as learnable slots, retrieved context, or task-specific priors.