Self-Attention Enhancement (SAE)

• Self-Attention Enhancement (SAE) is a family of techniques that modify standard self-attention by reweighting scores, altering outputs, or inserting additional operators to capture local and long-range dependencies.
• SAE methods include score shaping with Gaussian masks, output correction like exclusive attention, and residualized insertion, which together enhance performance in speech, vision, and language tasks.
• Empirical findings show that SAE mechanisms balance local context and global information while reducing computational complexity, making them effective in scenarios with high resolution or long sequences.

Self-Attention Enhancement (SAE) denotes a family of modifications that augment, constrain, or reweight self-attention so that sequence or spatial models capture dependencies that a baseline architecture handles poorly. In the cited literature, SAE appears in several forms: Gaussian-weighted self-attention for speech enhancement, non-local residual blocks inserted into SEGAN, self-attentive temporal convolutional stages, sparse or linear-complexity attention for long utterances, locally enhanced attention in vision, exclusive attention that removes a token’s self-value component, gated differential attention for robustness, chunked self-attention encoders for online ASR, and self-attention reweighting in diffusion U-Nets for anomaly generation (Kim et al., 2019, Phan et al., 2020, Lin et al., 2021, Zhang et al., 2023, Koizumi et al., 2021, Yang et al., 2021, Zhai, 10 Mar 2026, Lygizou et al., 29 May 2025, Miao et al., 2020, Zuo et al., 15 Aug 2025). The common objective is not a single canonical operator, but the deliberate reshaping of attention so that locality, long-range context, causality, efficiency, or robustness are encoded more explicitly than in vanilla scaled dot-product attention.

1. Scope and conceptual taxonomy

The literature uses SAE in both narrow and broad senses. In a narrow sense, a paper may introduce a specific self-attention modification and explicitly present it as the enhancement mechanism of a given system, such as the Gaussian mask in T-GSA, the non-local block in SASEGAN, the temporal-frequency-channel branches in OFIF-Net, or the anomaly-aware reweighting mask in diffusion self-attention (Kim et al., 2019, Phan et al., 2020, Zhang et al., 21 Jan 2025, Zuo et al., 15 Aug 2025). In a broader sense, SAE functions as an umbrella label for inserting or redesigning self-attention inside CNN, RNN, Conformer, Transformer, or diffusion backbones so that the resulting model better balances local structure and non-local context (Zhang et al., 2022, Yang et al., 2021, Pandey et al., 2020).

A recurring taxonomy in the cited work separates SAE mechanisms by where the modification enters the computation. Some methods alter the attention scores themselves, for example by multiplying scores with a Gaussian distance prior, a sparse ripple mask, or an anomaly-aware reweighting mask (Kim et al., 2019, Zhang et al., 2023, Zuo et al., 15 Aug 2025). Some alter the attention output, as in Exclusive Self Attention, which subtracts the projection of the output onto the token’s own value vector (Zhai, 10 Mar 2026). Some add parallel local operators, such as grouped convolutions in LESA or convolutional modulation plus local frequency attention and TCN blocks in Spectrum Attention Fusion (Yang et al., 2021, Long et al., 2023). Others replace dense attention with efficient approximations, including Linformer, Longformer, FAVOR+, chunked attention, or causal multi-branch attention (Zhang et al., 2022, Koizumi et al., 2021, Miao et al., 2020, Zhang et al., 21 Jan 2025).

Domain	SAE form	Representative papers
Speech enhancement	Gaussian weighting, non-local residual attention, sparse masks, linear attention, causal TFCA	(Kim et al., 2019, Phan et al., 2020, Zhang et al., 2023, Koizumi et al., 2021, Zhang et al., 21 Jan 2025)
Vision and language	Efficient hybrid attention, local-context fusion, orthogonalized or gated attention	(Zhang et al., 2022, Yang et al., 2021, Zhai, 10 Mar 2026, Lygizou et al., 29 May 2025)
Streaming or generative systems	Chunked online encoders, mask-guided diffusion reweighting	(Miao et al., 2020, Zuo et al., 15 Aug 2025)

This suggests that SAE is best understood as a design space rather than a single algorithm. What unifies the space is the claim that standard self-attention, used without additional inductive bias, is often mismatched to the correlation structure, latency budget, or robustness requirements of the target task.

2. Mathematical design patterns

The baseline object throughout the literature is scaled dot-product attention,

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V,$$

with the SAE mechanism changing either the score matrix, the value aggregation, or the post-attention fusion (Kim et al., 2019, Zhai, 10 Mar 2026, Lygizou et al., 29 May 2025).

One major pattern is score shaping by priors or masks. T-GSA introduces a trainable Gaussian weight

$$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$

forms

$S = G \circ (QK^\top/\sqrt{d_k}),$

then applies

$A=\mathrm{softmax}(|S|),\qquad O=AV,$

so that temporal distance explicitly attenuates attention while absolute-value softmax preserves the influence of negative correlations (Kim et al., 2019). Ripple sparse self-attention likewise modifies the admissible score pattern through a binary mask that combines a local window of size $w$ with dilated positions spaced by factor $d$, yielding total complexity

$\mathcal{O}\bigl(Tw + T(T-w)/d\bigr)$

instead of full $\mathcal{O}(T^2)$ attention (Zhang et al., 2023). In diffusion anomaly generation, SAE constructs a reweighting mask $M^s$ so that anomaly queries up-weight normal keys by a factor $\beta$ and set anomaly-to-anomaly scores to zero, then uses

$$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$0

before softmax (Zuo et al., 15 Aug 2025).

A second pattern is output correction or orthogonalization. Exclusive Self Attention computes standard multi-head outputs $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$1 and then removes, for each token, the component aligned with its own value vector:

$$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$2

By construction, $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$3, and the paper interprets this as enforcing a division of labor between contextual aggregation and the residual-plus-FFN pathway (Zhai, 10 Mar 2026). Differential Gated Self-Attention instead computes excitatory and inhibitory attention maps $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$4 and $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$5 and fuses them with a token- and head-dependent sigmoid gate $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$6,

$$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$7

to implement context-aware suppression of attention noise (Lygizou et al., 29 May 2025).

A third pattern is residualized insertion with learnable scalar stabilization. SASEGAN computes self-attended features

$$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$8

with $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$9 initialized to zero so that training begins as standard SEGAN (Phan et al., 2020). The SA block in multi-stage SA-TCN uses the same style of residual scaling,

$S = G \circ (QK^\top/\sqrt{d_k}),$0

with $S = G \circ (QK^\top/\sqrt{d_k}),$1 initialized to $S = G \circ (QK^\top/\sqrt{d_k}),$2 (Lin et al., 2021). In both cases the scalar gate is presented as a stabilizer of gradient flow at the start of training.

A fourth pattern is explicit decomposition into local and context terms. LESA interprets self-attention through a CRF-like unary/binary split and replaces the weak vanilla unary contribution with a grouped convolutional descriptor $S = G \circ (QK^\top/\sqrt{d_k}),$3, while the context term $S = G \circ (QK^\top/\sqrt{d_k}),$4 remains attention-based. A sigmoid fusion module then computes

$S = G \circ (QK^\top/\sqrt{d_k}),$5

This formulation makes local detail an explicit pathway rather than an incidental by-product of dense attention (Yang et al., 2021).

3. Speech enhancement as the principal development line

Speech enhancement is the domain in which SAE is most densely explored in the cited work. The motivations are consistent: speech frames that are temporally close tend to be far more correlated than distant ones; speech waveforms contain supra-frame periodicities and formant patterns that span many milliseconds; and quadratic attention becomes computationally difficult for long recordings (Kim et al., 2019, Phan et al., 2020, Zhang et al., 2023).

“T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancement” replaces each Transformer encoder self-attention block with Gaussian-weighted self-attention. On VoiceBank-DEMAND it reports PESQ $S = G \circ (QK^\top/\sqrt{d_k}),$6, CSIG $S = G \circ (QK^\top/\sqrt{d_k}),$7, CBAK $S = G \circ (QK^\top/\sqrt{d_k}),$8, COVL $S = G \circ (QK^\top/\sqrt{d_k}),$9, SSNR $A=\mathrm{softmax}(|S|),\qquad O=AV,$0, and SDR $A=\mathrm{softmax}(|S|),\qquad O=AV,$1, compared with CNN-LSTM at PESQ $A=\mathrm{softmax}(|S|),\qquad O=AV,$2, CSIG $A=\mathrm{softmax}(|S|),\qquad O=AV,$3, CBAK $A=\mathrm{softmax}(|S|),\qquad O=AV,$4, COVL $A=\mathrm{softmax}(|S|),\qquad O=AV,$5, SSNR $A=\mathrm{softmax}(|S|),\qquad O=AV,$6, and SDR $A=\mathrm{softmax}(|S|),\qquad O=AV,$7 (Kim et al., 2019). The same paper reports that fixed, hand-tuned $A=\mathrm{softmax}(|S|),\qquad O=AV,$8 degraded performance back toward the vanilla Transformer, while learned $A=\mathrm{softmax}(|S|),\qquad O=AV,$9 adapted to layer depth.

“Self-Attention Generative Adversarial Network for Speech Enhancement” augments the convolution-only SEGAN with a lightweight non-local module operating on raw waveforms. The study varies the layer index $w$0 and also tests attention at all eligible layers. Over 824 test utterances, baseline SEGAN obtains PESQ $w$1, CSIG $w$2, CBAK $w$3, COVL $w$4, SSNR $w$5, and STOI $w$6, whereas SASEGAN-All reaches PESQ $w$7, CSIG $w$8, CBAK $w$9, COVL $d$0, SSNR $d$1, and STOI $d$2 (Phan et al., 2020). The paper further reports no statistically significant difference between SASEGAN-$d$3 at different layers, suggesting that a single block at the highest layer can maximize memory savings.

Time-domain and mask-estimation systems adopt related ideas. Dense CNN with Self-Attention inserts attention modules after down- or up-sampling in each encoder and decoder block and reports, in a non-causal ablation, a progression from STOI $d$4, PESQ $d$5, SNR $d$6 for “Dense CNN, no context” to STOI $d$7, PESQ $d$8, SNR $d$9 for “+ self-attention” (Pandey et al., 2020). Multi-stage SA-TCN places a self-attention block before stacks of dilated TCN blocks in each stage and refines a mask across stages; on LibriSpeech, the 5-stage SA-TCN reports PESQ $\mathcal{O}\bigl(Tw + T(T-w)/d\bigr)$0, STOI $\mathcal{O}\bigl(Tw + T(T-w)/d\bigr)$1, SI-SDR $\mathcal{O}\bigl(Tw + T(T-w)/d\bigr)$2, and WER $\mathcal{O}\bigl(Tw + T(T-w)/d\bigr)$3, and the paper attributes additional gains to both the SA block and the fusion block (Lin et al., 2021).

Several later systems make SAE more structured rather than merely inserted. Ripple sparse self-attention combines local band attention with dilated attention and, at 5 dB SNR under the PSM objective, reports PESQ $\mathcal{O}\bigl(Tw + T(T-w)/d\bigr)$4 and ESTOI $\mathcal{O}\bigl(Tw + T(T-w)/d\bigr)$5, compared with full attention at PESQ $\mathcal{O}\bigl(Tw + T(T-w)/d\bigr)$6 and ESTOI $\mathcal{O}\bigl(Tw + T(T-w)/d\bigr)$7 (Zhang et al., 2023). DF-Conformer integrates FAVOR+ self-attention and 1-D dilated depthwise convolution into a Conv-TasNet-style mask predictor; DF-Conformer-8 reports SI-SNR$\mathcal{O}\bigl(Tw + T(T-w)/d\bigr)$8 $\mathcal{O}\bigl(Tw + T(T-w)/d\bigr)$9 dB, compared with TDCN++ at $\mathcal{O}(T^2)$0 dB, while retaining real-time feasibility (Koizumi et al., 2021). OFIF-Net introduces overlapped-frame information fusion and a causal time-frequency-channel attention block; on VoiceBank+DEMAND, TF-DCTCRN + TFSM yields WB-PESQ $\mathcal{O}(T^2)$1, adding OFIF alone gives $\mathcal{O}(T^2)$2, adding TFCA alone gives $\mathcal{O}(T^2)$3, and full OFIF-Net reaches $\mathcal{O}(T^2)$4 (Zhang et al., 21 Jan 2025).

A neighboring line uses self-attention more sparingly or replaces it partly. Spectrum Attention Fusion replaces one or more global self-attention layers in a speech Transformer with a convolutional module, local frequency attention, and TCN blocks, achieving WB-PESQ $\mathcal{O}(T^2)$5, STOI $\mathcal{O}(T^2)$6, CSIG $\mathcal{O}(T^2)$7, CBAK $\mathcal{O}(T^2)$8, COVL $\mathcal{O}(T^2)$9, and SSNR $M^s$0 with only $M^s$1 M parameters on Voice Bank + DEMAND (Long et al., 2023). A plausible implication is that, in speech enhancement, SAE increasingly serves not only to add global context but also to decide when global self-attention should be narrowed, sparsified, linearized, or replaced.

4. Locality, causality, and computational efficiency

A central theme in SAE research is that unconstrained dense self-attention is often computationally prohibited for long signals or high-resolution feature maps, and may neglect strong local correlations. The cited work responds by imposing locality priors, chunking computation, or replacing $M^s$2 attention with approximate variants (Zhang et al., 2023, Zhang et al., 2022, Koizumi et al., 2021, Miao et al., 2020).

The memory arithmetic is made explicit in SASEGAN. If attention is inserted after layer $M^s$3, the temporal length is $M^s$4 and the module has an $M^s$5 memory footprint. At $M^s$6, the attention map is $M^s$7 (approximately $M^s$8 M entries), whereas at $M^s$9 it is $\beta$0 ($\beta$1 entries). Parameter count per block is $\beta$2; at $\beta$3, $\beta$4, $\beta$5, giving approximately $\beta$6k parameters, while at $\beta$7, $\beta$8, giving approximately $\beta$9k parameters (Phan et al., 2020). The result is an explicit placement rule: higher layers sharply reduce memory overhead.

Sparse and linearized variants formalize the same trade-off. Ripple attention reduces full $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$00 attention to local cost $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$01 plus dilated cost $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$02 (Zhang et al., 2023). DF-Conformer approximates softmax self-attention by

$$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$03

which changes the dominant cost from quadratic to effectively linear in sequence length when the random feature dimension is small (Koizumi et al., 2021). Chunk-SAE in online CTC/attention ASR divides input into chunks with left and right context and, with state reuse, reduces per-layer cost from full offline $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$04 to $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$05 while keeping bounded latency (Miao et al., 2020).

Real-time systems add causality constraints rather than merely reducing FLOPs. DP-SARNN performs intra-chunk and inter-chunk attention inside a dual-path RNN, uses causal attention in the inter-chunk SARNN for real-time deployment, and reports $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$06 ms CPU time to process a signal chunk of $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$07 ms (Pandey et al., 2020). OFIF-Net exploits the fact that the overlap-and-add operation in inverse TF transformation already incurs an algorithmic delay equal to the window size; OFIF constructs pseudo overlapped-frames and applies causal time-, frequency-, and channel-wise self-attention within that inherent delay (Zhang et al., 21 Jan 2025).

Vision systems show the same scaling concern. EAANet augments ResNet18 with efficient self-attention blocks based on Linformer or Longformer. The paper reports that peak GPU memory of full self-attention rises as $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$08, hitting $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$09 quickly as resolution grows, whereas EAANet-Linformer and EAANet-Longformer stay under $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$10 GB up to $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$11 at batch $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$12 (Zhang et al., 2022). This suggests that, in SAE practice, “enhancement” frequently means adding inductive bias precisely to avoid the pathological scaling of unconstrained attention.

5. Expansion beyond speech

Although speech enhancement provides the most extensive SAE literature here, the same design principles appear in computer vision, language modeling, and diffusion-based generation. In these domains the enhancement target shifts from speech quality or intelligibility to classification accuracy, robustness, sequence modeling, or controllable generation (Yang et al., 2021, Zhai, 10 Mar 2026, Lygizou et al., 29 May 2025, Zuo et al., 15 Aug 2025).

LESA is the clearest vision analogue of speech-oriented SAE. It interprets self-attention as a sum of a unary local term and a binary context term, argues that the vanilla unary term receives only small softmax mass, then replaces that term with grouped convolution and dynamically fuses it with the context branch. On ImageNet1K, ResNet50-Conv reports $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$13 top-1 accuracy, ResNet50-SA reports $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$14, and ResNet50-LESA reports $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$15; on COCO object detection with Mask-R-CNN, R50-Conv gives AP$$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$16, R50-SA gives $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$17, and R50-LESA gives $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$18 (Yang et al., 2021). The paper also reports that removing the unary term from SA drops ImageNet top-1 from $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$19 to $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$20.

EAANet approaches enhancement through efficient hybridization rather than local/context decomposition. It injects efficient ViT blocks into ResNet18 using either replacement or concatenation, and finds concatenation more effective in practice. On CIFAR-10, ResNet18 reports Top-1 $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$21, AA-Net reports $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$22, EAANet-Replacement (Long) reports $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$23, and EAANet-Concatenation (Long) reports $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$24 (Zhang et al., 2022). The same work emphasizes scaling to high-resolution images as a primary advantage of efficient SAE mechanisms.

Language modeling introduces a different SAE objective: reducing self-information leakage. Exclusive Self Attention is positioned explicitly within the “Self-Attention Enhancement (SAE) spectrum” and enforces orthogonality between each token’s attention output and its own value vector. On downstream tasks, the paper reports average accuracy gains of $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$25 points for the $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$26 B model, $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$27 for the $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$28 B model, and $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$29 for the $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$30 B model, with gains increasing as sequence length grows (Zhai, 10 Mar 2026). The same study states that removing the orthogonalization step reverts XSA to baseline.

Differential Gated Self-Attention focuses on corrupted inputs and noise resilience rather than sequence length. On language benchmarks, DGT (proj $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$31) improves over both vanilla Transformer and Differential Transformer baselines; for example, on AG News the reported accuracies are $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$32, and on 20 Newsgroups $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$33 (Lygizou et al., 29 May 2025). On vision benchmarks, the best DGViT improves over ViT from $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$34 to $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$35 on CIFAR-10 and from $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$36 to $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$37 on SVHN (Lygizou et al., 29 May 2025).

In diffusion-based anomaly generation, SAE acquires a mask-conditioned generative role. AAG modifies Stable Diffusion self-attention so that anomaly tokens attend only to normal visual tokens, scaled by $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$38, thereby encouraging anomalies to remain coherent with the original pattern. The ablation reports, on MVTec AD, baseline I-AUROC $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$39, $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$40CAE only $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$41, and $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$42CAE+SAE $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$43, with PRO rising from $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$44 to $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$45; on VisA, I-AUROC improves from $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$46 to $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$47 and PRO from $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$48 to $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$49 when SAE is added on top of CAE (Zuo et al., 15 Aug 2025). Here SAE is not used to classify or denoise, but to bias visual token aggregation so that generated anomalies inherit texture and color from normal regions.

6. Empirical regularities, misconceptions, and open problems

Several empirical regularities recur across the literature. First, SAE gains are often strongest precisely where baseline self-attention is least well matched to the data. T-GSA attributes its gains to the fact that speech frames close in time are more correlated than distant ones (Kim et al., 2019). Ripple attention reports superiority over full attention, blockwise attention, and SepFormer while using a sparse mask tailored to local and coarse global dependencies (Zhang et al., 2023). XSA reports that gains increase with sequence length, from $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$50 at $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$51 up to $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$52 at $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$53 K in validation loss (Zhai, 10 Mar 2026). A plausible implication is that SAE mechanisms are most valuable when the task imposes a strong structural asymmetry—local continuity, streaming latency, long-context distraction, or mask-guided generation—that vanilla attention does not encode directly.

Second, more attention is not automatically better. SASEGAN reports no statistically significant difference among layer placements and therefore recommends a single top-layer attention block when memory is a concern (Phan et al., 2020). Spectrum Attention Fusion reports that stacking two AF layers without skip-connections increases parameters from $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$54 M to $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$55 M but lowers PESQ from $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$56 to $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$57, whereas adding skip-connections recovers PESQ to $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$58 (Long et al., 2023). M-DGSA reports that one-layer gating works best and deeper MLP gates degrade performance (Lygizou et al., 29 May 2025). These results argue against an oversimplified view in which SAE is merely the addition of more global attention blocks.

Third, the literature contains a substantive but productive tension about the role of the self component. XSA treats the strong alignment of attention output with a token’s own value vector as an “attention similarity bias” that wastes contextual capacity (Zhai, 10 Mar 2026). LESA, by contrast, reports that the self or unary term carries only about $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$59 softmax mass in vanilla SA but that removing it causes a large accuracy drop, motivating a stronger local pathway rather than its deletion (Yang et al., 2021). Diffusion SAE goes further and explicitly zeroes anomaly-to-anomaly attention, but only for the masked generative setting (Zuo et al., 15 Aug 2025). Objectively read together, these papers do not yield a single doctrine about self-attention’s diagonal component; instead they show that the useful treatment of “self” depends on whether the target problem is contextual language modeling, dense visual recognition, or masked anomaly synthesis.

Finally, the cited papers identify different open directions. XSA proposes multi-direction exclusion, dynamic exclusion, cross-modal XSA, and combinations with sparse or low-rank attention (Zhai, 10 Mar 2026). EAANet identifies position embedding flexibility, runtime efficiency of sparse kernels, and larger-scale benchmarks as unresolved issues (Zhang et al., 2022). LESA notes that high memory usage from the $$G(i,j)=\exp\!\left[-\frac{(i-j)^2}{2\sigma^2}\right],$$60 attention map remains a principal limitation and points toward linearized or locality-restricted variants (Yang et al., 2021). Across the literature, this suggests that future SAE research is likely to continue in two directions at once: richer task-specific inductive bias and tighter control of the computational footprint of attention.