Local Attention in Neural Models

• Local attention is a mechanism that limits query interactions to a specific neighborhood (e.g. windows, syntactic structures, spatial patches), reducing unnecessary long-range mixing.
• It is applied in various domains such as language modeling, vision, and speech enhancement to lower computational load while maintaining task-relevant context.
• Hybrid designs combining local and global pathways often yield superior performance by balancing focused processing with broad contextual aggregation.

Local attention is a family of attention mechanisms in which each query interacts with a restricted neighborhood rather than the full input domain. In the literature, that neighborhood is defined in multiple ways: a bounded token window in autoregressive language modeling, a dependency-tree radius in syntax-aware encoders, a spatial neighborhood or patch in vision, a set of adjacent frequency bins in speech enhancement, or a learned cluster in feature space. Across these formulations, the common purpose is to impose a locality prior, reduce irrelevant long-range mixing, and often lower the computational burden of dense global attention, while preserving task-relevant context through stacking, overlap, or hybridization with global branches (Xu et al., 2 Jan 2025, Pan et al., 2023, Hou et al., 2023, Li et al., 2020).

1. Definitions and scope

In sequence models, local attention usually means that token $i$ attends only to a bounded set of predecessors or neighbors. In sliding-window attention, the attention range is a fixed window of size $w$; in syntax-aware local attention, the scope is local in dependency-tree distance rather than linear position; and in formal analyses of transformer recognizers, a $k$-local mask allows attention only to the previous $k$ positions (Xu et al., 2 Jan 2025, Li et al., 2020, Li et al., 1 May 2026).

In vision, the term is broader. It can denote attention inside local image windows, overlapping neighborhoods around each pixel, patch-wise attention over aerial-image regions, geometry-preserving sparse 2D neighborhoods, feature-space clusters of similar patches, or directional positional sequences obtained by strip pooling in CNN attention modules (Arar et al., 2021, Ding et al., 2019, Daras et al., 2019, Yu et al., 2022, Xu et al., 2024). In speech enhancement, local attention has been used to restrict spectral interactions to adjacent frequency bands or to emphasize short-range, multi-receptive-field speech structure instead of full non-local aggregation (Hou et al., 2023, Xu et al., 2022).

The term also has an interpretive meaning. In analyses of BERT, “local” refers to what an attention head does with respect to its immediate hidden-token inputs at layer $l-1$, whereas “global” refers to which original input tokens influence the head output after information has been mixed across layers. This distinction separates head-internal routing from input-level attribution (Pascual et al., 2020).

Setting	Neighborhood defining locality	Representative works
Language and sequence modeling	Bounded predecessor window, band mask, dependency-tree distance	(Xu et al., 2 Jan 2025, Li et al., 1 May 2026, Li et al., 2020)
Vision	Local windows, overlapping neighborhoods, patches, 2D sparse geometry, feature-space clusters, coordinate strips	(Pan et al., 2023, Arar et al., 2021, Daras et al., 2019, Yu et al., 2022, Xu et al., 2024)
Speech	Adjacent frequency bands, local receptive-field channel attention	(Hou et al., 2023, Xu et al., 2022)

A recurring misconception is that local attention is synonymous with a fixed linear window. The cited literature does not support that reduction. Locality may be linear, syntactic, geometric, patch-based, coordinate-aware, frequency-local, or feature-space-based, depending on the structure regarded as most informative for the task.

2. Core formulations and design patterns

The most common implementation is masked softmax. In full-band speech enhancement, local spectral attention introduces a binary mask

$$M_{F,i,j} = \begin{cases} 0, & |i-j| \le N_l \ -\infty, & |i-j| > N_l \end{cases}$$

so that each frequency bin attends only to nearby bands (Hou et al., 2023). In formal treatments of causal transformers, $k$-local attention is defined by

$$M^{\le k}_{n,m}= \begin{cases} 1 & \text{if } \max(1,n-k)\le m<n,\ 0 & \text{otherwise,} \end{cases}$$

which makes locality a property of the mask alone (Li et al., 1 May 2026). Encoder-side local transformers have also been implemented with fixed band masks such as band-1 and band-2, as well as directional masks like prev-1, prev-2, next-1, next-2, and identity (Pande et al., 2020).

A second pattern is overlapping local aggregation. QnA replaces per-window queries with learned queries shared across overlapping windows, so attention remains local and shift-invariant while avoiding the memory cost of explicit unfolding. Its single-query form can be written as

$$z_{i,j}=\text{Attention}\!\left(\tilde{q},K_{\mathcal{N}_{i,j}}\right)\cdot V_{\mathcal{N}_{i,j}},$$

with $\mathcal{N}_{i,j}$ the $w$0 neighborhood (Arar et al., 2021). Slide Attention likewise keeps the query-centric local neighborhood, but reinterprets Im2Col row-wise and replaces the shifts with depthwise convolutions, then adds a deformed shifting branch merged by re-parameterization at inference time (Pan et al., 2023).

A third pattern is locality defined by structure other than raw position. BOAT supplements image-space local attention with feature-space local attention by balanced hierarchical clustering, so attention is computed inside equal-sized clusters of similar tokens rather than only inside spatial windows (Yu et al., 2022). Syntax-aware local attention uses dependency-tree distance thresholds $w$1, implemented through an attention mask, so a token attends to syntactically nearby words even if they are far apart linearly (Li et al., 2020). In aerial segmentation, LANet computes patch descriptors over $w$2 regions and derives patch-specific channel attention maps, making the patch rather than the full image the basic local unit (Ding et al., 2019).

These constructions show that local attention is less a single algorithm than a design principle: attention is restricted to a task-specific neighborhood chosen to preserve a salient inductive bias while suppressing interactions deemed weak, noisy, or redundant.

3. Local attention in language and sequence modeling

In autoregressive LLMs, sliding-window attention is the canonical local mechanism. Standard self-attention has quadratic time complexity and a cache that grows linearly with sequence length, whereas sliding-window attention restricts attention to a fixed recent context of size $w$3, reducing per-head cost to $w$4 and cache to $w$5 (Xu et al., 2 Jan 2025). MSWA refines this by assigning different window sizes across heads and layers. Within a layer, head groups receive $w$6, $w$7, $w$8, and $w$9; across depth, layer groups receive $k$0, $k$1, $k$2, and $k$3. On 12-layer LLMs trained from scratch, SWA with $k$4 gives Wikitext-103 perplexity $k$5 and enwik8 $k$6 bpc, whereas full MSWA reaches $k$7 and $k$8 at lower relative cost; in Llama-7B fine-tuning, average accuracy rises from $k$9 to $k$0 in 3-shot and from $k$1 to $k$2 in 5-shot (Xu et al., 2 Jan 2025).

RAttention addresses a central limitation of sliding-window attention: complete disregard for out-of-window tokens. It adds a Residual Linear Attention branch with recurrence

$k$3

so the local branch models tokens in $k$4 exactly while the recurrent state summarizes tokens in $k$5 (Wang et al., 18 Jun 2025). At 12B scale and 8K pretraining length, RAttention-512 reports Average (0/1-shot) $k$6 versus $k$7 for Full 8k, MMLU $k$8 versus $k$9, and GSM8K $l-1$0 versus $l-1$1. On RULER after 4K pretraining, RAttn-512 obtains $l-1$2 at 4K, $l-1$3 at 8K, $l-1$4 at 16K, and $l-1$5 at 32K, far above both Full-4k and SWA-2k (Wang et al., 18 Jun 2025).

Local attention in encoders need not be purely linear-window-based. Syntax-aware local attention for BERT constrains attention by dependency-tree distance and leaves $l-1$6 and $l-1$7 unmasked. Fine-tuned on downstream tasks, it reports on Chinese benchmarks $l-1$8 accuracy on ChnSentiCorp, $l-1$9 F1 on MSRA NER, and $$M_{F,i,j} = \begin{cases} 0, & |i-j| \le N_l \ -\infty, & |i-j| > N_l \end{cases}$$0 F1 on CGED, outperforming both a BERT reimplementation and a window-based local baseline (Li et al., 2020). A separate study on the importance of local information finds, via gradient sensitivity, that $$M_{F,i,j} = \begin{cases} 0, & |i-j| \le N_l \ -\infty, & |i-j| > N_l \end{cases}$$1 across QQP, SST-2, MRPC, and QNLI, and shows that even fully local-attention encoder configurations achieve comparable performance to unconstrained transformers on both GLUE and machine translation (Pande et al., 2020).

Theoretical work sharpens the picture. In fixed-precision transformers, global attention corresponds to a fragment with an unbounded past operator, $$M_{F,i,j} = \begin{cases} 0, & |i-j| \le N_l \ -\infty, & |i-j| > N_l \end{cases}$$2-local attention corresponds to a fragment with a bounded-past operator, and hybrid global-local transformers correspond to a richer fragment containing both. The paper proves that global and local attention are expressively complementary, neither subsumes the other, and hybrid global-local attention is strictly more expressive than either alone (Li et al., 1 May 2026).

4. Local attention in vision and multimodal perception

Vision work has treated locality as a way to restore inductive bias that is weakened by dense global self-attention. Slide-Transformer defines local attention as restricting each query to its neighboring pixels, then makes the mechanism practical by replacing Im2Col-style neighborhood extraction with standard convolution primitives. Integrated into hierarchical backbones, Slide Attention improves ImageNet-1K accuracy across PVT, PVTv2, Swin, CSWin, and NAT variants; for example, PVT-T improves from $$M_{F,i,j} = \begin{cases} 0, & |i-j| \le N_l \ -\infty, & |i-j| > N_l \end{cases}$$3 to $$M_{F,i,j} = \begin{cases} 0, & |i-j| \le N_l \ -\infty, & |i-j| > N_l \end{cases}$$4, Swin-T from $$M_{F,i,j} = \begin{cases} 0, & |i-j| \le N_l \ -\infty, & |i-j| > N_l \end{cases}$$5 to $$M_{F,i,j} = \begin{cases} 0, & |i-j| \le N_l \ -\infty, & |i-j| > N_l \end{cases}$$6, and NAT-S from $$M_{F,i,j} = \begin{cases} 0, & |i-j| \le N_l \ -\infty, & |i-j| > N_l \end{cases}$$7 to $$M_{F,i,j} = \begin{cases} 0, & |i-j| \le N_l \ -\infty, & |i-j| > N_l \end{cases}$$8 (Pan et al., 2023). QnA pursues a related goal with overlapping local aggregation and learned shared queries, reporting up-to x10 less memory and up-to x5 faster execution than existing methods while remaining competitive in accuracy (Arar et al., 2021).

Other vision formulations redefine the neighborhood itself. BOAT combines image-space local attention with feature-space local attention, grouping tokens by balanced hierarchical clustering so that semantically similar but spatially distant patches can interact. Relative to its base backbones, BOAT-Swin-T improves ImageNet-1K top-1 from $$M_{F,i,j} = \begin{cases} 0, & |i-j| \le N_l \ -\infty, & |i-j| > N_l \end{cases}$$9 to $k$0, and BOAT-CSWin-T from $k$1 to $k$2; on ADE20K, BOAT-CSWin-T rises from $k$3 to $k$4 mIoU (Yu et al., 2022). In generative modeling, YLG-SAGAN replaces dense SAGAN attention with a 2D local sparse attention layer that preserves geometry through Enumerate, Shift, Apply. On ImageNet-128, FID improves from $k$5 to $k$6 and Inception score from $k$7 to $k$8, with peak performance reached after about $k$9k steps instead of more than $$M^{\le k}_{n,m}= \begin{cases} 1 & \text{if } \max(1,n-k)\le m<n,\ 0 & \text{otherwise,} \end{cases}$$0M (Daras et al., 2019).

CNN-based local attention does not necessarily use token-to-token self-attention. ELA derives horizontal and vertical 1D positional descriptors by strip pooling, processes them with 1D convolution and Group Normalization, and applies the resulting gates without channel reduction, explicitly contrasting this design with Coordinate Attention (Xu et al., 2024). In aerial semantic segmentation, LANet treats the patch as the meaningful local unit: a Patch Attention Module computes patch-wise attention, and an Attention Embedding Module transfers local focus from high-level to low-level features. On Potsdam, LANet reaches mean F1 $$M^{\le k}_{n,m}= \begin{cases} 1 & \text{if } \max(1,n-k)\le m<n,\ 0 & \text{otherwise,} \end{cases}$$1 and OA $$M^{\le k}_{n,m}= \begin{cases} 1 & \text{if } \max(1,n-k)\le m<n,\ 0 & \text{otherwise,} \end{cases}$$2, improving over a baseline FCN at $$M^{\le k}_{n,m}= \begin{cases} 1 & \text{if } \max(1,n-k)\le m<n,\ 0 & \text{otherwise,} \end{cases}$$3 mean F1 and $$M^{\le k}_{n,m}= \begin{cases} 1 & \text{if } \max(1,n-k)\le m<n,\ 0 & \text{otherwise,} \end{cases}$$4 OA; on Vaihingen it reaches mean F1 $$M^{\le k}_{n,m}= \begin{cases} 1 & \text{if } \max(1,n-k)\le m<n,\ 0 & \text{otherwise,} \end{cases}$$5 and OA $$M^{\le k}_{n,m}= \begin{cases} 1 & \text{if } \max(1,n-k)\le m<n,\ 0 & \text{otherwise,} \end{cases}$$6 (Ding et al., 2019).

Multimodal and embodied systems adopt similar locality constraints. In vision-and-language navigation, Local Slot Attention applies a fixed sparse mask on a $$M^{\le k}_{n,m}= \begin{cases} 1 & \text{if } \max(1,n-k)\le m<n,\ 0 & \text{otherwise,} \end{cases}$$7 panoramic grid so that each candidate view attends only to a local $$M^{\le k}_{n,m}= \begin{cases} 1 & \text{if } \max(1,n-k)\le m<n,\ 0 & \text{otherwise,} \end{cases}$$8 neighborhood. The $$M^{\le k}_{n,m}= \begin{cases} 1 & \text{if } \max(1,n-k)\le m<n,\ 0 & \text{otherwise,} \end{cases}$$9 mask gives the best unseen validation performance, and the full model improves R2R validation seen SR from $$z_{i,j}=\text{Attention}\!\left(\tilde{q},K_{\mathcal{N}_{i,j}}\right)\cdot V_{\mathcal{N}_{i,j}},$$0 to $$z_{i,j}=\text{Attention}\!\left(\tilde{q},K_{\mathcal{N}_{i,j}}\right)\cdot V_{\mathcal{N}_{i,j}},$$1, SPL from $$z_{i,j}=\text{Attention}\!\left(\tilde{q},K_{\mathcal{N}_{i,j}}\right)\cdot V_{\mathcal{N}_{i,j}},$$2 to $$z_{i,j}=\text{Attention}\!\left(\tilde{q},K_{\mathcal{N}_{i,j}}\right)\cdot V_{\mathcal{N}_{i,j}},$$3, and test unseen SPL from $$z_{i,j}=\text{Attention}\!\left(\tilde{q},K_{\mathcal{N}_{i,j}}\right)\cdot V_{\mathcal{N}_{i,j}},$$4 to $$z_{i,j}=\text{Attention}\!\left(\tilde{q},K_{\mathcal{N}_{i,j}}\right)\cdot V_{\mathcal{N}_{i,j}},$$5 over Recurrent VLN-Bert (Zhuang et al., 2022). In object detection, Local-Global Attention uses small kernels $$z_{i,j}=\text{Attention}\!\left(\tilde{q},K_{\mathcal{N}_{i,j}}\right)\cdot V_{\mathcal{N}_{i,j}},$$6 for the local branch and a larger kernel for the global branch, then fuses them with learnable weights; on TinyPerson with a MobileNetV3 backbone, mAP@50 rises from $$z_{i,j}=\text{Attention}\!\left(\tilde{q},K_{\mathcal{N}_{i,j}}\right)\cdot V_{\mathcal{N}_{i,j}},$$7 to $$z_{i,j}=\text{Attention}\!\left(\tilde{q},K_{\mathcal{N}_{i,j}}\right)\cdot V_{\mathcal{N}_{i,j}},$$8 and mAP@50-95 from $$z_{i,j}=\text{Attention}\!\left(\tilde{q},K_{\mathcal{N}_{i,j}}\right)\cdot V_{\mathcal{N}_{i,j}},$$9 to $\mathcal{N}_{i,j}$0 (Shao, 2024).

5. Local attention in speech and acoustic enhancement

In full-band speech enhancement, the main locality axis is frequency. Local spectral attention replaces global frequency-wise self-attention with attention over adjacent frequency bins, motivated by the observation that low-frequency harmonic structure and high-frequency components differ substantially in full-band $\mathcal{N}_{i,j}$1 kHz enhancement (Hou et al., 2023). In MTFAA, replacing global frequency attention with local spectral attention changes PESQ from $\mathcal{N}_{i,j}$2 to $\mathcal{N}_{i,j}$3, CBAK from $\mathcal{N}_{i,j}$4 to $\mathcal{N}_{i,j}$5, STOI from $\mathcal{N}_{i,j}$6 to $\mathcal{N}_{i,j}$7, and SiSDR from $\mathcal{N}_{i,j}$8 dB to $\mathcal{N}_{i,j}$9 dB; in DPARN, PESQ moves from $w$00 to $w$01 and SiSDR from $w$02 dB to $w$03 dB (Hou et al., 2023). The same paper reports that the improved causal MTFAA with LSA achieves the best objective score on the full-band VoiceBank+DEMAND set, with PESQ $w$04, CSIG $w$05, CBAK $w$06, COVL $w$07, and STOI $w$08 (Hou et al., 2023).

Selector-Enhancer adopts a different notion of local attention. Its local branch is a channel-wise attention mechanism with multiple receptive fields, built from parallel two-layer and four-layer convolutional stacks. A feature-filter trained with REINFORCE chooses, per time-frequency region, whether features should be routed to local attention or non-local attention (Xu et al., 2022). The study reports that removing local attention hurts performance, and that adding it yields a gain of $w$09 PESQ and $w$10 STOI in ablation. The larger claim is not that local attention should always replace non-local attention, but that different regions of the same utterance may require different processing regimes, especially under transient or mixed noise conditions (Xu et al., 2022).

These speech examples illustrate a general principle that also appears elsewhere: local attention is frequently motivated by the unreliability of indiscriminate global mixing when the signal is heterogeneous across time, frequency, or space. In that setting, restricting the interaction span is treated as a denoising prior rather than only a complexity reduction.

6. Efficiency, expressivity, and persistent debates

A major reason for adopting local attention is computational scaling. Standard attention in language modeling costs $w$11, while sliding-window attention reduces this to $w$12 with cache $w$13 (Xu et al., 2 Jan 2025). In practice, this is not the full story, because smaller windows improve efficiency only by discarding context. RAttention makes that limitation explicit: the core weakness of standard local attention is complete disregard for out-of-window tokens, so it augments the local window with a recurrent linear memory instead of enlarging the window itself (Wang et al., 18 Jun 2025).

Another persistent result is that local attention is often strongest when combined with a global or non-local pathway. In relation classification, pure local attention with $w$14 yields $w$15 F1, pure global attention with $w$16 yields $w$17, and the best score occurs around $w$18 at $w$19 F1 (Sun, 2024). In image retrieval, GLAM reports that local attention alone is weak or even harmful in most cases, whereas baseline + global + local gives the best results (Song et al., 2021). In object detection, the Local-Global Attention paper reports that both individual LA and GA improve performance, but their combination gives the best results on TinyPerson (Shao, 2024). These findings suggest that locality and globality are usually complementary rather than interchangeable.

Theoretical and interpretive debates reinforce that conclusion. The expressivity results for fixed-precision transformers show that local and global attention are incomparable and that combining them yields the richest fragment among the studied variants (Li et al., 1 May 2026). The BERT analysis likewise shows that attention maps can provide insight into local head behavior, but they are not reliable global explanations of which original input tokens matter after multilayer mixing; by the last layers, the correlation between attention distributions and input contribution drops sharply (Pascual et al., 2020). A plausible implication is that local attention should be understood as a structured routing mechanism whose value depends on the surrounding architecture, depth, and aggregation pathway, not as a universal substitute for full attention.

Several limitations recur across the literature. Fixed local windows can miss long-range dependencies, and the optimal span is task- and layer-dependent (Xu et al., 2 Jan 2025, Hou et al., 2023). Syntax-aware local attention slightly hurts QNLI and RTE, with the paper attributing this to especially important cross-sentence interactions in textual entailment (Li et al., 2020). In retrieval, local attention by itself is not consistently beneficial (Song et al., 2021). These results do not weaken the case for local attention; rather, they delimit it. Locality is most effective when the neighborhood definition matches the structure of the data and when the architecture retains some mechanism for broader aggregation.