---
title: Locally Shifted Attention Mechanisms
url: https://www.emergentmind.com/topics/locally-shifted-attention
type: topic
---

# Locally Shifted Attention Mechanisms

Locally shifted attention refers to a broad family of neural attention mechanisms that constrain each query’s context to a spatially or temporally local neighborhood, with a learnable or adaptive “shift” or bias of the receptive field. This design aims to mitigate the inefficiencies and inductive limitations of global attention by emphasizing locality while preserving a receptive field that can shift, grow, or propagate over layers. Locally shifted attention is a core ingredient in high-efficiency transformer architectures, sequence-to-sequence models for monotonic tasks, and scalable vision networks, manifesting in variants such as monotonic attention, shifted window attention, group shifted attention, and virtual-patch shifted attention. Common goals are efficient computation, improved inductive bias for locality or monotonicity, and enhanced feature continuity across partition boundaries.

## 1. Monotonic Local Attention for Sequence Models

The first formalization of locally shifted attention appeared in the context of monotonic, sequence-to-sequence models such as end-to-end speech recognition and grapheme-to-phoneme conversion [1705.08091]. In global attention, each output step $t$ computes alignment over all input states $h^e_s$:

\[
c_t = \sum_{s=1}^S \alpha_{t,s} h^e_s,\quad
\alpha_{t,s} = \frac{\exp(\text{Score}(h^e_s, h^d_t))}{\sum_{s'} \exp(\text{Score}(h^e_{s'}, h^d_t))}
\]
This incurs $O(S \cdot T)$ complexity and permits jumps, unsuitable for monotonic mappings.

Local monotonic attention instead:
- Predicts a center $p_t \geq p_{t-1}$, with $\Delta p_t$ computed via an MLP, enforcing strict monotonicity.
- Attends only within a window of size $2\sigma$ around $p_t$, reducing context to $O(\sigma \cdot T)$.
- Computes a Gaussian prior $a_t^{\mathcal N}(s)$ centered at $p_t$, combined multiplicatively with a local scorer $a_t^{\mathcal S}(s)$ over the window.

Key formulas:
\[
\Delta p_t = \exp(V_p^\intercal \tanh(W_p h^d_t)), \quad p_t = p_{t-1} + \Delta p_t
\]
\[
a_t^{\mathcal N}(s) = \lambda_t \exp\left(-\frac{(s - p_t)^2}{2\sigma^2}\right)
\]
\[
c_t = \sum_{s \in \mathcal W_t} [a_t^{\mathcal N}(s)a_t^{\mathcal S}(s)]\,h^e_s
\]
This yields sharper, step-wise alignments well-suited for speech and mapping tasks, achieving both a 12% relative error reduction in TIMIT PER and reduced decoding complexity [1705.08091].

## 2. Shifted-Window Attention in Vision Transformers

Locally shifted attention achieved widespread impact through shifted window architectures in vision transformers, notably Swin Transformer and its descendants. The key technique partitions the feature map into non-overlapping $M \times M$ windows, then alternates layers with regular and half-window cyclically shifted partitions [2312.02725, 2504.15317, 2507.21922, 2509.08959, 2409.06206]. This strategy achieves:

- **Intra-window locality:** Within each window, self-attention is computed only among $M^2$ spatial tokens, drastically reducing per-block cost from $O(H^2 W^2)$ to $O(H W M^2)$.
- **Window shifting:** Every second layer, the window grid is shifted by $\lfloor M/2 \rfloor$ so that window boundaries move. Tokens at window boundaries in one layer are included in the center of a window in the next, enabling cross-window information flow.
- **Hierarchical stacking:** Multiple layers (or stages) alternate regular and shifted windows, progressively expanding the effective receptive field.

The attention within each window is computed as:
\[
Q^i = X^i W^Q, \quad
K^i = X^i W^K, \quad
V^i = X^i W^V, \quad
A^i = \mathrm{softmax}\left(\frac{Q^i (K^i)^T}{\sqrt{d}} + B\right)
\]
where $B$ is a learnable relative position bias per window.

**Complexity comparison**:

| Attention Type                  | Complexity per Layer      | Long-Range Propagation |
|---------------------------------|--------------------------|------------------------|
| Global (ViT)                    | $O(H^2 W^2)$             | Single pass           |
| Windowed (no shift)             | $O(H W M^2)$             | None across windows    |
| Shifted-window (locally shifted)| $O(H W M^2)$, with cross-window |
| Group shifted (AgileIR)         | $O(\frac{H W M^2}{G})$   | Same, with memory savings |

Alternating shifted windows yields nearly global receptive fields after a small number of blocks, while retaining linear cost in $H W$ [2312.02725, 2507.21922, 2409.06206].

## 3. Mechanisms for Locality and Shifted Context

Variants of locally shifted attention deploy different mechanisms for shifting or overlapping locality:

- **Cyclically shifted windows (Swin, R3D-SWIN, CoSwin, AgileIR):** Partition and cyclically roll the feature map spatially by $(-s, -s)$, where $s = \lfloor M/2 \rfloor$, before re-windowing, compute per-window attention, then inverse the shift.
- **Multiple shifted and overlapped windows (Sow-Attention, EleGANt):** Partition the feature map multiple times with different half-window offsets (e.g., four schemes with $o_j \in \{0, S/2\} \times \{0, S/2\}$), and merge the outcomes for smooth, block-artifact-free continuity [2207.09840].
- **Group-wise head decomposition (AgileIR):** Split multi-head self-attention into $G$ groups, applying window and shift-per-group, yielding further memory reduction at the same locality scale [2409.06206].
- **Learned local bias (LocAtViT):** Imposes a learnable Gaussian prior centered at each query patch, biasing attention logits toward spatial proximity without explicit windowing [2603.04892].

In all cases, locality is defined by window size or Gaussian scale, while the shift (either deterministic or parameterized) enforces dynamic spatial coverage and aggregation.

## 4. Representative Architectures and Algorithmic Patterns

### Swin Transformer and Derivatives

Swin Transformer alternates block types:

1. **Window MSA (W-MSA):** Non-overlapping $M \times M$ attention.
2. **Shifted Window MSA (SW-MSA):** Cyclically shift by $s$ spatially, re-partition and attend in new windows, then inverse shift.

This alternating sequence enables propagation of information beyond window boundaries. Each block includes attention, MLP, normalization, and skip connections. In SwinECAT and R3D-SWIN, shifted window attention is foundational for high-resolution fundus image and 3D voxel tasks [2312.02725, 2507.21922].

### CoSwin

Adds a parallel learnable local convolutional enhancement module at every block, fusing conv and attention features, preserving translation equivariance and achieving gains especially in low-data/low-resolution tasks [2509.08959].

### EleGANt Sow-Attention

Partitions into four overlapping, half-shifted window schemes and merges outputs by geometric bilinear weights, achieving fine-grained, artifact-free attention for high-frequency image manipulations [2207.09840].

### LocAtViT

Adds a learnable Gaussian bias to global attention logits, resulting in a “softly shifted” locality that preserves global receptive field but encourages strong local focus—well-suited for segmentation tasks where spatial detail is critical [2603.04892].

### Local Shifted Attention With Early Global Integration

Proposes per-patch locality constructed by soft-attending across $T$ circularly shifted patch neighborhoods, then aggregating via per-query attention, finally applying early-stage global self-attention for full receptive field [2112.05080].

## 5. Mathematical Formulation and Complexity

The core shared structure is:

- **Window partitioning**: $X \rightarrow\ $ Batch of $M \times M$ (or $M \times M \times M$ for 3D) tensors.
- **Optional shift**: $X_{shifted} = \mathrm{Roll}(X, \text{shift}=(-s, -s))$
- **Attention within window**:
  \[
  Q_i = X_i W^Q,\; K_i = X_i W^K,\; V_i = X_i W^V
  \]
  \[
  A_i = \mathrm{softmax}\left(\frac{Q_i K_i^T}{\sqrt{d}} + B\right)
  \] 
  \[
  \text{Output}_i = A_i V_i
  \]
- **Multi-shifting/overlap (if used)**: repeat above for each offset, then aggregate.

**Computational characteristics:**
- Standard global attention: $O((HW)^2 \cdot d)$
- Windowed or locally shifted: $O(HW \cdot M^2 \cdot d)$, $M \ll \min(H,W)$
- Grouped heads (AgileIR): $O(\frac{HW \cdot M^2 \cdot d}{G})$

**Empirical impact**: In AgileIR, group shifted window attention (GSWA) cuts training memory overhead by $\sim$50% at large batch size, with negligible performance drop (0.1 dB in PSNR for restoration) [2409.06206].

## 6. Empirical Performance and Application Domains

Locally shifted attention has demonstrated consistent efficiency and accuracy gains across domains:

- **Speech and Sequence Modeling:** Up to 12% relative error reduction and 3 BLEU improvement over global attention for monotonic alignment [1705.08091].
- **Vision (2D/3D):** SOTA or superior metrics in single-view 3D voxel reconstruction [2312.02725], 2–5% accuracy gain in small-scale image classification [2509.08959], +1.7 pp in fundus disease classification [2507.21922], and substantial gains in segmentation mIoU on ADE20K and others [2603.04892].
- **Image Generation and Restoration:** Linear cost Sow-Attention achieves artifact-free, high-res makeup transfer with up to 16$\times$ less compute than global attention [2207.09840]; GSWA provides scalable super-resolution/restoration with >50% memory reduction [2409.06206].

## 7. Trade-offs, Limitations, and Design Considerations

Locally shifted attention offers substantial efficiency and inductive bias advantages, but with specific trade-offs:

- **Window size vs. context:** Small $M$ yields better locality but slower global context propagation; too large $M$ dilutes locality.
- **Cross-window propagation:** Shifting is critical for breaking window isolation; overlap or group decomposition further ensures information continuity.
- **Task dependence:** For strictly monotonic or strongly local tasks, local shifting is ideal; for global or dense prediction, hybrid approaches (e.g., LocAtViT’s learnable bias) retain best-of-both performance.
- **Implementation complexity:** Overlapping windows, group decomposition, and bilinear aggregation (for smoothing) increase code complexity, but provide substantial memory and artifact improvements [2207.09840, 2409.06206].
- **Limitations:** Gains are task- and backbone-specific; in backbones like Swin, LocAtViT’s explicit Gaussian bias provides limited additional benefit [2603.04892].

In summary, locally shifted attention mechanisms unify a spectrum of strategies for balancing locality, efficiency, and global context in attention models. By leveraging spatial or temporal shifts—either explicitly through window realignment, overlap, or implicitly through learned priors—they achieve scalable, artifact-resistant modeling across sequence, vision, and generative domains, and now represent a foundational building block for modern scalable neural architectures [1705.08091, 2312.02725, 2507.21922, 2603.04892, 2207.09840, 2409.06206, 2509.08959, 2112.05080].

Source: https://www.emergentmind.com/topics/locally-shifted-attention