---
title: Shifted Window Self-Attention
url: https://www.emergentmind.com/topics/shifted-window-multi-head-self-attention-sw-msa
type: topic
---

# Shifted Window Self-Attention

Shifted Window Multi-head Self-Attention (SW-MSA) is a self-attention mechanism foundational to the Swin Transformer architecture, introduced in "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" [2103.14030]. SW-MSA replaces the standard global self-attention, which is computationally prohibitive for high-resolution images, with a window-based variant that alternates between non-overlapping and shifted partitions of the input tokens. This approach enables linear computational scaling with respect to image size and facilitates both local and cross-window interaction, addressing core challenges in adapting transformers to visual data.

## 1. Window-based Self-Attention and the Shifted Windowing Scheme

Traditional self-attention operations in vision transformers entail pairwise computation across all input tokens, resulting in $O((hw)^2 \cdot C)$ complexity for an $h \times w$ patch map and channel dimension $C$. SW-MSA circumvents this cost by segmenting the input into non-overlapping windows of fixed size $M \times M$ and applying multi-head self-attention locally:
\[
\text{Attention}(Q, K, V) = \text{SoftMax}\left(\frac{QK^\top}{\sqrt{d}} + B\right) \cdot V
\]
where $Q$, $K$, $V \in \mathbb{R}^{M^2 \times d}$, $d$ is the head dimension, and $B$ is a learnable relative position bias matrix of shape $M^2 \times M^2$. This windowing yields computational cost $O(hw \cdot M^2 \cdot C)$, linear in image size (given constant $M$).

A key innovation in SW-MSA is the introduction of shifted windows. In consecutive transformer blocks, the window partitioning is offset by $(\lfloor M/2 \rfloor, \lfloor M/2 \rfloor)$ pixels. This shift causes patches initially at window boundaries to become window centers in the following layer, establishing inter-window connections and promoting context propagation.

## 2. Formal SW-MSA Block Alternation and Architecture

The Swin Transformer alternates between standard window multi-head self-attention (W-MSA) and shifted window multi-head self-attention (SW-MSA), formalized as:
\[
\begin{aligned}
\hat{z}^l &= \text{W-MSA}(\text{LN}(z^{l-1})) + z^{l-1} \\
z^l      &= \text{MLP}(\text{LN}(\hat{z}^l)) + \hat{z}^l \\
\hat{z}^{l+1} &= \text{SW-MSA}(\text{LN}(z^l)) + z^l \\
z^{l+1}  &= \text{MLP}(\text{LN}(\hat{z}^{l+1})) + \hat{z}^{l+1}
\end{aligned}
\]
This alternating scheme ensures that each block captures local structure (W-MSA) as well as inter-window dependencies (SW-MSA), enhancing representational richness. Layer normalization (LN) and feed-forward networks (MLP) follow each attention step, as in standard transformer architectures.

## 3. Hierarchical Architecture and Patch Merging

SW-MSA is embedded within a hierarchical transformer powered by patch merging operations. Initially, images are split into patches (tokens). As depth increases, adjacent patches are merged (e.g., $2\times2$, reducing resolution and increasing channel dimension), forming a feature pyramid. This hierarchical design supports multi-scale modeling critical for dense prediction tasks (object detection, semantic segmentation) and ensures that model complexity grows linearly with image size.

## 4. Efficiency and Throughput Advantages

By restricting attention computation to local windows and using shifting only for cross-window mixing, SW-MSA achieves throughput superior to both global attention and sliding window approaches. For example, Swin-T registers 755 images/sec on a V100 GPU, compared to significantly lower throughput for ViT or naive sliding window models. Cyclic-shifting, used for efficient implementation, minimizes memory overhead.

SW-MSA thus offers a balance between local computation and effective contextual aggregation, resulting in favorable scalability for high-resolution inputs.

## 5. Empirical Performance Across Vision Benchmarks

Ablation studies reported in [2103.14030] established the effectiveness of shifted windows:
- Swin-T with shifted windows demonstrates $+1.1\%$ top-1 accuracy on ImageNet-1K versus non-shifted local attention
- On COCO, object detection box AP and mask AP improve by $+2.8$ and $+2.2$ points, respectively
- Semantic segmentation on ADE20K gains $+2.8$ mIoU

General-purpose backbone capability is validated by state-of-the-art results across image classification (up to $87.3\%$ top-1 accuracy), detection ($58.7$ box AP), and segmentation ($53.5$ mIoU), improving prior benchmarks by substantial margins (+2.7 box AP, +2.6 mask AP, +3.2 mIoU).

## 6. Contextual Innovations and Extensions

The paper reports that the shifted window mechanism extends naturally to architectures beyond canonical transformers, such as all-MLP models. SW-MSA provides a parameter-efficient route for multiscale modeling, competitive with convolutional and global attention-based designs. Notably, the inclusion of relative position bias $B$ in the attention scores encodes spatial relationships vital for vision tasks.

A plausible implication is that shifted window mechanisms could be generalized further, e.g., via adaptive or multi-scale windowing (see dynamic window strategies [2203.12856]), or through context-injected variants in medical imaging (as in CSW-SA [2401.13049]).

## 7. Summary and Significance

SW-MSA is distinguished by its combination of local computational efficiency, cross-window contextual connectivity via window shifting, incorporation of hierarchical patch merging, and superior empirical results. The approach marks a substantive advance in vision transformer design, reconciling the demands of spatial locality, scalability, and high performance across a diverse range of visual tasks, as extensively validated in [2103.14030].

Source: https://www.emergentmind.com/topics/shifted-window-multi-head-self-attention-sw-msa