---
title: Patchwise Self-Attention
url: https://www.emergentmind.com/topics/patchwise-self-attention
type: topic
---

# Patchwise Self-Attention

Patchwise self-attention is a class of neural attention mechanisms designed for image recognition and related tasks, wherein attention is computed over local spatial patches, allowing the network to model adaptive, content-dependent weighting patterns within regions of the input feature map. As developed in multiple works, notably in the patchwise Self-Attention Network (SAN) module, patchwise attention generalizes local convolution by replacing translation-invariant weighting with content-adaptive, per-channel vector attention. This mechanism substantially increases the expressive power and flexibility of local aggregation in visual models, as evidenced by empirical and theoretical analyses [2004.13621][1912.01521].

## 1. Mathematical Definition and Mechanism

Patchwise self-attention operates on a feature map $x \in \mathbb{R}^{H \times W \times C}$, where $H$, $W$, and $C$ are spatial height, width, and channel dimensions respectively. For each spatial location $i$, a local patch $R(i) \subset \{(u, v)\}$ is selected (e.g., a $7 \times 7$ window), yielding $x_{R(i)} \in \mathbb{R}^{|R| \times C}$. The new feature at position $i$ is computed as

$$
y_i = \sum_{j \in R(i)} \alpha(x_{R(i)})_j \odot \beta(x_j)
$$

where:

- $\beta: \mathbb{R}^C \to \mathbb{R}^{C/r_2}$ is a linear projection reducing dimensionality (bottleneck factor $r_2$).
- $\alpha(x_{R(i)}) \in \mathbb{R}^{|R| \times (C/r_2)}$ comprises attention weight vectors (one per patch location $j$).
- $\alpha$ is parameterized as $\gamma(\delta(x_{R(i)}))$, where:
  - $\delta: \mathbb{R}^{|R| \times C} \to \mathbb{R}^D$ aggregates the patch into a $D$-dimensional vector.
  - $\gamma: \mathbb{R}^D \to \mathbb{R}^{|R| \times (C/r_2)}$ unfolds to yield attention weights.

Three forms for $\delta$ are employed:
- **Star-product:** $[\varphi(x_i)^\top \psi(x_j)]_{j \in R(i)} \in \mathbb{R}^{|R|}$
- **Clique-product:** $[\varphi(x_j)^\top \psi(x_k)]_{j, k \in R(i)} \in \mathbb{R}^{|R|^2}$
- **Concatenation:** $[\varphi(x_i); \psi(x_j)_{j \in R(i)}] \in \mathbb{R}^{(1+|R|) \cdot (C/r_1)}$

Here, $\varphi,\, \psi: \mathbb{R}^C \to \mathbb{R}^{C/r_1}$ are linear projections, with $r_1$ a bottleneck factor.

After attention, a final Linear layer restores the channel dimension, followed by batch normalization, ReLU activation, and a residual skip connection to the block input [2004.13621].

## 2. The Expressive Power of Patchwise Self-Attention versus Convolution

Whereas standard convolution aggregates features using fixed, translation-invariant kernels $k(t) \in \mathbb{R}^C$ with weights applied solely as a function of relative spatial offset, patchwise self-attention determines aggregation weights $\alpha(x_{R(i)})_j$ as an adaptive function of the local content $x_{R(i)}$. The result is:

- Local weighting adapts to structure such as edges, textures, or object parts.
- Attention weights are per-channel vectors (length $C/r_2$), not scalars shared across all channels.
- No translation invariance: the same offset $j-i$ can have different weights in different contexts.

Any given convolution can be exactly reproduced by patchwise attention by using fixed $\alpha$. However, the converse is not true; patchwise attention can realize operators that are content-conditional, e.g., modulating or zeroing out specific channel groups based on patch context, which is beyond the scope of standard convolutional operations [2004.13621].

## 3. Network Architecture and Integration Strategies

A standard patchwise SAN block constitutes two primary computational streams:

- **Attention-weight computation:** $1 \times 1$ convolutions (bottlenecked by $r_1=16$ by default) to compute $\varphi$, $\psi$, followed by aggregation $\delta$ (with selectable relation function) and mapping $\gamma$ (two Linear$\to$ReLU$\to$Linear layers, $r_2=4$).
- **Value projection:** A $1\times1$ convolution for $\beta$, reducing channels by $r_2$.

Fusion is executed via a Hadamard product of the output of $\alpha$ and $\beta$, followed by batch normalization, ReLU, channel expansion, and residual addition.

The patchwise SAN architecture, exemplified by San15, comprises five stages (stride sequence $\{1,2,2,2,2\}$; channel widths $\{64,256,512,1024,2048\}$). Patch size is $3\times3$ for the initial stage and $7\times7$ afterwards. No explicit multihead splitting is applied; each attention vector is shared over 8-channel groups [2004.13621]. In the Self Attentive Convolution (SAC) formulation, the approach generalizes to arbitrary $n\times m$ kernels with overlapping patches, sliding across the image without explicit partitioning, and supports both single- and multi-head extensions [1912.01521].

## 4. Implementation Hyper-parameters and Complexity Analysis

The principal hyper-parameters for patchwise SAN include training for 100 epochs with batch size 256 (across 8 GPUs), SGD optimizer with momentum $0.9$, weight decay $10^{-4}$, cosine-decayed learning rate (base $0.1$), label smoothing $0.1$, and standard data augmentations (resized $224 \times 224$ crop, random horizontal flip, channel-wise normalization). By default, bottleneck factors $r_1=16$, $r_2=4$ control dimension reduction. Eight channels share the same attention weight vector [2004.13621].

Computational complexity for patchwise operators is dominated by convolutional projection and score computation. For input spatial size $N\times M$ and channels $d$:
- $O(P n m d^2)$ for projections (where $P=N\cdot M$),
- $O(P^2 d)$ for attention-weight computation and weighted sum,
- Memory $O(P^2 + P d)$.
Stride or dilation can reduce the number of attended patch positions $P'$ [1912.01521].

## 5. Benchmark Results and Empirical Insights

Direct comparison of patchwise SANs with convolutional ResNets on ImageNet single-crop validation indicates:

| Method             | Top-1 (%) | Top-5 (%) | Params (M) | FLOPs (G) |
|--------------------|-----------|-----------|------------|-----------|
| ResNet26           | 73.6      | 91.7      | 13.7       | 2.4       |
| SAN10 (patchwise)  | 77.1      | 93.5      | 11.8       | 1.9       |
| ResNet38           | 76.0      | 93.0      | 19.6       | 3.2       |
| SAN15 (patchwise)  | 78.0      | 93.9      | 16.2       | 2.6       |
| ResNet50           | 76.9      | 93.5      | 25.6       | 4.1       |
| SAN19 (patchwise)  | 78.2      | 93.9      | 20.5       | 3.3       |

Patchwise SAN models achieve 1–2% higher top-1 accuracy while requiring 20–40% fewer parameters and FLOPs relative to comparable convnets.

Patchwise SANs exhibit improved robustness: under 180° input rotation, SAN15 accuracy drops from 78.0% to 56.0% (−22.0 pp) versus ResNet38's drop from 76.0% to 52.2% (−23.8 pp); under a PGD $L_\infty$ adversarial attack ($\epsilon=8/255$, 4 steps), ResNet50 top-1 falls to 11.8% (success rate 82.5%), SAN19 to 24.8% (success rate 62.0%). These results suggest patchwise self-attention confers additional robustness to geometric and white-box adversarial attacks [2004.13621].

## 6. Ablations, Parameterizations, and Limitations

Multiple ablations reveal:
- Among relation functions for $\delta$, concatenation yields the highest validation accuracy (79.3% top-1) compared to star (78.7%) or clique (79.1%).
- Two layers for $\gamma$ (Linear$\to$ReLU$\to$Linear) provide optimal depth.
- Distinct $\varphi$, $\psi$, $\beta$ parameters perform better than tied versions.
- Increasing patch (footprint) size from $3 \times 3$ to $7 \times 7$ improves performance then saturates, with limited added FLOPs.

A limitation is the additional implementation complexity and memory overhead incurred by large, fully-connected local attention maps, though modest compared to global pixelwise attention. The module does not employ explicit multihead splitting, relying instead on group sharing of attention weights [2004.13621].

## 7. Connections and Extensions in Related Literature

Patchwise attention subsumes classical convolution as a strict generalization, fully capturing stationary kernels as a limiting case while supporting content-conditioned adaptation. Self Attentive Convolutions (SAC) extend the paradigm further, showing that standard global self-attention is a 1×1 convolution, and that patchwise attention is equivalent to generalizing to $n \times m$ localities. Multiscale SAC (MSAC) computes parallel patchwise attentions over varying scales, laterally concatenating their outputs and fusing via $1\times1$ convolutions. This approach enables simultaneous modeling of local and non-local dependencies without explicit patch partitioning and can be integrated within ResNet- or DenseNet-style backbones. Preliminary experiments, though unpublished in detail, demonstrate that replacing convolutional layers with SAC/MSAC consistently improves classification and segmentation with comparable parameter budgets [1912.01521].

Source: https://www.emergentmind.com/topics/patchwise-self-attention