Patchwise Self-Attention

• Patchwise self-attention is a neural mechanism that computes attention over localized image patches, enabling content-adaptive weighting for enhanced feature extraction.
• It generalizes standard convolution by replacing fixed kernels with dynamic, per-channel weight vectors, resulting in improved accuracy and efficiency as shown in benchmark comparisons.
• Its modular design integrates seamlessly into diverse architectures, offering increased robustness against geometric variations and adversarial attacks in visual recognition tasks.

Patchwise self-attention is a class of neural attention mechanisms designed for image recognition and related tasks, wherein attention is computed over local spatial patches, allowing the network to model adaptive, content-dependent weighting patterns within regions of the input feature map. As developed in multiple works, notably in the patchwise Self-Attention Network (SAN) module, patchwise attention generalizes local convolution by replacing translation-invariant weighting with content-adaptive, per-channel vector attention. This mechanism substantially increases the expressive power and flexibility of local aggregation in visual models, as evidenced by empirical and theoretical analyses (Zhao et al., 2020, Barkan, 2019).

1. Mathematical Definition and Mechanism

Patchwise self-attention operates on a feature map $x \in \mathbb{R}^{H \times W \times C}$, where $H$, $W$, and $C$ are spatial height, width, and channel dimensions respectively. For each spatial location $i$, a local patch $R(i) \subset \{(u, v)\}$ is selected (e.g., a $7 \times 7$ window), yielding $x_{R(i)} \in \mathbb{R}^{|R| \times C}$. The new feature at position $i$ is computed as

$$y_i = \sum_{j \in R(i)} \alpha(x_{R(i)})_j \odot \beta(x_j)$$

where:

• $\beta: \mathbb{R}^C \to \mathbb{R}^{C/r_2}$ is a linear projection reducing dimensionality (bottleneck factor $r_2$).
• $$\alpha(x_{R(i)}) \in \mathbb{R}^{|R| \times (C/r_2)}$$ comprises attention weight vectors (one per patch location $j$).
• $\alpha$ is parameterized as $\gamma(\delta(x_{R(i)}))$, where:
  • $\delta: \mathbb{R}^{|R| \times C} \to \mathbb{R}^D$ aggregates the patch into a $D$-dimensional vector.
  • $$\gamma: \mathbb{R}^D \to \mathbb{R}^{|R| \times (C/r_2)}$$ unfolds to yield attention weights.

Three forms for $\delta$ are employed:

• Star-product: $$[\varphi(x_i)^\top \psi(x_j)]_{j \in R(i)} \in \mathbb{R}^{|R|}$$
• Clique-product: $$[\varphi(x_j)^\top \psi(x_k)]_{j, k \in R(i)} \in \mathbb{R}^{|R|^2}$$
• Concatenation: $$[\varphi(x_i); \psi(x_j)_{j \in R(i)}] \in \mathbb{R}^{(1+|R|) \cdot (C/r_1)}$$

Here, $$\varphi,\, \psi: \mathbb{R}^C \to \mathbb{R}^{C/r_1}$$ are linear projections, with $r_1$ a bottleneck factor.

After attention, a final Linear layer restores the channel dimension, followed by batch normalization, ReLU activation, and a residual skip connection to the block input (Zhao et al., 2020).

2. The Expressive Power of Patchwise Self-Attention versus Convolution

Whereas standard convolution aggregates features using fixed, translation-invariant kernels $k(t) \in \mathbb{R}^C$ with weights applied solely as a function of relative spatial offset, patchwise self-attention determines aggregation weights $\alpha(x_{R(i)})_j$ as an adaptive function of the local content $x_{R(i)}$. The result is:

• Local weighting adapts to structure such as edges, textures, or object parts.
• Attention weights are per-channel vectors (length $C/r_2$), not scalars shared across all channels.
• No translation invariance: the same offset $j-i$ can have different weights in different contexts.

Any given convolution can be exactly reproduced by patchwise attention by using fixed $\alpha$. However, the converse is not true; patchwise attention can realize operators that are content-conditional, e.g., modulating or zeroing out specific channel groups based on patch context, which is beyond the scope of standard convolutional operations (Zhao et al., 2020).

3. Network Architecture and Integration Strategies

A standard patchwise SAN block constitutes two primary computational streams:

• Attention-weight computation: $1 \times 1$ convolutions (bottlenecked by $r_1=16$ by default) to compute $\varphi$, $\psi$, followed by aggregation $\delta$ (with selectable relation function) and mapping $\gamma$ (two Linear$\to$ReLU$\to$Linear layers, $r_2=4$).
• Value projection: A $1\times1$ convolution for $\beta$, reducing channels by $r_2$.

Fusion is executed via a Hadamard product of the output of $\alpha$ and $\beta$, followed by batch normalization, ReLU, channel expansion, and residual addition.

The patchwise SAN architecture, exemplified by San15, comprises five stages (stride sequence $\{1,2,2,2,2\}$; channel widths $\{64,256,512,1024,2048\}$). Patch size is $3\times3$ for the initial stage and $7\times7$ afterwards. No explicit multihead splitting is applied; each attention vector is shared over 8-channel groups (Zhao et al., 2020). In the Self Attentive Convolution (SAC) formulation, the approach generalizes to arbitrary $n\times m$ kernels with overlapping patches, sliding across the image without explicit partitioning, and supports both single- and multi-head extensions (Barkan, 2019).

4. Implementation Hyper-parameters and Complexity Analysis

The principal hyper-parameters for patchwise SAN include training for 100 epochs with batch size 256 (across 8 GPUs), SGD optimizer with momentum $0.9$, weight decay $10^{-4}$, cosine-decayed learning rate (base $0.1$), label smoothing $0.1$, and standard data augmentations (resized $224 \times 224$ crop, random horizontal flip, channel-wise normalization). By default, bottleneck factors $r_1=16$, $r_2=4$ control dimension reduction. Eight channels share the same attention weight vector (Zhao et al., 2020).

Computational complexity for patchwise operators is dominated by convolutional projection and score computation. For input spatial size $N\times M$ and channels $d$:

• $O(P n m d^2)$ for projections (where $P=N\cdot M$),
• $O(P^2 d)$ for attention-weight computation and weighted sum,
• Memory $O(P^2 + P d)$. Stride or dilation can reduce the number of attended patch positions $P'$ (Barkan, 2019).

5. Benchmark Results and Empirical Insights

Direct comparison of patchwise SANs with convolutional ResNets on ImageNet single-crop validation indicates:

Method	Top-1 (%)	Top-5 (%)	Params (M)	FLOPs (G)
ResNet26	73.6	91.7	13.7	2.4
SAN10 (patchwise)	77.1	93.5	11.8	1.9
ResNet38	76.0	93.0	19.6	3.2
SAN15 (patchwise)	78.0	93.9	16.2	2.6
ResNet50	76.9	93.5	25.6	4.1
SAN19 (patchwise)	78.2	93.9	20.5	3.3

Patchwise SAN models achieve 1–2% higher top-1 accuracy while requiring 20–40% fewer parameters and FLOPs relative to comparable convnets.

Patchwise SANs exhibit improved robustness: under 180° input rotation, SAN15 accuracy drops from 78.0% to 56.0% (−22.0 pp) versus ResNet38's drop from 76.0% to 52.2% (−23.8 pp); under a PGD $L_\infty$ adversarial attack ($\epsilon=8/255$, 4 steps), ResNet50 top-1 falls to 11.8% (success rate 82.5%), SAN19 to 24.8% (success rate 62.0%). These results suggest patchwise self-attention confers additional robustness to geometric and white-box adversarial attacks (Zhao et al., 2020).

6. Ablations, Parameterizations, and Limitations

Multiple ablations reveal:

• Among relation functions for $\delta$, concatenation yields the highest validation accuracy (79.3% top-1) compared to star (78.7%) or clique (79.1%).
• Two layers for $\gamma$ (Linear$\to$ReLU$\to$Linear) provide optimal depth.
• Distinct $\varphi$, $\psi$, $\beta$ parameters perform better than tied versions.
• Increasing patch (footprint) size from $3 \times 3$ to $7 \times 7$ improves performance then saturates, with limited added FLOPs.

A limitation is the additional implementation complexity and memory overhead incurred by large, fully-connected local attention maps, though modest compared to global pixelwise attention. The module does not employ explicit multihead splitting, relying instead on group sharing of attention weights (Zhao et al., 2020).

7. Connections and Extensions in Related Literature

Patchwise attention subsumes classical convolution as a strict generalization, fully capturing stationary kernels as a limiting case while supporting content-conditioned adaptation. Self Attentive Convolutions (SAC) extend the paradigm further, showing that standard global self-attention is a 1×1 convolution, and that patchwise attention is equivalent to generalizing to $n \times m$ localities. Multiscale SAC (MSAC) computes parallel patchwise attentions over varying scales, laterally concatenating their outputs and fusing via $1\times1$ convolutions. This approach enables simultaneous modeling of local and non-local dependencies without explicit patch partitioning and can be integrated within ResNet- or DenseNet-style backbones. Preliminary experiments, though unpublished in detail, demonstrate that replacing convolutional layers with SAC/MSAC consistently improves classification and segmentation with comparable parameter budgets (Barkan, 2019).