---
title: Hierarchical Swin Transformer Encoder
url: https://www.emergentmind.com/topics/hierarchical-swin-transformer-encoder
type: topic
---

# Hierarchical Swin Transformer Encoder

A Hierarchical Swin Transformer Encoder is a multi-resolution transformer architecture designed to enable efficient, expressive feature extraction for high-resolution visual restoration tasks, particularly compressed image super-resolution. It uses a multi-branch, multi-scale pathway structure to jointly capture local details and long-range dependencies at several spatial resolutions, leveraging a series of Swin Transformer blocks integrated with CNN-based downsampling, hierarchical feature fusion, and cross-scale interaction mechanisms [2208.09885].

## 1. Multi-Branch Hierarchical Architecture

The encoder begins by extracting three parallel feature branches at decreasing spatial resolutions from a low-resolution compressed image $I_l \in \mathbb{R}^{H \times W \times C}$:
- **High-scale branch:** $F_h = \mathrm{Conv}_{7 \times 7, s=1, p=3}(I_l) \in \mathbb{R}^{H \times W \times 60}$
- **Middle-scale branch:** $F_m = \mathrm{Conv}_{5 \times 5, s=2, p=2}(I_l) \in \mathbb{R}^{H/2 \times W/2 \times 60}$
- **Low-scale branch:** $F_l = \mathrm{Conv}_{3 \times 3, s=2, p=1}(F_m) \in \mathbb{R}^{H/4 \times W/4 \times 60}$

Each branch thus processes the image at a distinct resolution, with 60 embedding channels per feature map. Convolutional strides and kernel sizes are selected to subsample the feature maps hierarchically.

Each scale-specific branch passes through a Feature Enhancement Module (FEM), comprising stacks of Residual Swin Transformer Blocks (RSTBs). The numbers of RSTBs per branch are set to $\{2,4,6\}$ for low, middle, and high scales, respectively.

## 2. Swin Transformer Block Design and Residual Pathways

### 2.1 Residual Swin Transformer Block (RSTB)
Within a FEM, a single RSTB consists of $K=6$ consecutive Swin Transformer Layers (STLs), followed by a $3 \times 3$ convolution and a skip connection:

\[
\begin{aligned}
&\text{for }i=1 \ldots K:~F_i = \mathrm{STL}(F_{i-1}) \\
&F_{\mathrm{out}} = \mathrm{Conv}_{3 \times 3}(F_K) + F_0
\end{aligned}
\]

### 2.2 Swin Transformer Layer (STL)
Each STL processes $X \in \mathbb{R}^{h \times w \times d}$ as follows:
\[
\begin{aligned}
Y &= \mathrm{W\text{-}MSA}(\mathrm{LN}(X)) + X \\
Z &= \mathrm{SW\text{-}MSA}(\mathrm{LN}(Y)) + Y \\
X' &= \mathrm{MLP}(\mathrm{LN}(Z)) + Z
\end{aligned}
\]
- $\mathrm{W\text{-}MSA}$: window-based multi-head self-attention.
- $\mathrm{SW\text{-}MSA}$: shifted window MSA, using cyclic shift and masking.
- $\mathrm{MLP}$: feed-forward network, hidden ratio 4, GELU nonlinearity.

### 2.3 Window-based Self-Attention (W-MSA) and Shifted Window MSA (SW-MSA)
Non-overlapping windows of size $M \times M$ (here $M=8$) are used. In each, $X_w \in \mathbb{R}^{M^2 \times d}$ is projected to queries, keys, and values; $H \approx 6$ attention heads split $d=60$ into $d/H \approx 10$ channels per head. Scaled dot-product attention with learned relative position bias is used:

\[
\mathrm{Attention}(Q,K,V) = \mathrm{Softmax}\left(\frac{QK^T}{\sqrt{d/H}}\right) V
\]

In SW-MSA, prior to windowing, features are cyclically shifted by $(\lfloor M/2\rfloor, \lfloor M/2\rfloor)$. A precomputed mask restricts attention to within shifted windows.

## 3. Cross-Scale Feature Fusion

Following enhancement, the outputs at coarser scales are upsampled using PixelShuffle (sub-pixel convolution, $\times2$ upscaling) and concatenated channel-wise with the finer-scale features:

\[
\begin{aligned}
&F'_m = \mathrm{Conv}_{3 \times 3}(\mathrm{concat}[F_m, \mathrm{PixelShuffle}_2(F^*_l)]) \\
&F'_h = \mathrm{Conv}_{3 \times 3}(\mathrm{concat}[F_h, \mathrm{PixelShuffle}_2(F^*_m)])
\end{aligned}
\]
After final refinement at the highest scale, two further sub-pixel upsampling layers (PixelShuffle $\times 2$ each) followed by a $3 \times 3$ convolution reconstruct a $4 \times$ super-resolved high-resolution output.

Fusion is restricted to adjacent scales only (i.e., no direct low-to-high long skip).

## 4. Mathematical Properties and Implementation

- **Strided convolution** implements patch merging (downsampling), maintaining channel width 60 at all scales.
- **Window size** $M=8$, **heads** $H\approx 6$, **d** $= 60$ (per token).
- **PixelShuffle** is used throughout for spatial upsampling by rearranging features from channel to space.
- Unlike canonical Swin, channel dimension is fixed across scales; only spatial dimension shrinks via stride.

## 5. Distinctive Design Choices and Comparative Analysis

- **CNN-based Multi-Scale Branching:** Rather than Swin’s patch embeddings plus four-level patch merging (with channel doubling per merge), HST extracts three parallel resolutions using fixed-width convolution, fusing them progressively upward.
- **Residual Swin Transformer per Scale:** Local inductive bias (convolutions) is exploited for robust low-level feature extraction, while global self-attention (shifted windows) enables context modeling within each scale.
- **PixelShuffle Fusion:** Finer-scale branches receive contextual cues from coarser-scale branches, increasing effective receptive field without excessively deepening any single branch.
- **Performance Gains:** In ablation, a three-branch HST encoder exceeds one-branch SwinIR-style backbones by up to +0.15 dB PSNR on JPEG Q=40, with consistent SSIM improvements across compression quality levels.

## 6. Practical Usage and Performance

The HST encoder, in conjunction with super-resolution pretraining, reached 5th place (PSNR: 23.51 dB) in the AIM 2022 low-quality compressed image super-resolution challenge. The combination of hierarchical convolutional and transformer-based enhancement, narrow-channel multi-scale design, and cross-scale fusion delivers state-of-the-art compressed image restoration with robust generalization and competitive computational efficiency [2208.09885].

## 7. Broader Context and Implications

The architectural innovations in the Hierarchical Swin Transformer Encoder highlight the trend toward hybrid, multi-scale transformer backbones for challenging low-level vision tasks. By integrating convolutional and attention mechanisms in a hierarchical schema, the encoder realizes both strong locality (necessary for artifact removal) and scalable non-local modeling (critical for reconstruction from highly compressed signals). The empirical evidence underlines the utility of such hybrid designs in domains where purely flat or single-branch transformers struggle to jointly handle heterogeneous, multi-scale distortions. The HST approach is foundational for extensions in other domains—such as video or multimodal hierarchical transformers—where analogous multi-scale fusion and residual self-attention modules have appeared.

Source: https://www.emergentmind.com/topics/hierarchical-swin-transformer-encoder