Residual Swin Transformer Blocks (RSTBs)

• Residual Swin Transformer Blocks (RSTBs) are hierarchical modules that leverage localized window and shifted window self-attention to capture both local and global dependencies efficiently.
• They integrate convolutional input lifting, multi-head self-attention layers with residual connections, and MLPs to enable stable training and expressive feature extraction.
• RSTBs have been successfully applied in image restoration, medical reconstruction, and segmentation tasks, demonstrating improvements in metrics like PSNR and SSIM compared to conventional methods.

Residual Swin Transformer Blocks (RSTBs) are hierarchical attention modules central to the Swin Transformer framework, designed to overcome the quadratic cost of global self-attention in standard Vision Transformers (ViTs) by localizing attention computation within windows and enabling cross-window information flow via spatial shifts. RSTBs combine window-based and shifted-window multi-head self-attention with internal residual connections to propagate both local and non-local dependencies efficiently, supporting stable training and expressive feature extraction. They are widely used in advanced image restoration, medical reconstruction, and segmentation networks, where their architectural properties address the limitations of pure convolutional and transformer approaches in capturing both fine texture detail and global context (Huang et al., 2022, Liang et al., 2021, Hu et al., 2022, Wang et al., 2022, Naz et al., 9 Dec 2025).

1. Structural Foundations of the RSTB

In all major implementations, an RSTB is a composite block that encapsulates several core operations:

• Input Lifting: A 3×3 convolution (stride 1, padding 1, no activation) projects the input feature map $X_0 \in \mathbb{R}^{H \times W \times C}$ to a latent space $X_1$ of dimension $C$.
• Swin Transformer Layers (STLs): $N$ successive STLs process $X_1$. Each STL consists of:
  • Pre-layernorm ($\mathrm{LN}$) normalization.
  • Window-based multi-head self-attention (W-MSA) or shifted window multi-head self-attention (SW-MSA) alternated per layer, exploiting spatial locality.
  • DropPath stochastic depth regularization and residual skip connections.
  • A two-layer MLP (hidden dimension $r \cdot C$, typically $r=4$) applied with skip connection.
• Block-level Residual: After $N$ STLs, a 3×3 convolution fuses features, and the result is added to $X_1$.
• Output Projection: An additional 3×3 convolution projects the features back to the channel dimension of the input.

The generic RSTB forward computation (as described in (Huang et al., 2022, Liang et al., 2021)) is:

$X_1$0

2. Window-Based and Shifted Window Self-Attention

A central innovation within RSTBs is the localization of self-attention computation:

• Window Partitioning: The input tensor is divided into non-overlapping windows of size $X_1$1. Within each window, multi-head self-attention is performed independently. For window $X_1$2, $X_1$3, attention proceeds via

$X_1$4

$X_1$5

where $X_1$6 is the head dimension, $X_1$7 is a learnable relative positional bias (Huang et al., 2022, Liang et al., 2021, Wang et al., 2022, Naz et al., 9 Dec 2025).

• Shifted Windows: On alternating layers, the feature map is cyclically shifted by $X_1$8. After the shift, standard W-MSA is applied, but with masked attention to prevent spatial leakage. This shift propagates information across window boundaries, allowing global context aggregation with linear complexity in image size.

3. Residual Pathways and Layer Normalization

Residual connections are tightly integrated at multiple granularities:

• Internal Residuals: Each MSA and MLP sublayer is wrapped with a skip connection, enabling uninterrupted gradient flow, promoting stable optimization, and alleviating vanishing gradients in deep cascades (Hu et al., 2022, Naz et al., 9 Dec 2025).
• Block-Level Residual: The skip from initial input to output at the block level constrains the learning dynamics to predict only the correction, complementing the locality of windowed attention (Huang et al., 2022, Liang et al., 2021).
• Pre-LayerNorm: All attention and MLP sublayers use layer normalization on their inputs.

This structure endows the RSTB with both large effective receptive fields and the ability to preserve local feature integrity.

4. Comparative Analysis with Other Transformer Blocks

A summary comparison is given in the following table:

Block Type	Attention Scope	Computational Cost	Residual Structure
Standard Transformer Block	Global (H×W)	$X_1$9	Layer/block skip
Swin Transformer Block	Local window	$C$0	Layer/block skip
Residual Swin Transformer Block (RSTB)	Window + shifted window	$C$1	Layer/block skip, plus final Conv

RSTB's localized attention, shifted window mechanism, and hierarchical stacking enable global context integration at linear cost, reducing parameter count and accelerating convergence relative to global-attention transformer blocks (Huang et al., 2022, Liang et al., 2021).

5. Hyperparameterization and Implementation Norms

Key hyperparameters, as adopted in influential architectures, are:

• Embedding dimension ($C$2): e.g., 64 (Huang et al., 2022), 180 (Liang et al., 2021), 96 (Wang et al., 2022)
• Number of STLs per RSTB ($C$3 or $C$4): typically 6
• Window size ($C$5): 7 or 8, depending on dataset/task
• Number of heads ($C$6): e.g., 6–8
• MLP hidden dimension: $C$7 with $C$8
• DropPath rate: grows with block depth (e.g., to 0.1)
• LayerNorm $C$9: $N$0

The drop-in architecture for image tasks employs patch embedding, a cascade of RSTBs, and symmetric patch unembedding or reconstruction layers (Huang et al., 2022, Liang et al., 2021, Wang et al., 2022).

6. Integration in Vision Architectures and Empirical Performance

RSTBs are the architectural backbone in domains including:

• Image Restoration: SwinIR organizes deep feature extraction as a cascade of RSTBs. RSTBs in SwinIR enable state-of-the-art results in super-resolution, denoising, and JPEG artifact reduction, while reducing parameters by up to 67% and increasing PSNR by up to 0.45 dB over CNN baselines (Liang et al., 2021).
• Medical Image Reconstruction: SwinMR adopts RSTBs for reconstruction from undersampled k-space data, showing robustness under variable subsampling and noise scenarios (Huang et al., 2022). TransEM integrates RSTRs (RSTBs with shallow feature fusion) as regularizers in PET reconstruction, increasing both PSNR and SSIM relative to CNN-based priors (Hu et al., 2022).
• Multimodal Fusion and Segmentation: SwinFuse employs a stack of RSTBs as a fully attentional feature encoder for infrared/visible image fusion, with specific configurations (e.g., window size 7, heads [1,2,4]) (Wang et al., 2022). Residual-SwinCA-Net integrates RSTBs as global context extractors, outperforming hybrid baselines on lesion segmentation tasks (Naz et al., 9 Dec 2025).

Empirically, RSTBs confer improved boundary sharpness, global structure preservation, and faster convergence relative to pure ViT or CNN configurations (Liang et al., 2021, Wang et al., 2022, Naz et al., 9 Dec 2025).

7. Limitations and Evolving Extensions

RSTBs offer window-based efficiency but are limited by the window size for direct non-local interaction; global dependency mixing requires stacking multiple blocks and alternating the window shift. Recent advances introduce additional structures (e.g., multi-scale channel attention, patch hierarchies) that integrate with or extend the standard RSTB to further improve feature fusion, multi-scale consistency, and class-aware discrimination in medical and low-level vision tasks (Naz et al., 9 Dec 2025).

The modular design, linear computational cost, and effectiveness in stabilizing deep transformer training make RSTBs foundational in both baseline and application-driven transformer models across computer vision and medical imaging (Huang et al., 2022, Liang et al., 2021, Hu et al., 2022, Wang et al., 2022, Naz et al., 9 Dec 2025).