Swin-Res-Net: Hybrid Transformer & Residual Design

• Swin-Res-Net is a hybrid architecture that combines Swin Transformer’s shifted-window self-attention with ResNet-style residual connections for multi-scale feature extraction.
• It fuses local convolutional details and long-range dependencies, enabling stable optimization and enhanced performance in tasks like PET reconstruction, retinal segmentation, super-resolution, and image fusion.
• The design employs dual residual structures—internal transformer residuals and external skip connections—to effectively merge global context with fine local details.

Searching arXiv for papers and usage of the term "Swin-Res-Net" to ground the article. Swin-Res-Net is not a single canonical architecture but a family of hybrid designs that combine Swin Transformer components with ResNet-style residual learning. Across the literature, the term denotes models in which shifted-window self-attention supplies long-range or global-context modeling, while convolutional or residual pathways preserve local structure, shallow features, or stable optimization. In some works, Swin-Res-Net refers to a concrete named module for retinal vessel segmentation (Yang et al., 2024); in others, it is an accurate architectural characterization of a residual Swin-Transformer regularizer for PET reconstruction (Hu et al., 2022), an enhanced Swin-based super-resolution network (Huang et al., 2023), or a residual Swin Transformer fusion network for infrared-visible image fusion (Wang et al., 2022). The broader background is the Swin Transformer itself, which was introduced as a hierarchical vision Transformer using shifted windows and designed to occupy the same systems role as a general-purpose ResNet-like backbone (Liu et al., 2021).

1. Definition and conceptual scope

The defining idea of Swin-Res-Net is the combination of two architectural principles. The first is the Swin Transformer principle: window-based multi-head self-attention with shifted windows, hierarchical staging, and linear computational complexity with respect to image size (Liu et al., 2021). The second is the ResNet principle: explicit residual or skip connections of the form $y = x + \mathcal{F}(x)$, used to stabilize optimization, preserve shallow information, and learn residual corrections rather than absolute outputs.

In the PET reconstruction literature, this hybrid appears as a “residual swin-transformer based regularizer (RSTR),” described as a residual block with a Swin Transformer Layer and two convolutional layers (Hu et al., 2022). In retinal vessel segmentation, “Swin-Res-Net” is the explicit model name for a U-Net-based network with two coordinated encoders, one based on Swin Transformer and the other on Res2Net, coupled through a two-path interactive fusion module (Yang et al., 2024). In super-resolution, the term is not the formal model name, but the proposed ESTN is explicitly interpretable as a Swin-Res-Net because it alternates local and global feature aggregation inside a residual-style reconstruction pipeline (Huang et al., 2023). In image fusion, SwinFuse is a residual Swin Transformer network whose backbone is built from Residual Swin Transformer Blocks (RSTBs) (Wang et al., 2022).

This variation suggests that “Swin-Res-Net” is best understood as an architectural pattern rather than a standardized model definition. A plausible implication is that the term designates any network in which Swin-style attention and ResNet-style residual pathways are jointly treated as first-class building blocks, with task-specific adaptations determining whether the result resembles an encoder-decoder, an unrolled inverse-problem solver, a restoration pipeline, or a fusion backbone.

2. Architectural foundations: Swin Transformer and residual learning

The Swin component originates in “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows” (Liu et al., 2021). Swin is hierarchical and multi-stage, with patch embedding followed by four stages whose resolutions typically follow the sequence $\frac{H}{4}\times\frac{W}{4}$, $\frac{H}{8}\times\frac{W}{8}$, $\frac{H}{16}\times\frac{W}{16}$, and $\frac{H}{32}\times\frac{W}{32}$. Between stages, patch merging reduces resolution and increases channel dimensionality. Within a stage, Swin blocks alternate window-based self-attention and shifted-window self-attention: $$\hat{\mathbf{z}}^l = \text{W-MSA}(\text{LN}(\mathbf{z}^{l-1})) + \mathbf{z}^{l-1},$$

$$\mathbf{z}^l = \text{MLP}(\text{LN}(\hat{\mathbf{z}}^l)) + \hat{\mathbf{z}}^l,$$

$$\hat{\mathbf{z}}^{l+1} = \text{SW-MSA}(\text{LN}(\mathbf{z}^l)) + \mathbf{z}^l,$$

$$\mathbf{z}^{l+1} = \text{MLP}(\text{LN}(\hat{\mathbf{z}}^{l+1})) + \hat{\mathbf{z}}^{l+1}.$$

The shifted window mechanism limits attention computation to non-overlapping local windows while also allowing cross-window connection, which gives Swin linear complexity with respect to image size and makes it directly compatible with dense prediction backbones (Liu et al., 2021). This is the principal reason Swin can replace ResNet in pipelines that expect multi-scale features with strides $4, 8, 16, 32$.

The residual component enters at multiple levels. First, Swin blocks themselves use transformer-standard residual routes around both attention and MLP sublayers (Liu et al., 2021). Second, Swin-Res-Net variants add outer residual connections around larger computational units. In PET RSTR, the final output is

$\frac{H}{4}\times\frac{W}{4}$0

which creates an external residual around a Conv–Swin–Conv sequence (Hu et al., 2022). In SwinFuse, an entire RSTB wraps several Swin Transformer layers: $\frac{H}{4}\times\frac{W}{4}$1 In ESTN for super-resolution, residuals appear in local shift-convolution stages, channel-attention stages, block-level aggregation, and the global shallow-to-deep fusion before upsampling (Huang et al., 2023).

This dual residual structure—internal residuals inside transformer layers and external residuals around hybrid blocks—is one of the most stable recurring signatures of Swin-Res-Net designs (Hu et al., 2022).

3. Core design patterns across implementations

Although the instantiations differ substantially by task, the literature reveals a small set of recurring design patterns.

Pattern	Description	Representative paper
Conv–Swin–Conv residual block	Shallow local features via convolution, deep/global features via Swin, output fused by residual skip	(Hu et al., 2022)
Dual-path Swin + CNN encoder	Swin path for global context, Res2Net path for multi-scale convolutional detail	(Yang et al., 2024)
Alternating local-global aggregation	Shift convolution, block sparse global perception, multi-scale window attention, channel attention	(Huang et al., 2023)
Residual Swin Transformer stack	Several Swin layers wrapped by an outer residual block	(Wang et al., 2022)

The first pattern is the clearest “Swin-Res block” in the narrow sense. In TransEM, a $\frac{H}{4}\times\frac{W}{4}$2 convolution extracts shallow features, a Swin Transformer Layer performs deep feature extraction, and a second $\frac{H}{4}\times\frac{W}{4}$3 convolution aggregates the result with a residual learning operation (Hu et al., 2022). The architecture is described as a hybrid of Swin Transformer and ResNet because the block begins with local convolutional feature extraction, inserts transformer-based attention in the middle, and ends with a ResNet-style external skip.

The second pattern appears in retinal vessel segmentation. Here the model does not merely insert Swin into a residual block; it creates two coordinated encoders. One encoder is hierarchical Swin Transformer, with patch partition, linear embedding, Swin Transformer blocks, and patch merging across four stages. The other encoder is based on Res2Net residual blocks, whose multi-scale hierarchical splitting enlarges the receptive field of the convolutional kernel (Yang et al., 2024). Fusion occurs stage by stage via Fu-Blocks, so the network’s “Swin-Res-Net” identity derives from the coordinated interaction between global-attention and multi-scale residual-convolution branches rather than from a single block motif.

The third pattern is represented by ESTN for super-resolution. Its Enhanced Swin Transformer Module alternates local feature extraction through shift convolution and global feature extraction through a Block Sparse Global-Awareness Module, Window Multi-Scale Self-Attention, Shifted Window Multi-Scale Self-Attention, and a Low-Parameter Residual Channel Attention Block (Huang et al., 2023). The result is a local–global–local–global rhythm wrapped in residual learning at multiple scales.

The fourth pattern is the purest residual-Swin interpretation. SwinFuse uses a “fully attentional feature encoding backbone” in which the main units are Residual Swin Transformer Blocks, each consisting of several Swin Transformer layers plus an external residual skip (Wang et al., 2022). Unlike canonical hierarchical Swin, this design uses patch size $\frac{H}{4}\times\frac{W}{4}$4, fixed resolution, and a thin reconstruction head, showing that Swin-Res-Net need not preserve the original stagewise downsampling strategy.

4. Task-specific realizations

PET image reconstruction

In “TransEM:Residual Swin-Transformer based regularized PET image reconstruction” (Hu et al., 2022), Swin-Res-Net appears as a learned regularizer embedded in an iterative ML-EM reconstruction algorithm. PET data are modeled as Poisson: $\frac{H}{4}\times\frac{W}{4}$5 and reconstruction is posed as

$\frac{H}{4}\times\frac{W}{4}$6

Using Forward–Backward Splitting, the classical regularizer step is replaced by the residual Swin-transformer regularizer: $\frac{H}{4}\times\frac{W}{4}$7 Each unrolled TransEM block contains an EM update, the RSTR regularizer, and an analytic fusion step that combines the EM estimate and the regularized reference image (Hu et al., 2022).

Architecturally, RSTR is a residual block with a Swin Transformer Layer and two convolutional layers. Given input $\frac{H}{4}\times\frac{W}{4}$8,

$\frac{H}{4}\times\frac{W}{4}$9

$\frac{H}{8}\times\frac{W}{8}$0

$\frac{H}{8}\times\frac{W}{8}$1

$\frac{H}{8}\times\frac{W}{8}$2

The window size is $\frac{H}{8}\times\frac{W}{8}$3, LayerNorm is applied before MSA and MLP, and the MLP uses GELU (Hu et al., 2022). The model is trained in PyTorch 1.7 with Adam, learning rate $\frac{H}{8}\times\frac{W}{8}$4, batch size 4, and unrolling into 60 blocks, corresponding to 10 EM iterations and 6 ordered subsets. On realistic 3D brain simulated low-count data, TransEM is reported to achieve the highest PSNR, SSIM, and MCRC across most count levels and to outperform OSEM, MAPEM, DeepPET, and FBSEM (Hu et al., 2022).

Retinal vessel segmentation

In “Enhancing Retinal Vascular Structure Segmentation in Images With a Novel Design Two-Path Interactive Fusion Module Model” (Yang et al., 2024), Swin-Res-Net is a named retinal vessel segmentation model. It is motivated by the difficulty of recovering fine micro-vessels after repeated encoder downsampling in conventional autoencoding architectures. The architecture is U-Net-based and contains a feature extraction encoder, a redundant information reduction module, and a decoder (Yang et al., 2024).

The encoder has four stages. At each stage, a Swin Transformer path runs in parallel with a Res2Net residual block path, and their outputs are fused through a Fu-Block. The Swin path begins with non-overlapping $\frac{H}{8}\times\frac{W}{8}$5 patches, each producing a token of dimension $\frac{H}{8}\times\frac{W}{8}$6, followed by linear embedding to dimension $\frac{H}{8}\times\frac{W}{8}$7, Swin Transformer blocks, and patch merging. The Res2Net path uses CBR, MaxPool, and stagewise stacks of Res2Net blocks with repetition counts $\frac{H}{8}\times\frac{W}{8}$8 (Yang et al., 2024).

The redundant information elimination module lies between encoder and decoder. It upsamples deeper features by bilinear interpolation and computes absolute element-wise differences between aligned layers: $\frac{H}{8}\times\frac{W}{8}$9 with subsequent repeated difference operations across scales (Yang et al., 2024). The model uses BCELoss, Adam, initial learning rate $\frac{H}{16}\times\frac{W}{16}$0, weight decay $\frac{H}{16}\times\frac{W}{16}$1, CosineAnnealingLR, 40 epochs, and a post-processing threshold of 0.5.

Performance is reported on CHASE-DB1, DRIVE, and STARE. The AUC values are 0.9956, 0.9931, and 0.9946, respectively, and the model is reported to outperform alternative architectures in both IoU and F1 measure metrics (Yang et al., 2024). On CHASE-DB1, Swin-Res-Net achieves specificity 0.9918, accuracy 0.9813, AUC 0.9956, F1 0.8665, and IoU 0.7646; on DRIVE, specificity 0.9872, accuracy 0.9728, AUC 0.9931, F1 0.8394, and IoU 0.7234; on STARE, sensitivity 0.8383, specificity 0.9892, accuracy 0.9779, AUC 0.9946, F1 0.8498, and IoU 0.7389 (Yang et al., 2024).

Single-image super-resolution

In “Image Super-resolution Reconstruction Network based on Enhanced Swin Transformer via Alternating Aggregation of Local-Global Features” (Huang et al., 2023), ESTN is described by a three-stage super-resolution pipeline: Shallow Feature Extraction Module, Deep Feature Extraction Module, and Upsampling Module. The global residual relation is

$\frac{H}{16}\times\frac{W}{16}$2

where $\frac{H}{16}\times\frac{W}{16}$3 is the shallow feature and $\frac{H}{16}\times\frac{W}{16}$4 the final deep feature (Huang et al., 2023).

Its core unit, the Enhanced Swin Transformer Module, alternates local and global stages. Local stages use shift convolution and residual connection; global stages use a Block Sparse Global-Awareness Module, Window Multi-Scale Self-Attention, Shifted Window Multi-Scale Self-Attention, and a Low-Parameter Residual Channel Attention Block (Huang et al., 2023). Training uses L1 loss, Adam, initial learning rate 0.0002, halving at iterations 250, 400, 425, 450, and 475, with total 500 iterations. The network uses 12 ESTM blocks, channel dimension $\frac{H}{16}\times\frac{W}{16}$5, BSGM window size $\frac{H}{16}\times\frac{W}{16}$6, and attention window sizes $\frac{H}{16}\times\frac{W}{16}$7, $\frac{H}{16}\times\frac{W}{16}$8, and $\frac{H}{16}\times\frac{W}{16}$9 (Huang et al., 2023).

On benchmark datasets, ESTN reports higher PSNR and SSIM than SwinIR-light and ELAN-light across $\frac{H}{32}\times\frac{W}{32}$0, $\frac{H}{32}\times\frac{W}{32}$1, and $\frac{H}{32}\times\frac{W}{32}$2 upscaling. For example, on Manga109 at $\frac{H}{32}\times\frac{W}{32}$3, ESTN reports 31.13 dB / 0.9166, versus 30.92 / 0.9150 for both SwinIR-light and ELAN-light; on Urban100 at $\frac{H}{32}\times\frac{W}{32}$4, ESTN reports 26.67 / 0.8040 versus 26.47 / 0.7980 for SwinIR-light (Huang et al., 2023).

Infrared-visible image fusion

In “SwinFuse: A Residual Swin Transformer Fusion Network for Infrared and Visible Images” (Wang et al., 2022), the architecture contains a global feature extraction module, an L1-norm-based fusion layer, and a feature reconstruction module. The backbone uses three Residual Swin Transformer Blocks, each with six Swin Transformer layers, window size $\frac{H}{32}\times\frac{W}{32}$5, and head counts 1, 2, and 4 across the three blocks (Wang et al., 2022). Input images are projected by a single $\frac{H}{32}\times\frac{W}{32}$6 convolution to channel dimension $\frac{H}{32}\times\frac{W}{32}$7, flattened, processed independently through the RSTB backbone, fused by row-wise and column-wise L1-norm activity measures, and reconstructed by a final $\frac{H}{32}\times\frac{W}{32}$8 convolution with Tanh.

Training is performed as an autoencoder on MS-COCO using Adam, learning rate $\frac{H}{32}\times\frac{W}{32}$9, batch size 4, 50 epochs, and a combined SSIM-plus-L1 objective

$$\hat{\mathbf{z}}^l = \text{W-MSA}(\text{LN}(\mathbf{z}^{l-1})) + \mathbf{z}^{l-1},$$0

with $$\hat{\mathbf{z}}^l = \text{W-MSA}(\text{LN}(\mathbf{z}^{l-1})) + \mathbf{z}^{l-1},$$1 selected in ablation (Wang et al., 2022). Across TNO, Roadscene, and OTCBVS, SwinFuse is reported as best for AG, SF, SD, MS_SSIM, SCD, and VIFF on all three datasets, while being second-best or third-best on MI and FMI_w depending on the dataset (Wang et al., 2022).

5. Functional rationale and empirical behavior

The main functional rationale of Swin-Res-Net is the complementarity between local convolutional inductive bias and shifted-window attention. Convolution is repeatedly characterized as local: in PET reconstruction, the paper explicitly states that the local characteristics of the convolution operator potentially limit image quality (Hu et al., 2022); in retinal vessel segmentation, standard convolutional encoders are said to lose semantic signal from micro-vessels during repeated downsampling and to struggle with long-range dependencies (Yang et al., 2024); in super-resolution, the authors argue that standard Swin Transformer attention focuses on long-range relationships but ignores local features and channel interactions, which motivates alternating local-global aggregation (Huang et al., 2023).

The Swin component addresses these limitations by modeling non-local or broader contextual structure. In TransEM, Swin is used to capture long-range dependencies that help distinguish true structures from noise (Hu et al., 2022). In retinal segmentation, Swin is said to provide a broader global receptive field, reduce network complexity through shifted windows with displacement for partitioning, and accelerate model convergence (Yang et al., 2024). In SwinFuse, the fully attentional backbone is designed to model long-range dependency and is presented as having stronger representation ability than convolutional neural networks (Wang et al., 2022).

Residual learning supplies a second, equally important function. In PET reconstruction, the ablation study is reported to show that removing residual connections can cause convergence issues and poorer reconstruction quality (Hu et al., 2022). In SwinFuse, block-level residuals improve AG, SF, SD, MS_SSIM, FMI_w, SCD, and VIFF in the ablation table (Wang et al., 2022). In ESTN, residual design is pervasive, from shift-convolution stages to global shallow–deep feature fusion, and is associated with stable optimization and efficient correction learning (Huang et al., 2023).

This suggests that Swin-Res-Net should not be reduced to “Swin plus skip connection.” In the cited works, the residual pathways are not merely auxiliary optimization aids; they govern how shallow spatial detail, intermediate feature refinements, and transformer-derived global context are recombined. A plausible implication is that the empirical effectiveness of these models depends as much on residual routing and feature fusion topology as on self-attention itself.

6. Ambiguities, misconceptions, and relation to adjacent architectures

A common misconception is that Swin-Res-Net names a universally recognized backbone analogous to ResNet-50 or Swin-T. The literature surveyed here does not support that interpretation. Instead, the term is used in at least four distinct senses: a residual Swin-transformer regularizer inside an unrolled PET solver (Hu et al., 2022), a dual-path retinal segmentation network that explicitly bears the name Swin-Res-Net (Yang et al., 2024), a useful characterization of an enhanced Swin super-resolution model (Huang et al., 2023), and a residual Swin transformer fusion network (Wang et al., 2022).

A second misconception is that Swin-Res-Net simply means replacing every convolutional residual block with a standard Swin block. The foundational Swin Transformer paper does show that Swin can be used as a direct drop-in replacement for ResNet in detection and segmentation pipelines (Liu et al., 2021). For example, under Cascade Mask R-CNN, Swin-T reports AP$$\hat{\mathbf{z}}^l = \text{W-MSA}(\text{LN}(\mathbf{z}^{l-1})) + \mathbf{z}^{l-1},$$2 = 50.5 and AP$$\hat{\mathbf{z}}^l = \text{W-MSA}(\text{LN}(\mathbf{z}^{l-1})) + \mathbf{z}^{l-1},$$3 = 43.7 versus 46.3 and 40.1 for ResNet-50, with similar parameter counts and FLOPs (Liu et al., 2021). Under UPerNet on ADE20K, Swin-S reports 49.3 mIoU versus 44.9 for ResNet-101 (Liu et al., 2021). However, the application-specific papers considered here generally go further: they retain convolutional modules, multi-scale CNN branches, or residual wrappers around transformer stacks, rather than performing a pure one-for-one backbone substitution.

A third misconception is that all Swin-Res-Net variants are hierarchical in the original Swin sense. SwinFuse is not: it uses patch size $$\hat{\mathbf{z}}^l = \text{W-MSA}(\text{LN}(\mathbf{z}^{l-1})) + \mathbf{z}^{l-1},$$4, no patch merging, and constant spatial resolution throughout the backbone (Wang et al., 2022). Conversely, the retinal Swin-Res-Net preserves the full four-stage hierarchy of the Swin encoder and couples it with a parallel Res2Net encoder (Yang et al., 2024). The PET RSTR is even more specialized, operating as a regularization block inside iterative reconstruction rather than as an ordinary feedforward backbone (Hu et al., 2022).

The relationship to Res2Net also requires precision. In the retinal paper, Swin-Res-Net includes interactive fusion with a functional module in the Res2Net architecture, and Res2Net is responsible for multi-scale convolutional feature extraction (Yang et al., 2024). That does not imply that all Swin-Res-Net architectures use Res2Net. In the other cited works, the residual component is implemented through ordinary convolutional layers, shift convolutions, or outer residual skips around transformer layers rather than through Res2Net specifically (Hu et al., 2022, Huang et al., 2023, Wang et al., 2022).

7. Significance and prospective directions

The significance of Swin-Res-Net lies in its role as a transferable hybridization strategy. Across inverse reconstruction, segmentation, super-resolution, and multimodal fusion, the same design thesis recurs: local structure and shallow detail are preserved or emphasized by residual and convolutional pathways, while shifted-window attention supplies scalable non-local modeling (Hu et al., 2022, Yang et al., 2024, Huang et al., 2023, Wang et al., 2022).

The broader Swin literature supports this interpretation. Swin Transformer was introduced precisely to provide a hierarchical, multi-scale, vision-native transformer backbone that could replace ResNet across classification, detection, and segmentation (Liu et al., 2021). Application papers then reinterpret that backbone principle in domain-specific ways. In PET reconstruction, Swin-Res-Net becomes a learned regularizer coupled to the forward physics and EM update. In retinal vessel segmentation, it becomes a dual-encoder U-Net variant tuned for micro-vessel localization and separation. In super-resolution, it becomes a residual restoration pipeline with alternating local and global feature aggregation. In image fusion, it becomes a residual fully attentional encoder paired with hand-crafted activity-based fusion.

Future directions are stated only selectively in the source material. The retinal vessel paper notes an intention to extend the method to diverse data modalities (Yang et al., 2024). SwinFuse identifies the current fusion strategy as hand-crafted and indicates future development toward end-to-end models and other fusion tasks such as multi-focus and multi-exposure fusion (Wang et al., 2022). More generally, this suggests that Swin-Res-Net is likely to remain a productive design label wherever a task demands both efficient long-range dependency modeling and stable residual integration of multi-scale local detail.

In that sense, Swin-Res-Net is best regarded not as a fixed network specification but as a research lineage: a class of architectures that operationalize the compatibility between Swin’s shifted-window transformer mechanics and ResNet’s residual optimization geometry.