ConSwin: Convolution-Augmented Swin Transformer

• The paper introduces ConSwin as a dual-branch encoder that fuses ResNet and Swin Transformer features using a tanh-based fusion to align local and global cues.
• It leverages a novel fusion strategy to balance detailed texture extraction with holistic contextual understanding, improving road continuity and boundary precision.
• Experiments on benchmark road datasets demonstrate significant gains in F1, IoU, and overall accuracy compared to pure CNN or Transformer-based methods.

Searching arXiv for the ConSwin paper and closely related methods mentioned in the provided material. First, locating the primary ConSwin paper by arXiv ID. Now searching for "TransUNet" on arXiv to identify the cited related method. ConSwin is a convolution-augmented Swin Transformer block introduced as the core encoding module of a road-extraction network for very high resolution remote sensing images. In the authors’ formulation, it is a dual-branch network block that couples a ResNet/CNN branch with a Swin Transformer branch so that local texture and detail cues and global contextual and geometric cues are learned jointly and fused stage by stage for binary semantic segmentation of roads (Chen et al., 2022).

1. Problem setting and motivation

ConSwin was proposed for road extraction from very high resolution aerial or satellite imagery, a task the paper frames as difficult because of substantial intra-class variations, indistinct inter-class distinctions, and occlusions caused by shadows, trees, and buildings (Chen et al., 2022). The target is binary semantic segmentation of roads, and the paper identifies three persistent difficulties: roads of the same semantic class can vary because of pavement materials, lighting, blur, scale, and surface appearance; roads can resemble rooftops, parking lots, bare land, or other man-made structures; and roads are frequently partially hidden, which leads to broken predictions and poor connectivity.

The paper argues that effective road extraction requires two capabilities simultaneously: fine local modeling, to preserve road edges, narrow widths, and texture details, and global contextual and geometric modeling, to reason about long, coherent, connected road structures, especially through occlusion and ambiguous appearance (Chen et al., 2022). In that framing, CNNs or ResNet are strong at extracting local detail features and provide useful inductive biases for relatively small remote-sensing datasets, whereas Transformers, especially Swin Transformer, are better at modeling long-range dependencies and global context.

This leads to the central design claim of ConSwin. The stated gap is not simply to use a CNN and a Transformer together, but to construct a stronger contextual encoder in which the two feature types are learned interactively within each encoding stage. The paper explicitly contrasts this with methods such as TransUNet, which first converts CNN features into Transformer tokens, and methods such as RoadFormer, which rely mainly on Transformer encoders (Chen et al., 2022).

2. Internal definition and module structure

The paper describes ConSwin both as a “dual-branch network block” and as a “convolution-augmented Swin Transformer structure” (Chen et al., 2022). The name refers to convolution or CNN or ResNet augmentation on one side and a Swin Transformer branch on the other.

According to the text and figure description, the ConSwin module contains two branches operating on the same stage input. The CNN branch is implemented with ResNet blocks and is responsible for local feature extraction. The Swin Transformer branch first transforms the input into tokens by Patch Merging, then applies two successive Swin Transformer blocks: the first uses W-MSA and the second uses SW-MSA, with LayerNorm, MLP, and GELU as standard components (Chen et al., 2022).

After the two branches produce features, the Swin branch output is passed through a hyperbolic tangent transformation and then fused with the CNN branch by elementwise addition. The defining fusion rule given by the paper is

$z_j = x_j + \tanh(y_j)$

where $x_j$ is the $j$-th feature from the ResNet or CNN branch, $y_j$ is the $j$-th feature from the Swin Transformer branch, and $z_j$ is the fused output feature (Chen et al., 2022). The fused result is passed to the next encoder stage.

The branch roles are stated clearly. The ResNet or CNN branch captures local texture statistics, edge detail, and fine-grained structure, while the Swin Transformer branch captures global context, geometric relations, and long-range dependencies (Chen et al., 2022). For road extraction, the paper treats this division as meaningful because roads are thin, elongated, and topologically connected objects.

3. Feature alignment and the significance of the $\tanh$ fusion

A central technical claim of ConSwin is that direct fusion of CNN and Swin features produces a feature-scale mismatch. The paper defines the discrepancy between the two branches for sample $i$ as

$$S_i = \frac{1}{N}\sum_{j=1}^{N} \left|x_j - y_j\right| \tag{1}$$

where $S_i$ is the average absolute feature difference for the $x_j$0-th sample, $x_j$1 is the number of features, $x_j$2 is the feature from the ResNet branch, and $x_j$3 is the feature from the Swin branch (Chen et al., 2022).

The reported problem is that direct fusion causes the Transformer branch to dominate training, because Swin features, normalized by LayerNorm, have larger magnitude, while ResNet features tend to be smaller. The paper also states that applying LayerNorm again does not adequately solve this (Chen et al., 2022). Its proposed remedy is to compress the Swin branch using $x_j$4, yielding the fusion equation above.

This mechanism is distinctive within the paper’s presentation of ConSwin. The fusion is not described as a more elaborate attention-gated or cross-attentional operator; the defining operation is additive coupling after $x_j$5-based magnitude compression. A plausible implication is that the authors regard optimization stability and branch balance as at least as important as representational richness in the hybrid block.

The architecture logic is summarized explicitly in the paper: start from an incoming feature map, send it through a CNN or ResNet path and a Swin path in parallel, use Patch Merging and two Swin blocks with W-MSA and SW-MSA in the Transformer path, align the branch outputs spatially and channel-wise, apply $x_j$6 to the Swin output, then compute

$x_j$7

and use the result as the encoded representation for the next level (Chen et al., 2022).

4. Network context: hourglass architecture, FeatConn, and shapConn

ConSwin is not presented as an isolated block. The paper states that the full model is an hourglass-shaped encoder-decoder architecture with a downsampling encoder, an upsampling decoder, and intermediate connections, and that the encoder contains three connected ConSwin modules (Chen et al., 2022). The decoder is described as similar to U-Net, progressively upsampling features and combining them with encoder-side information.

Two additional connection structures are introduced to improve information transmission to the decoder. The first is FeatConn, a feature-enhanced connection used between encoder and decoder at the same level, functionally similar to a skip connection but designed to better transmit local information. Its operation is

$x_j$8

where $x_j$9 is the input feature map from the encoder, $j$0 and $j$1 are channelwise average and maximum maps, and $j$2 is the enhanced feature sent to the decoder (Chen et al., 2022). The paper interprets this as transferring channel-statistical descriptors that preserve texture and detail cues useful for reconstructing thin road structures.

The second is shapConn, a shape-augmented connection inserted in the middle of the hourglass network. It first produces a low-resolution segmentation map using a $j$3 convolution and a BCE loss, then computes Sobel derivatives in the $j$4- and $j$5-directions to obtain directional features:

$j$6

with Sobel kernels

$j$7

where $j$8 is the intermediate low-resolution segmentation map (Chen et al., 2022). The purpose is to encode road orientation information and improve continuity and topology.

The paper’s own summary is that shapConn enhances geometric orientation features and FeatConn enhances texture features (Chen et al., 2022). Together, these connections complement ConSwin by improving the transmission of texture and structural detail information to the decoder.

5. Mathematical formulation, loss design, and implementation details

Beyond the branch-discrepancy measure and the ConSwin fusion rule, the paper specifies the training objective for the complete network. The final segmentation loss is given as a weighted cross-entropy style formula with an added Smooth-L1 regularizer:

$j$9

where $y_j$0 is the total number of pixels, $y_j$1 is the ground-truth label of pixel $y_j$2, $y_j$3 is the predicted probability for pixel $y_j$4, $y_j$5 is a class-balancing weight related to the road-pixel proportion, $y_j$6 is a constant hyperparameter, and $y_j$7 is the Smooth-L1 regularization term (Chen et al., 2022). The stated intent is to handle road-background imbalance and stabilize training.

Because the network includes both the final segmentation output and the intermediate shapConn segmentation supervision, the total loss combines $y_j$8 and $y_j$9 using uncertainty weighting:

$j$0

where $j$1 is the intermediate segmentation loss from shapConn and $j$2 are learnable standard deviations controlling relative task weights (Chen et al., 2022). The paper characterizes this as a way to avoid manual tuning of the weights for the two losses.

Implementation details are reported only partially. The hardware is Ubuntu 16.04 with an NVIDIA 1080Ti. Images are resized or scaled to $j$3. Training uses Adam with an initial learning rate of $j$4, batch size 2 or 3, and data augmentation consisting of random horizontal rotation, random vertical rotation, random scaling, and random hue-saturation-value adjustment (Chen et al., 2022). At inference, “the binary output was generated by averaging each prediction over the augmented image,” which suggests test-time augmentation or averaging, although the exact augmentation set is not detailed.

The paper does not explicitly state whether the ResNet or Swin components in ConSwin were initialized from ImageNet-pretrained weights, and it does not report the number of epochs, learning-rate decay schedule, warmup, weight decay, dropout, or the exact test-time augmentation recipe (Chen et al., 2022). It also does not fully enumerate Swin hyperparameters such as patch size, window size, number of heads, or per-stage depths for the proposed ConSwin module.

6. Datasets, empirical results, and interpretation

The experiments are conducted on two road-extraction benchmarks. The Massachusetts road dataset contains 1108 training images, 14 validation images, and 49 test images; the original image size is $j$5, resized to $j$6, with spatial resolution $j$7 (Chen et al., 2022). The CHN6-CUG road dataset contains 4511 labeled images at $j$8, split into 3608 train and 903 test, derived from original $j$9 images with resolution up to 50 cm/pixel and covering representative cities in China with different urbanization levels (Chen et al., 2022).

On Massachusetts, the proposed method using ResNet34 reports Precision $z_j$0, Recall $z_j$1, F1 $z_j$2, IoU $z_j$3, and OA $z_j$4, outperforming the listed baselines in overall accuracy, IoU, and F1 (Chen et al., 2022). The paper highlights improvements over the second-best methods of $z_j$5 F1 over Swin Transformer, $z_j$6 IoU over Swin Transformer, and $z_j$7 OA.

On CHN6-CUG, the proposed method reports Precision $z_j$8, Recall $z_j$9, F1 $\tanh$0, IoU $\tanh$1, and OA $\tanh$2 (Chen et al., 2022). Compared with the next best metrics, the paper reports $\tanh$3 F1 over DeepLabv3+, $\tanh$4 IoU, and $\tanh$5 OA. The gains are emphasized as especially important on CHN6-CUG because it is the more detailed and difficult dataset.

Ablation studies evaluate combinations of ConSwin, shapConn, and FeatConn. On Massachusetts, the bare baseline has F1 $\tanh$6, IoU $\tanh$7, and OA $\tanh$8, while the best full configuration reaches F1 $\tanh$9, IoU $i$0, and OA $i$1 (Chen et al., 2022). On CHN6-CUG, the baseline has F1 $i$2, IoU $i$3, and OA $i$4, while the full model reaches F1 $i$5, IoU $i$6, and OA $i$7. The paper interprets this as evidence that the hybrid contextual encoding and enhanced connections are especially beneficial when imagery is more complex and detailed.

Qualitative findings are aligned with this interpretation. The paper reports that under tree occlusion the proposed model extracts roads more continuously than Swin Transformer and other baselines; at intersections and blurred boundaries it better maintains connectivity without overbreaking the road; in complex shape layouts, especially on CHN6-CUG, it segments roads more accurately than pure CNN or pure Transformer methods; and under shadows it benefits from combining local and global cues (Chen et al., 2022).

7. Significance, limitations, and terminological clarification

The paper presents ConSwin as more than a generic CNN-Transformer hybrid. Its distinctive features are stage-wise dual-branch encoding, explicit feature-scale alignment through

$i$8

and embedding within a road-specific hourglass architecture that includes FeatConn for texture-detail transmission and shapConn for orientation and shape guidance (Chen et al., 2022). The strongest interpretation supported by the reported evidence is that ConSwin functions as a road-oriented contextual encoder designed to preserve road integrity, connectivity, and boundary quality in difficult very high resolution scenes.

At the same time, the paper leaves several questions open. It does not provide FLOPs, parameter count, memory, or inference-speed comparisons. Since ConSwin is dual-branch, a plausible implication is that it may be more expensive than single-backbone alternatives, but the paper does not quantify this (Chen et al., 2022). The ablation studies show that ConSwin helps, but they do not isolate a CNN-only encoder with the same decoder, a Swin-only encoder with the same decoder, or direct-addition versus $i$9-addition under otherwise identical settings. The paper also does not provide systematic failure-case analysis.

A common source of confusion is the similarity between the names “ConSwin” and “ConSFT.” These denote different methods in different domains. “Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT” introduces ConSFT, short for Conservative Supervised Fine-Tuning, and explicitly is not about ConSwin (Zhang et al., 9 May 2026). ConSwin refers to the convolution-augmented Swin Transformer encoder block for road extraction in very high resolution remote sensing imagery (Chen et al., 2022).