CoSwin: Local-Global Fusion in Vision Transformers

• CoSwin is a hybrid vision-transformer architecture that integrates convolution-based local feature enhancement with Swin Transformer's hierarchical shifted-window design.
• It incorporates a learnable local feature enhancement module in each attention block to capture local details, aiding robust pattern recognition on limited data.
• Experimental results show accuracy improvements up to +4.92% over the baseline, proving its effectiveness on challenging benchmarks like CIFAR-100 and Tiny ImageNet.

Searching arXiv for the primary paper and closely related vision-transformer baselines to ground the article with citations. CoSwin is a hybrid vision-transformer architecture introduced for small-scale vision, defined as image classification settings in which training data are limited and standard vision transformers often underperform because they lack inductive biases that convolutional neural networks naturally provide. It augments the Swin Transformer’s hierarchical shifted window attention with localized convolutional feature learning so that local spatial details and global or long-range dependencies are modeled jointly. In the reported formulation, CoSwin preserves the hierarchical design of Swin Transformer while inserting a learnable local feature enhancement module into each attention block, with the stated goal of improving generalization and robustness on limited-data benchmarks (Khadka et al., 10 Sep 2025, Liu et al., 2021).

1. Motivation and problem setting

The rationale for CoSwin is the reported limitation of pure vision transformers on small datasets. The core argument is that transformer-based models can model global context effectively but often struggle to learn robust local representations when data are limited, particularly because they do not encode locality, translation equivariance, or a strong built-in preference for nearby pixel interactions. This limitation is presented as especially consequential in small-scale vision, where the data do not sufficiently constrain a high-capacity model and an attention-only design may overfit or fail to extract fine-grained patterns reliably (Khadka et al., 10 Sep 2025).

Within this framing, CoSwin is positioned as an augmentation rather than a replacement of hierarchical attention. Swin Transformer already restricts attention to windows and builds a hierarchy, but it remains fundamentally an attention-only design. CoSwin modifies that design by adding a lightweight, learnable convolutional enhancement branch into each attention block. The intended effect is to combine the locality and translation equivariance of convolutional networks with the hierarchical shifted-window mechanism used by Swin Transformer.

This positioning places CoSwin in the broader lineage of vision transformers that emphasize self-attention for visual representation learning, including ViT and data-efficient transformer variants such as DeiT, while directly targeting a setting in which stronger spatial priors are argued to be necessary (Dosovitskiy et al., 2020, Touvron et al., 2020).

2. Hierarchical structure and block-level modification

CoSwin follows the same overall 4-stage hierarchical structure as Swin Transformer. The reported pipeline is: split the image into non-overlapping patches, linearly embed patches into tokens, progressively downsample with patch merging, and apply stacked transformer blocks at each stage (Khadka et al., 10 Sep 2025).

The principal change is at the attention-block level. Instead of relying on window-based multi-head self-attention alone, CoSwin uses a convolution-fused window attention mechanism. Each CoSwin block contains a standard window-based multi-head self-attention branch, a shifted-window attention branch in alternating blocks, and a local feature enhancement module built from convolutions. The paper explicitly describes this as integrating “weighted local feature extraction into the shifted window attention mechanism.”

This design is presented as a correction to a specific limitation of the baseline. Swin’s windowing provides locality in the sense of restricted receptive fields, but the paper distinguishes this from explicit convolutional inductive bias. In CoSwin, locality is not merely an effect of window partitioning; it is explicitly introduced through learnable $3\times 3$ convolutional operators within every attention block. A plausible implication is that the model reduces the burden on attention to discover local structure from scratch.

3. Learnable local feature enhancement module

The local feature enhancement module is the defining architectural addition. Its function is to extract spatially local patterns and fuse them back into the attention pathway. The reported procedure begins by reshaping the token sequence into an “imaginary image” or pseudo-image so that convolution can be applied. The paper defines

$$G_p \in \mathbb{R}^{\sqrt{N}\times \sqrt{N}\times d_{\text{embed}}}.$$

Here the token grid is interpreted as a spatial feature map with channel dimension $d_{\text{embed}}$ (Khadka et al., 10 Sep 2025).

A two-layer $3\times 3$ convolution block is then applied:

$$F_{\rm conv}= \operatorname{Conv}_{3\times3}\left(\sigma\!\left(\operatorname{Conv}_{3\times3}(G_p)\right)\right),$$

where $\sigma(\cdot)$ is ReLU. The first convolution expands channels by 10%, ReLU is applied, and the second convolution projects back to the embedding dimension.

The convolutional output is scaled by a learnable scalar $\Gamma$:

$$F_{\text{weighted\_conv}} = \Gamma \times F_{\text{conv}}.$$

The role of $\Gamma$ is to allow the model to adaptively control how much local convolutional information influences the attention output. The paper treats this learnable balancing as an important distinction from fixed fusion schemes, arguing that different small-scale datasets may require different balances of local and global cues.

The appendix-level interpretation reported for this module is that the fused hypothesis space combines convolutional and attention-based functions. This suggests a bias-variance rationale: attention lowers bias by modeling broader relationships, convolution lowers variance by enforcing local and translational structure, and the learnable fusion weight $\Gamma$ adaptively balances the two.

4. Fusion with window and shifted-window attention

CoSwin modifies the Swin attention block by fusing local convolution features with attention outputs after window partitioning. For ordinary window attention, the paper defines

$$G_p \in \mathbb{R}^{\sqrt{N}\times \sqrt{N}\times d_{\text{embed}}}.$$0

For shifted-window attention, it defines

$$G_p \in \mathbb{R}^{\sqrt{N}\times \sqrt{N}\times d_{\text{embed}}}.$$1

In this notation, $$G_p \in \mathbb{R}^{\sqrt{N}\times \sqrt{N}\times d_{\text{embed}}}.$$2 is the token set in the $$G_p \in \mathbb{R}^{\sqrt{N}\times \sqrt{N}\times d_{\text{embed}}}.$$3-th window, $$G_p \in \mathbb{R}^{\sqrt{N}\times \sqrt{N}\times d_{\text{embed}}}.$$4 is the number of windows, $$G_p \in \mathbb{R}^{\sqrt{N}\times \sqrt{N}\times d_{\text{embed}}}.$$5 denotes the shifted-window partition, and $$G_p \in \mathbb{R}^{\sqrt{N}\times \sqrt{N}\times d_{\text{embed}}}.$$6 is the convolutional feature map scaled by $$G_p \in \mathbb{R}^{\sqrt{N}\times \sqrt{N}\times d_{\text{embed}}}.$$7 (Khadka et al., 10 Sep 2025).

The underlying attention within each window is described as standard multi-head self-attention:

$$G_p \in \mathbb{R}^{\sqrt{N}\times \sqrt{N}\times d_{\text{embed}}}.$$8

with each head given by

$$G_p \in \mathbb{R}^{\sqrt{N}\times \sqrt{N}\times d_{\text{embed}}}.$$9

and single-head attention written as

$d_{\text{embed}}$0

The paper intends $d_{\text{embed}}$1 to represent relative positional bias, consistent with Swin-style attention.

The resulting mechanism can be summarized as a local-global fusion block: self-attention supplies semantic aggregation across windows, while the convolutional branch injects explicit spatial locality. The paper’s distinction from baseline Swin Transformer is therefore fourfold: convolution is fused into every attention block, locality is explicit rather than implicit, a learnable scalar balances local and global cues, and the resulting design is claimed to behave better on small datasets.

5. Experimental protocol and evaluation benchmarks

The reported evaluation covers five image-classification benchmarks: CIFAR-10, CIFAR-100, MNIST, SVHN, and Tiny ImageNet. The paper specifies the benchmark characteristics as follows.

Benchmark	Resolution / training size	Classes
CIFAR-10	$d_{\text{embed}}$2, 50,000 training	10
CIFAR-100	$d_{\text{embed}}$3, 50,000 training	100
MNIST	$d_{\text{embed}}$4, 60,000 training	10
SVHN	$d_{\text{embed}}$5, 73,257 training	10
Tiny ImageNet	$d_{\text{embed}}$6, 100,000 training	200

For images smaller than $d_{\text{embed}}$7, the patch size was reduced to 2; otherwise, the default patch size was 4. The implementation was reported in PyTorch and trained from scratch for 100 epochs using AdamW with weight decay $d_{\text{embed}}$8, batch size 128, initial learning rate $d_{\text{embed}}$9, 10 epochs of warm-up, and stochastic drop rate 0.1. The stated regularization and augmentation components were MixUp, CutMix, Random Erasing, Repeated Augment, AutoAugment, and Stochastic Depth, and the hardware was an NVIDIA V100 GPU (Khadka et al., 10 Sep 2025).

The comparison set includes Swin Transformer as the central baseline, together with ResNet18, ResNet56, MobileNetV2, ViT, DeiT-Tiny, and T2T-ViT. In contextual terms, this places CoSwin against both convolutional and transformer-based families, including token-to-token transformer designs that also seek to improve visual tokenization and representation (Yuan et al., 2021).

6. Reported results, ablations, and interpretation

The principal quantitative finding is that CoSwin improves over the baseline Swin Transformer on all five reported datasets. The paper gives the following accuracies and gains.

Dataset	CoSwin vs Swin Transformer	Gain
CIFAR-10	96.63% vs 94.46%	+2.17%
CIFAR-100	81.64% vs 76.72%	+4.92%
MNIST	99.60% vs 99.50%	+0.10%
SVHN	98.07% vs 97.81%	+0.26%
Tiny ImageNet	65.06% vs 60.59%	+4.47%

The paper also states that CoSwin outperforms other CNN and transformer baselines such as ResNet18, ResNet56, MobileNetV2, ViT, DeiT-Tiny, and T2T-ViT (Khadka et al., 10 Sep 2025).

The interpretation attached to these results emphasizes robustness, generalization, and the utility of local-global fusion. The largest gains occur on CIFAR-100 and Tiny ImageNet, which the paper identifies as more challenging small-scale datasets. This is taken to indicate better generalization under limited-data conditions. The qualitative evidence includes Grad-CAM-based visualization showing that CoSwin focuses more sharply on the main object than ViT, DeiT, or Swin Transformer, which the paper presents as support for improved localization of discriminative regions.

The ablation study is central to the model’s justification. Removing convolution entirely causes noticeable drops relative to the full model: $3\times 3$0 on CIFAR-10, $3\times 3$1 on CIFAR-100, $3\times 3$2 on MNIST, $3\times 3$3 on SVHN, and $3\times 3$4 on Tiny ImageNet. Additional ablations report that removing only one convolution causes a moderate drop and that adding the convolution branch without a trainable weight performs worse than weighted fusion. The full model with both convolutions and the learnable influence weight performs best.

A common misconception would be to treat CoSwin as a departure from hierarchical shifted-window attention. The reported design does not replace that mechanism; it preserves the same hierarchical backbone and augments it with local feature learning. The broader implication drawn in the paper is that local-global feature fusion can materially improve accuracy, robustness, and sample efficiency in vision transformers, especially when datasets are small or image resolution is limited. Code and pretrained weights are reported as available at the project repository linked by the paper (Khadka et al., 10 Sep 2025).