---
title: 'CoSwin: Local-Global Fusion in Vision Transformers'
url: https://www.emergentmind.com/topics/coswin
type: topic
---

# CoSwin: Local-Global Fusion in Vision Transformers

Searching arXiv for the primary paper and closely related vision-transformer baselines to ground the article with citations.
CoSwin is a hybrid vision-transformer architecture introduced for small-scale vision, defined as image classification settings in which training data are limited and standard vision transformers often underperform because they lack inductive biases that convolutional neural networks naturally provide. It augments the Swin Transformer’s hierarchical shifted window attention with localized convolutional feature learning so that local spatial details and global or long-range dependencies are modeled jointly. In the reported formulation, CoSwin preserves the hierarchical design of Swin Transformer while inserting a learnable local feature enhancement module into each attention block, with the stated goal of improving generalization and robustness on limited-data benchmarks [2509.08959; 2103.14030].

## 1. Motivation and problem setting

The rationale for CoSwin is the reported limitation of pure vision transformers on small datasets. The core argument is that transformer-based models can model global context effectively but often struggle to learn robust local representations when data are limited, particularly because they do not encode locality, translation equivariance, or a strong built-in preference for nearby pixel interactions. This limitation is presented as especially consequential in small-scale vision, where the data do not sufficiently constrain a high-capacity model and an attention-only design may overfit or fail to extract fine-grained patterns reliably [2509.08959].

Within this framing, CoSwin is positioned as an augmentation rather than a replacement of hierarchical attention. Swin Transformer already restricts attention to windows and builds a hierarchy, but it remains fundamentally an attention-only design. CoSwin modifies that design by adding a lightweight, learnable convolutional enhancement branch into each attention block. The intended effect is to combine the locality and translation equivariance of convolutional networks with the hierarchical shifted-window mechanism used by Swin Transformer.

This positioning places CoSwin in the broader lineage of vision transformers that emphasize self-attention for visual representation learning, including ViT and data-efficient transformer variants such as DeiT, while directly targeting a setting in which stronger spatial priors are argued to be necessary [2010.11929; 2012.12877].

## 2. Hierarchical structure and block-level modification

CoSwin follows the same overall 4-stage hierarchical structure as Swin Transformer. The reported pipeline is: split the image into non-overlapping patches, linearly embed patches into tokens, progressively downsample with patch merging, and apply stacked transformer blocks at each stage [2509.08959].

The principal change is at the attention-block level. Instead of relying on window-based multi-head self-attention alone, CoSwin uses a convolution-fused window attention mechanism. Each CoSwin block contains a standard window-based multi-head self-attention branch, a shifted-window attention branch in alternating blocks, and a local feature enhancement module built from convolutions. The paper explicitly describes this as integrating “weighted local feature extraction into the shifted window attention mechanism.”

This design is presented as a correction to a specific limitation of the baseline. Swin’s windowing provides locality in the sense of restricted receptive fields, but the paper distinguishes this from explicit convolutional inductive bias. In CoSwin, locality is not merely an effect of window partitioning; it is explicitly introduced through learnable \(3\times 3\) convolutional operators within every attention block. A plausible implication is that the model reduces the burden on attention to discover local structure from scratch.

## 3. Learnable local feature enhancement module

The local feature enhancement module is the defining architectural addition. Its function is to extract spatially local patterns and fuse them back into the attention pathway. The reported procedure begins by reshaping the token sequence into an “imaginary image” or pseudo-image so that convolution can be applied. The paper defines

$$
G_p \in \mathbb{R}^{\sqrt{N}\times \sqrt{N}\times d_{\text{embed}}}.
$$

Here the token grid is interpreted as a spatial feature map with channel dimension \(d_{\text{embed}}\) [2509.08959].

A two-layer \(3\times 3\) convolution block is then applied:

$$
F_{\rm conv}= \operatorname{Conv}_{3\times3}\left(\sigma\!\left(\operatorname{Conv}_{3\times3}(G_p)\right)\right),
$$

where \(\sigma(\cdot)\) is ReLU. The first convolution expands channels by 10%, ReLU is applied, and the second convolution projects back to the embedding dimension.

The convolutional output is scaled by a learnable scalar \(\Gamma\):

$$
F_{\text{weighted\_conv}} = \Gamma \times F_{\text{conv}}.
$$

The role of \(\Gamma\) is to allow the model to adaptively control how much local convolutional information influences the attention output. The paper treats this learnable balancing as an important distinction from fixed fusion schemes, arguing that different small-scale datasets may require different balances of local and global cues.

The appendix-level interpretation reported for this module is that the fused hypothesis space combines convolutional and attention-based functions. This suggests a bias-variance rationale: attention lowers bias by modeling broader relationships, convolution lowers variance by enforcing local and translational structure, and the learnable fusion weight \(\Gamma\) adaptively balances the two.

## 4. Fusion with window and shifted-window attention

CoSwin modifies the Swin attention block by fusing local convolution features with attention outputs after window partitioning. For ordinary window attention, the paper defines

$$
\operatorname{CoSwin\text{-}MSA}(X)=\left(\bigcup_{i=1}^{S}\operatorname{MSA}(X_i)\right)+F_{\text{weighted\_conv}}.
$$

For shifted-window attention, it defines

$$
\operatorname{CoSwin\text{-}Shifted\text{-}MSA}(X)=\left(\bigcup_{i=1}^{S}\operatorname{MSA}(\operatorname{Shift}(X_i))\right)+F_{\text{weighted\_conv}}.
$$

In this notation, \(X_i\) is the token set in the \(i\)-th window, \(S\) is the number of windows, \(\operatorname{Shift}(X_i)\) denotes the shifted-window partition, and \(F_{\text{weighted\_conv}}\) is the convolutional feature map scaled by \(\Gamma\) [2509.08959].

The underlying attention within each window is described as standard multi-head self-attention:

$$
\operatorname{MSA}(X_i)=\operatorname{concat}[\operatorname{head}_1,\ldots,\operatorname{head}_k],
$$

with each head given by

$$
\operatorname{head}_k=\operatorname{Attention}(Q_k,K_k,V_k),
$$

and single-head attention written as

$$
\operatorname{Attention}(Q,K,V)=\operatorname{Softmax}\left(\frac{QK^\top}{\sqrt{d}} + B\right)V.
$$

The paper intends \(B\) to represent relative positional bias, consistent with Swin-style attention.

The resulting mechanism can be summarized as a local-global fusion block: self-attention supplies semantic aggregation across windows, while the convolutional branch injects explicit spatial locality. The paper’s distinction from baseline Swin Transformer is therefore fourfold: convolution is fused into every attention block, locality is explicit rather than implicit, a learnable scalar balances local and global cues, and the resulting design is claimed to behave better on small datasets.

## 5. Experimental protocol and evaluation benchmarks

The reported evaluation covers five image-classification benchmarks: CIFAR-10, CIFAR-100, MNIST, SVHN, and Tiny ImageNet. The paper specifies the benchmark characteristics as follows.

| Benchmark | Resolution / training size | Classes |
|---|---|---|
| CIFAR-10 | \(32\times 32\), 50,000 training | 10 |
| CIFAR-100 | \(32\times 32\), 50,000 training | 100 |
| MNIST | \(28\times 28\), 60,000 training | 10 |
| SVHN | \(32\times 32\), 73,257 training | 10 |
| Tiny ImageNet | \(64\times 64\), 100,000 training | 200 |

For images smaller than \(32\times 32\), the patch size was reduced to 2; otherwise, the default patch size was 4. The implementation was reported in PyTorch and trained from scratch for 100 epochs using AdamW with weight decay \(0.05\), batch size 128, initial learning rate \(1\times 10^{-3}\), 10 epochs of warm-up, and stochastic drop rate 0.1. The stated regularization and augmentation components were MixUp, CutMix, Random Erasing, Repeated Augment, AutoAugment, and Stochastic Depth, and the hardware was an NVIDIA V100 GPU [2509.08959].

The comparison set includes Swin Transformer as the central baseline, together with ResNet18, ResNet56, MobileNetV2, ViT, DeiT-Tiny, and T2T-ViT. In contextual terms, this places CoSwin against both convolutional and transformer-based families, including token-to-token transformer designs that also seek to improve visual tokenization and representation [2101.11986].

## 6. Reported results, ablations, and interpretation

The principal quantitative finding is that CoSwin improves over the baseline Swin Transformer on all five reported datasets. The paper gives the following accuracies and gains.

| Dataset | CoSwin vs Swin Transformer | Gain |
|---|---|---|
| CIFAR-10 | 96.63% vs 94.46% | +2.17% |
| CIFAR-100 | 81.64% vs 76.72% | +4.92% |
| MNIST | 99.60% vs 99.50% | +0.10% |
| SVHN | 98.07% vs 97.81% | +0.26% |
| Tiny ImageNet | 65.06% vs 60.59% | +4.47% |

The paper also states that CoSwin outperforms other CNN and transformer baselines such as ResNet18, ResNet56, MobileNetV2, ViT, DeiT-Tiny, and T2T-ViT [2509.08959].

The interpretation attached to these results emphasizes robustness, generalization, and the utility of local-global fusion. The largest gains occur on CIFAR-100 and Tiny ImageNet, which the paper identifies as more challenging small-scale datasets. This is taken to indicate better generalization under limited-data conditions. The qualitative evidence includes Grad-CAM-based visualization showing that CoSwin focuses more sharply on the main object than ViT, DeiT, or Swin Transformer, which the paper presents as support for improved localization of discriminative regions.

The ablation study is central to the model’s justification. Removing convolution entirely causes noticeable drops relative to the full model: \(-2.22\%\) on CIFAR-10, \(-4.92\%\) on CIFAR-100, \(-0.10\%\) on MNIST, \(-0.26\%\) on SVHN, and \(-4.47\%\) on Tiny ImageNet. Additional ablations report that removing only one convolution causes a moderate drop and that adding the convolution branch without a trainable weight performs worse than weighted fusion. The full model with both convolutions and the learnable influence weight performs best.

A common misconception would be to treat CoSwin as a departure from hierarchical shifted-window attention. The reported design does not replace that mechanism; it preserves the same hierarchical backbone and augments it with local feature learning. The broader implication drawn in the paper is that local-global feature fusion can materially improve accuracy, robustness, and sample efficiency in vision transformers, especially when datasets are small or image resolution is limited. Code and pretrained weights are reported as available at the project repository linked by the paper [2509.08959].

Source: https://www.emergentmind.com/topics/coswin