---
title: Siamese Swin Transformer with Dual Cross-Attention
url: https://www.emergentmind.com/topics/siamese-swin-transformer-with-dual-cross-attention-ssdca
type: topic
---

# Siamese Swin Transformer with Dual Cross-Attention

The Siamese Swin Transformer with Dual Cross-Attention (SSDCA) is a neural architecture that integrates twin Swin Transformer backbones with a dual cross-attention (DCA) module, enabling direct context-aware fusion of information from paired images or volumes. Originally pioneered for longitudinal tumor regrowth assessment in rectal endoscopy and multimodal tumor segmentation in head-and-neck oncology, SSDCA systematically emphasizes correspondences across time or modality, overcoming the need for image registration and enhancing both discriminative representation and resilience to severe imaging variabilities [2512.03883, 2302.03861].

## 1. Architectural Foundation

SSDCA employs a two-branch ("Siamese") topology wherein each branch processes a distinct input—such as pre- and post-treatment endoscopic images [2512.03883], or CT and PET image volumes [2302.03861]. Each branch leverages a hierarchical Swin Transformer backbone, characterized by window-based multi-head self-attention (W-MSA), patch embedding, and progressive spatial downsampling. For 2D tasks (rectal endoscopy), the shared Swin-Small backbone utilizes:
- Input resolution: $224\times224$
- Patch size: $4\times4$
- Window size: $7\times7$
- 4 hierarchical stages with patch merging, reducing $H,W$ by $2\times$ per stage

For 3D segmentation (head-and-neck PET/CT), each branch begins with a $3\times3\times3$ convolutional embedder ($C=48$), followed by 4 Swin stages with 3D windowed attention and patch merging. Dual-branch encoding in both settings results in parallel deep feature representations suitable for cross-contextual fusion via the DCA mechanism.

## 2. Dual Cross-Attention Mechanism

The DCA module is applied to the stage-4 output feature maps (or at multiple Swin stages for 3D segmentation), producing cross-attended features that encode mutual context between the two inputs. At the feature fusion point:
- Let $F_{pre}, F_{post} \in \mathbb{R}^{M \times C}$ denote the flattened features from each branch.
- Learned projections $W_q, W_k, W_v$ generate $Q, K, V$ matrices per branch and per attention head.

The DCA computes bidirectional cross-attention as:
\[
\begin{aligned}
Q_{pre} &= F_{pre} W_q,\quad K_{post} = F_{post} W_k,\quad V_{post} = F_{post} W_v \\
\mathrm{CA}_{pre\leftarrow post}(F_{pre}) &= \mathrm{Softmax}\left(\frac{Q_{pre} K_{post}^\top}{\sqrt{D_h}}\right)V_{post} \\
Q_{post} &= F_{post} W_q,\quad K_{pre} = F_{pre} W_k,\quad V_{pre} = F_{pre} W_v \\
\mathrm{CA}_{post\leftarrow pre}(F_{post}) &= \mathrm{Softmax}\left(\frac{Q_{post} K_{pre}^\top}{\sqrt{D_h}}\right) V_{pre}
\end{aligned}
\]
Each cross-attended feature is added residually and layer-normalized:
\[
\begin{aligned}
H_{pre} &= \mathrm{LN}\bigl(F_{pre} + \mathrm{CA}_{pre\leftarrow post}(F_{pre})\bigr) \\
H_{post} &= \mathrm{LN}\bigl(F_{post} + \mathrm{CA}_{post\leftarrow pre}(F_{post})\bigr)
\end{aligned}
\]
This bidirectional linking enables the model to focus attention on relevant regions in either input, without explicit alignment [2512.03883, 2302.03861].

## 3. Feature Aggregation and Prediction

For classification, each DCA-refined feature map undergoes global average pooling (GAP) to yield $h_{pre}, h_{post} \in \mathbb{R}^{C}$, which are concatenated into a single vector $h = [h_{pre}; h_{post}] \in \mathbb{R}^{2C}$. This vector is fed into a classification head:
\[
p = \sigma\left(W_2\,\mathrm{Dropout}\left(\mathrm{ReLU}(W_1 h)\right)\right)
\]
where $W_1 \in \mathbb{R}^{D_{fc}\times 2C}$, $W_2 \in \mathbb{R}^{1 \times D_{fc}}$, and $\sigma$ denotes sigmoid activation, yielding the class probability.

For segmentation tasks, multi-scale DCA fusion features are passed through a U-Net-like convolutional decoder, incorporating skip connections and upsampling layers to reconstruct the spatial segmentation map [2302.03861].

## 4. Training Protocols and Data Regimes

Key aspects include:
- **Optimization**: Adam (classification; learning rate $2\times10^{-4}$, batch size 8) [2512.03883], AdamW (segmentation; learning rate $1\times10^{-4}$, batch size 1, poly decay) [2302.03861].
- **Loss**: Binary cross-entropy for classification; composite Dice + cross-entropy for segmentation.
- **Augmentation**: Extensive geometric and intensity-based augmentations are employed (rotations, flips for 2D; 3D rotations, scaling, mirroring, noise, gamma for 3D).
- **Evaluation**: Five-fold cross-validation with stratified splits. Classification uses balanced sampling per mini-batch [2512.03883].

### Datasets
- Rectal endoscopy: 135 training patients ($\sim$7,392 pairs), 62 held-out test patients ($\sim$368 pairs) [2512.03883].
- Head-and-neck segmentation: 224 patients (HECKTOR 2021), with 5-fold CV [2302.03861].

## 5. Empirical Performance and Ablations

### Classification (Rectal Tumor Regrowth)
SSDCA achieves superior aggregate results compared with Swin-Single-Image (Swin-S SI) and simple stage-4 feature concatenation (SSFC):

| Model                | Balanced Acc.       | Sensitivity         | Specificity         |
|----------------------|---------------------|---------------------|---------------------|
| Swin-S SI            | 76.24% ± 0.02       | 65.32% ± 0.09       | 87.14% ± 0.07       |
| SSFC (concat)        | 81.13% ± 0.06       | 84.00% ± 0.13       | 78.57% ± 0.12       |
| SSDCA (DCA)          | 81.76% ± 0.05       | 90.07% ± 0.08       | 72.86% ± 0.05       |

SSDCA exhibits statistically significant improvements in balanced accuracy (p < 0.01) and outperforms in sensitivity.

### Segmentation (Head-and-Neck PET/CT)

| Model                | 5-fold Dice         |
|----------------------|---------------------|
| UNETR (PET+CT)       | 0.723 ± 0.035       |
| Swin UNETR (PET+CT)  | 0.754 ± 0.032       |
| nnU-Net (PET+CT)     | 0.767 ± 0.029       |
| SSDCA (DCA)          | 0.769 ± 0.026       |

Ablation studies reveal that omitting cross-attention by simple channel-wise concatenation results in Dice=0.754; integrating dual DCA at each Swin stage raises performance to Dice=0.769 (+0.015). DCA application at earlier stages in the 2D use case did not provide further benefits [2512.03883], while in 3D segmentation, multi-stage fusion is standard.

### Robustness and Representation

On endoscopic images affected by artifacts (blood, stool, telangiectasia, poor quality), SSDCA maintains stable sensitivity and specificity, with only marginal drops for stool and poor-quality occlusions [2512.03883].

Cluster analysis using UMAP demonstrates that SSDCA fosters maximal inter-cluster separation (1.45 ± 0.18) and minimal intra-cluster dispersion (1.07 ± 0.19) of GAP feature embeddings, indicative of superior discriminative structure.

## 6. Context and Relevance

SSDCA's dual cross-attention mechanism draws direct ancestry from prior cross-modal Swin architectures such as SwinCross, which established the integration of cross-attention at multiple hierarchical Swin stages for PET/CT fusion [2302.03861]. The primary innovation in SSDCA is its adaptation to the Siamese setting for temporal or paired analysis, leveraging DCA to robustly match feature correspondences across spatially unaligned scans or disparate modalities.

Use of pretrained Swin backbones enables SSDCA to extract domain-agnostic features, enhancing resistance to imaging noise and variability [2512.03883]. Its explicit contextual fusion at multiple resolutions is demonstrated to benefit both large- and small-scale lesion detection.

A plausible implication is broad applicability to diverse multi-temporal or multi-modal scenarios beyond endoscopic cancer monitoring and PET/CT segmentation.

## 7. Broader Implications and Directions

SSDCA's results suggest a systematic strategy for dual-branch Transformer fusion, effectively capturing cross-input dependencies without spatial registration—a feature essential for clinical domains with high intra- and inter-patient variability. Its competitive performance and domain-adaptive robustness indicate strong promise for future multimodal and longitudinal image analysis tasks.

Further extensions could address scalability to larger spatial and temporal contexts, integration with clinical metadata, or hybridization with attention-based meta-learning frameworks. The evolution of DCA modules may include adaptive weighting or self-supervised attention guidance to further enhance feature discrimination and generalization across domains [2512.03883, 2302.03861].

Source: https://www.emergentmind.com/topics/siamese-swin-transformer-with-dual-cross-attention-ssdca