---
title: Swin Transformer-tiny Texture Branch
url: https://www.emergentmind.com/topics/texture-branch-swin-transformer-tiny
type: topic
---

# Swin Transformer-tiny Texture Branch

The texture branch based on the Swin Transformer-tiny backbone within the ForensicFlow tri-modal forensic framework is designed to expose fine-grained, local blending or skin-texture artifacts characteristic of advanced Deepfake forgeries. In contrast to single-stream CNNs, this branch leverages hierarchical transformer-based modeling—incorporating patch partition, shifted window attention, and patch merging—to capture structure and texture discrepancies with high spatial precision. Its output, distilled to a video-level feature via attention-based temporal pooling, is adaptively fused with complementary RGB and frequency cues for robust video Deepfake detection [2511.14554].

## 1. Architectural Motivation and Backbone Selection

The principal function of the texture branch is the detection of sub-pixel, locally-blended artifacts in manipulated face frames that typically evade global CNN feature extractors. Swin Transformer-tiny, pretrained on ImageNet, is adopted as the backbone due to its ability to construct multi-scale, hierarchical feature representations via localized self-attention operating over non-overlapping windows, which are shifted between blocks to enable cross-region dependency modeling. This configuration enhances sensitivity to subtle and spatially localized irregularities, outperforming global CNN filters at identifying micro-level inconsistencies in texture and blending.

## 2. Input Preprocessing and Patch Embedding

Each input face frame is resized to 224×224 and normalized using ImageNet statistics. The frame is partitioned into non-overlapping patches of size P×P, with P=4 by default, resulting in $N = (224/4)^2 = 3136$ patches. Each patch, $x_i \in \mathbb{R}^{C·P^2}$, undergoes linear embedding:

$$
X_0 = [x_1, x_2, \ldots, x_N] W_e, \quad W_e \in \mathbb{R}^{(C·P^2) \times D}
$$

where $D = 96$ (default). The resulting tensor $X_0 \in \mathbb{R}^{N \times D}$ forms the input tokens for downstream transformer blocks.

## 3. Shifted Window Multi-Head Self-Attention

Core to the Swin architecture are two alternating self-attention blocks per stage:

- **Windowed Multi-Head Self-Attention (W-MSA):** Operates locally on non-overlapping $M \times M$ windows (default $M=7$), limiting computation while providing localized context.
- **Shifted-Window MSA (SW-MSA):** Prior to window partitioning, the feature map is cyclically shifted by $(\lfloor M/2 \rfloor, \lfloor M/2 \rfloor)$ positions, enabling tokens on window boundaries to attend across former partition borders.

For each attention head:

$$
Q_h = X W^Q_h, \quad K_h = X W^K_h, \quad V_h = X W^V_h
$$

with $W^Q_h, W^K_h, W^V_h \in \mathbb{R}^{D \times (D/H)}$. The self-attention incorporates a learnable relative position bias $B_{ij}$ per window:

$$
A = \text{Softmax} \left( \frac{Q K^T}{\sqrt{D/H}} + B \right) V
$$

where $A$ is of shape $(M^2) \times (D/H)$. SW-MSA ensures the model can aggregate evidence of local artifacts even when distributed near boundaries of standard attention windows.

## 4. Swin Transformer-tiny Stagewise Structure

Swin Transformer-tiny is organized in four stages, each progressively reducing spatial resolution while increasing channel depth and attention head count, as detailed below:

| Stage   | Output Resolution | Depth $L_i$ | Embed Dim $D_i$ | #Heads $H_i$ | MLP Ratio | Dropout |
|---------|-------------------|-------------|-----------------|--------------|-----------|---------|
| 1       | 56 × 56           | 2           | 96              | 3            | 4         | 0       |
| 2       | 28 × 28           | 2           | 192             | 6            | 4         | 0       |
| 3       | 14 × 14           | 6           | 384             | 12           | 4         | 0       |
| 4       | 7 × 7             | 2           | 768             | 24           | 4         | 0       |

After each W-MSA or SW-MSA block, a sequence of LayerNorm, MSA/MLP, DropPath (with total stochastic depth ≈0.1), and residual connections are applied.

At the start of each stage (except the first), a patch-merging layer reduces the spatial dimensions by 2 and doubles the embedding dimension. Specifically, for neighborhood $2 \times 2$ regions:

$$
x'_{i,j} = \text{Concat}(X_{2i,2j}, X_{2i+1,2j}, X_{2i,2j+1}, X_{2i+1,2j+1}), \quad x'_{i,j} \in \mathbb{R}^{4D_i}
$$

Followed by:

$$
X_{i+1} = \text{LayerNorm}(x') W_\text{down}, \quad W_\text{down} \in \mathbb{R}^{4D_i \times 2D_i}
$$

## 5. Feature Post-processing and Temporal Aggregation

The stage 4 output, of shape $7\times 7\times 768$, undergoes adaptive average pooling to yield a per-frame feature $z_t \in \mathbb{R}^{768}$. Linear projection reduces this to $f_t^{\text{Tex}} \in \mathbb{R}^{512}$ for each video frame. Across $K$ frames, the set $\{f_t^{\text{Tex}}\}_{t=1}^K$ represents the temporal progression of local texture features.

Temporal aggregation is performed via attention-based pooling, where per-frame weights $\alpha_t$ are computed as:

$$
\alpha_t = \text{Softmax}(u^T \tanh(W_f [f_t^{\text{RGB}}; f_t^{\text{Tex}}; f_t^{\text{Freq}}]))
$$

resulting in a pooled texture feature:

$$
f^{\text{Tex}}_{\text{pooled}} = \sum_{t=1}^K \alpha_t f_t^{\text{Tex}}
$$

## 6. Adaptive Cross-Branch Fusion

The video-level representations from each branch—$f_{\text{RGB}}$, $f_{\text{Tex}}$, $f_{\text{Freq}}$ (all in $\mathbb{R}^{512}$)—are combined using a “fusion attention” mechanism. This network outputs scalar weights $[\alpha_{\text{RGB}}, \alpha_{\text{Tex}}, \alpha_{\text{Freq}}]$ (softmax-normalized):

$$
[\alpha_{\text{RGB}}, \alpha_{\text{Tex}}, \alpha_{\text{Freq}}] = \text{Softmax}(W_a [f_{\text{RGB}}; f_{\text{Tex}}; f_{\text{Freq}}] + b_a)
$$

The fused representation is then:

$$
f_{\text{fused}} = \alpha_{\text{RGB}} f_{\text{RGB}} + \alpha_{\text{Tex}} f_{\text{Tex}} + \alpha_{\text{Freq}} f_{\text{Freq}}
$$

This architecture enables dynamic, content-aware weighting of each cue modality at inference.

## 7. Training Protocols and Empirical Significance

Optimization employs AdamW with weight decay $1\times 10^{-4}$, and layer-wise progressive unfreezing during the initial 15 training epochs. The projection and classification head are trained with an initial learning rate of $2\times 10^{-5}$, which is halved at each unfreeze stage:

- Epochs 1–3: only projection & head trainable
- Epochs 4–6: unfreeze final backbone blocks
- Epochs 7–8: unfreeze mid-blocks
- Epochs 9–15: full backbone fine-tuning

Ablation at epoch 5 quantifies the contribution of the texture branch:

| Composition           | F1 Score | AUC    |
|-----------------------|----------|--------|
| RGB + Texture         | 0.8285   | 0.8309 |
| Tri-branch (full)     | 0.9408   | 0.9752 |

This validates the synergy between the texture branch and other modalities in the ForensicFlow framework.

## 8. Context and Forensic Impact

The deployment of the Swin Transformer-tiny as a texture branch within ForensicFlow demonstrates the utility of hierarchical, local-attention transformers in forensic tasks requiring extreme detail sensitivity. The branch's design, combining multi-scale representation, patch merging, and shifted window attention, is specifically tailored to the detection of subtle blending imperfections in Deepfake media that global or frequency-oriented models may neglect. Temporal attention pooling and adaptive fusion further ensure that micro-level evidence is neither overwhelmed nor overlooked in the final prediction, underpinning the system's superior AUC (0.9752), F1 (0.9408), and accuracy (0.9208) on challenging datasets such as Celeb-DF (v2) [2511.14554].

Source: https://www.emergentmind.com/topics/texture-branch-swin-transformer-tiny