---
title: 'USE-CMHSA-GAN: Anime Synthesis Framework'
url: https://www.emergentmind.com/topics/use-cmhsa-gan
type: topic
---

# USE-CMHSA-GAN: Anime Synthesis Framework

USE-CMHSA-GAN is a generative adversarial network (GAN) framework for synthesizing high-quality anime character images. It extends the Deep Convolutional GAN (DCGAN) architecture with two novel modules: Unit-wise Squeeze-and-Excitation (USE) for channel-wise attention and Convolution-based Multi-Head Self-Attention (CMHSA) for modeling long-range spatial dependencies. This dual augmentation addresses critical challenges in anime image synthesis, particularly the need to capture salient stylistic features and enforce global coherence, significantly outperforming canonical GAN architectures in quantitative and qualitative evaluations on the anime-face-dataset [2411.11179].

## 1. Motivations and Core Design Principles

USE-CMHSA-GAN is engineered to overcome two persistent bottlenecks in anime image synthesis using GANs:

- **Channel-Level Feature Importance**: Traditional DCGAN architectures process all feature channels uniformly, which fails to adequately emphasize distinctive stylistic cues such as hair contours or intricate eye reflections—key elements in anime faces.
  
- **Long-Range Spatial Dependencies**: Anime character illustrations often exhibit globally coherent patterns (e.g., continuous hair strands, globally consistent shading) not readily captured by purely local convolutional filters.

The generator in USE-CMHSA-GAN incorporates the USE module to dynamically reweight feature channels based on global context, and the CMHSA module to enable each spatial location to attend to all other locations, thereby modeling correlations at arbitrary distances.

## 2. Architectural Composition

The architecture retains the two-player adversarial framework but introduces key enhancements in the generator:

- **Generator Structure**:
  - Input: 100-dimensional latent vector $z \sim \mathcal{N}(0, I)$.
  - Pipeline: DeConv $\rightarrow$ BatchNorm $\rightarrow$ ReLU $\rightarrow$ DeConv $\rightarrow$ USE $\rightarrow$ DeConv $\rightarrow$ CMHSA $\rightarrow$ DeConv $\rightarrow$ Tanh output layer.
  - Output: $64 \times 64 \times 3$ RGB image.
  
- **Discriminator Structure**:
  - Identical to DCGAN: stack of strided convolutions with LeakyReLU activations, culminating in a sigmoid for binary classification.

The following schematic summarizes the generator's main path:

| Input         | DeConv1 | USE | DeConv2 | CMHSA | DeConv3 | Tanh | Output   |
|---------------|---------|-----|---------|-------|---------|------|----------|
| $z$           | →       | →   | →       | →     | →       | →    | $G(z)$   |

## 3. Specialized Modules

### 3.1. Unit-wise Squeeze-and-Excitation (USE)

The USE block generalizes squeeze-and-excitation by introducing per-channel attention based on global feature statistics:

1. **Squeeze**: Global average pooling reduces $X \in \mathbb{R}^{C \times H \times W}$ to $z \in \mathbb{R}^C$ by $z_c = \frac{1}{HW} \sum_{i=1}^H \sum_{j=1}^W X_{c, i, j}$.
2. **Excitation**: Bottleneck MLP applied to $z$ yields $a = \sigma\left(W_2 \,\mathrm{ReLU}(W_1 z)\right)$, where $W_1 \in \mathbb{R}^{\frac{C}{r} \times C}$, $W_2 \in \mathbb{R}^{C \times \frac{C}{r}}$, $r$ is the reduction ratio (typically 16), and $\sigma$ is the sigmoid activation.
3. **Channel Reweighting**: Each original feature channel is rescaled as $Y_{c, i, j} = X_{c, i, j} \cdot a_c$.
4. **Upsampling**: When necessary, a transposed convolution restores the original spatial dimensions.

### 3.2. Convolution-based Multi-Head Self-Attention (CMHSA)

The CMHSA module is designed to capture global spatial dependency patterns:

- The input $X \in \mathbb{R}^{C \times H \times W}$ is reshaped to $N \times C$ with $N = H W$.
- For each of $h$ heads:
  - **Head dimension**: $d_\mathrm{head} = \frac{C}{h}$.
  - **Linear projections**: $Q = W_Q X$, $K = W_K X$, $V = W_V X$.
  - **Scaled dot-product attention**: $\mathrm{attn}_{ij} = \frac{Q_i \cdot K_j}{\sqrt{d_\mathrm{head}}}$.
  - **Weights**: $\alpha_{ij} = \frac{\exp(\mathrm{attn}_{ij})}{\sum_{k=1}^N \exp(\mathrm{attn}_{ik})}$.
  - **Aggregation**: $\mathrm{head}_i = \sum_{j=1}^N \alpha_{ij}\, V_j$.
- All heads are concatenated and projected via $W_O \in \mathbb{R}^{C \times C}$; the result is summed with the input (residual connection).

All heads attend to the same feature modality in the presented implementation.

## 4. Training Process and Loss Formulations

The adversarial training aligns with classical GAN methodology:

- **Discriminator Loss**:
  $$
  \mathcal{L}_{GAN}^D = -\mathbb{E}_{x \sim p_\mathrm{data}}[\log D(x)] - \mathbb{E}_{z \sim p_z} [\log (1 - D(G(z)))]
  $$

- **Generator Loss**:
  $$
  \mathcal{L}_{GAN}^G = -\mathbb{E}_{z \sim p_z} [\log D(G(z))]
  $$

There are no auxiliary losses for the USE or CMHSA blocks: $\mathcal{L}_{USE} = 0$, $\mathcal{L}_{CMHSA} = 0$. Both modules are implicitly optimized via the standard GAN supervision.

Optimization is performed using Adam ($\beta_1=0.5$, $\beta_2=0.999$) with a learning rate of $2 \times 10^{-4}$, batch size 64, and 200 epochs with alternating discriminator–generator steps.

## 5. Experimental Setup and Evaluation Metrics

- **Dataset**: The anime-face-dataset (learner-lu, 2022) is employed, comprising 27,588 faces ($256 \times 256$ px) filtered for quality, with an 80:10:10 split for training, validation, and test. Images are resized to $64 \times 64$, pixel-normalized to $[-1, 1]$, and converted to PyTorch tensors.
  
- **Metrics**: Performance on generated images is assessed using Fréchet Inception Distance (FID, lower is better) and Inception Score (IS, higher is better).

| Model            | FID ↓   | IS ↑   |
|------------------|---------|--------|
| VAE-GAN          | 64.45   | 2.60   |
| WGAN             | 79.34   | 2.35   |
| DCGAN            | 63.92   | 2.52   |
| USE-CMHSA-GAN    | 53.74   | 2.85   |

Ablation studies on the DCGAN backbone demonstrate progressive performance gains:

| Model                     | FID ↓   | IS ↑   |
|---------------------------|---------|--------|
| DCGAN                     | 63.92   | 2.52   |
| + USE module              | 58.99   | 2.69   |
| + CMHSA module            | 55.82   | 2.68   |
| + Both (full model)       | 53.74   | 2.85   |

Qualitative analysis highlights finer hair strand modeling, improved eye detail, and reduced background artifacts compared to DCGAN.

## 6. Mechanistic Analysis and Observed Benefits

The USE module enhances the representation of stylistically informative channels, including hair color gradients and eye shapes, by allocating higher network capacity to these features. The CMHSA module facilitates long-distance pixel interactions—such as between eyes and hair—enforcing global style coherence. Multiple attention heads allow the concurrent modeling of distinct stylistic attributes, for instance, encoding parallel cues on line thickness, shading gradients, and textural patterns.

These mechanisms collectively contribute to the observed improvements in both image quality and diversity, as evidenced by lower FID and higher IS relative to baseline and alternative GANs.

## 7. Limitations and Prospective Developments

- **Dataset Coverage**: The dataset's limited size and varying quality restrict the model's ability to synthesize highly detailed and varied facial components (e.g., noses, mouths). Dataset expansion and refinement are likely to yield further improvements.

- **Discriminator Architecture**: The discriminator remains unmodified from DCGAN, suggesting that extending attention mechanisms (USE, CMHSA) to the discriminator could enhance real/fake discrimination for subtle anime characteristics.

- **Cross-Modal Attention**: Despite CMHSA's designation as "cross-modal," the current instantiation attends only within the same feature channels. Future research may integrate explicit cross-modal fusion (e.g., semantic labels for controllable generation) by extending CMHSA to attend across distinct attribute spaces.

USE-CMHSA-GAN demonstrates the effectiveness of fusing channel-wise and spatial self-attention in generative modeling for anime image synthesis, with empirical evidence supporting its superiority over prevailing GAN variants in both fidelity and diversity of outputs [2411.11179].

Source: https://www.emergentmind.com/topics/use-cmhsa-gan