USE-CMHSA-GAN: Anime Synthesis Framework

• The paper proposes USE-CMHSA-GAN, integrating USE for channel attention and CMHSA for spatial dependencies to improve anime image synthesis.
• It extends the DCGAN architecture by augmenting the generator with specialized modules that capture key stylistic cues in anime faces.
• Experimental results demonstrate superior performance on FID and Inception Score metrics compared to baseline GANs, highlighting enhanced image fidelity and diversity.

USE-CMHSA-GAN is a generative adversarial network (GAN) framework for synthesizing high-quality anime character images. It extends the Deep Convolutional GAN (DCGAN) architecture with two novel modules: Unit-wise Squeeze-and-Excitation (USE) for channel-wise attention and Convolution-based Multi-Head Self-Attention (CMHSA) for modeling long-range spatial dependencies. This dual augmentation addresses critical challenges in anime image synthesis, particularly the need to capture salient stylistic features and enforce global coherence, significantly outperforming canonical GAN architectures in quantitative and qualitative evaluations on the anime-face-dataset (Lu, 2024).

1. Motivations and Core Design Principles

USE-CMHSA-GAN is engineered to overcome two persistent bottlenecks in anime image synthesis using GANs:

• Channel-Level Feature Importance: Traditional DCGAN architectures process all feature channels uniformly, which fails to adequately emphasize distinctive stylistic cues such as hair contours or intricate eye reflections—key elements in anime faces.
• Long-Range Spatial Dependencies: Anime character illustrations often exhibit globally coherent patterns (e.g., continuous hair strands, globally consistent shading) not readily captured by purely local convolutional filters.

The generator in USE-CMHSA-GAN incorporates the USE module to dynamically reweight feature channels based on global context, and the CMHSA module to enable each spatial location to attend to all other locations, thereby modeling correlations at arbitrary distances.

2. Architectural Composition

The architecture retains the two-player adversarial framework but introduces key enhancements in the generator:

• Generator Structure:
  • Input: 100-dimensional latent vector $z \sim \mathcal{N}(0, I)$.
  • Pipeline: DeConv $\rightarrow$ BatchNorm $\rightarrow$ ReLU $\rightarrow$ DeConv $\rightarrow$ USE $\rightarrow$ DeConv $\rightarrow$ CMHSA $\rightarrow$ DeConv $\rightarrow$ Tanh output layer.
  • Output: $64 \times 64 \times 3$ RGB image.
• Discriminator Structure:
  • Identical to DCGAN: stack of strided convolutions with LeakyReLU activations, culminating in a sigmoid for binary classification.

The following schematic summarizes the generator's main path:

Input	DeConv1	USE	DeConv2	CMHSA	DeConv3	Tanh	Output
$\rightarrow$0	→	→	→	→	→	→	$\rightarrow$1

3. Specialized Modules

3.1. Unit-wise Squeeze-and-Excitation (USE)

The USE block generalizes squeeze-and-excitation by introducing per-channel attention based on global feature statistics:

1. Squeeze: Global average pooling reduces $\rightarrow$2 to $\rightarrow$3 by $\rightarrow$4.
2. Excitation: Bottleneck MLP applied to $\rightarrow$5 yields $\rightarrow$6, where $\rightarrow$7, $\rightarrow$8, $\rightarrow$9 is the reduction ratio (typically 16), and $\rightarrow$0 is the sigmoid activation.
3. Channel Reweighting: Each original feature channel is rescaled as $\rightarrow$1.
4. Upsampling: When necessary, a transposed convolution restores the original spatial dimensions.

3.2. Convolution-based Multi-Head Self-Attention (CMHSA)

The CMHSA module is designed to capture global spatial dependency patterns:

• The input $\rightarrow$2 is reshaped to $\rightarrow$3 with $\rightarrow$4.
• For each of $\rightarrow$5 heads:
  • Head dimension: $\rightarrow$6.
  • Linear projections: $\rightarrow$7, $\rightarrow$8, $\rightarrow$9.
  • Scaled dot-product attention: $\rightarrow$0.
  • Weights: $\rightarrow$1.
  • Aggregation: $\rightarrow$2.
• All heads are concatenated and projected via $\rightarrow$3; the result is summed with the input (residual connection).

All heads attend to the same feature modality in the presented implementation.

4. Training Process and Loss Formulations

The adversarial training aligns with classical GAN methodology:

• Discriminator Loss:

$\rightarrow$4

• Generator Loss:

$\rightarrow$5

There are no auxiliary losses for the USE or CMHSA blocks: $\rightarrow$6, $\rightarrow$7. Both modules are implicitly optimized via the standard GAN supervision.

Optimization is performed using Adam ($\rightarrow$8, $\rightarrow$9) with a learning rate of $\rightarrow$0, batch size 64, and 200 epochs with alternating discriminator–generator steps.

5. Experimental Setup and Evaluation Metrics

• Dataset: The anime-face-dataset (learner-lu, 2022) is employed, comprising 27,588 faces ($\rightarrow$1 px) filtered for quality, with an 80:10:10 split for training, validation, and test. Images are resized to $\rightarrow$2, pixel-normalized to $\rightarrow$3, and converted to PyTorch tensors.
• Metrics: Performance on generated images is assessed using Fréchet Inception Distance (FID, lower is better) and Inception Score (IS, higher is better).

Model	FID ↓	IS ↑
VAE-GAN	64.45	2.60
WGAN	79.34	2.35
DCGAN	63.92	2.52
USE-CMHSA-GAN	53.74	2.85

Ablation studies on the DCGAN backbone demonstrate progressive performance gains:

Model	FID ↓	IS ↑
DCGAN	63.92	2.52
+ USE module	58.99	2.69
+ CMHSA module	55.82	2.68
+ Both (full model)	53.74	2.85

Qualitative analysis highlights finer hair strand modeling, improved eye detail, and reduced background artifacts compared to DCGAN.

6. Mechanistic Analysis and Observed Benefits

The USE module enhances the representation of stylistically informative channels, including hair color gradients and eye shapes, by allocating higher network capacity to these features. The CMHSA module facilitates long-distance pixel interactions—such as between eyes and hair—enforcing global style coherence. Multiple attention heads allow the concurrent modeling of distinct stylistic attributes, for instance, encoding parallel cues on line thickness, shading gradients, and textural patterns.

These mechanisms collectively contribute to the observed improvements in both image quality and diversity, as evidenced by lower FID and higher IS relative to baseline and alternative GANs.

7. Limitations and Prospective Developments

• Dataset Coverage: The dataset's limited size and varying quality restrict the model's ability to synthesize highly detailed and varied facial components (e.g., noses, mouths). Dataset expansion and refinement are likely to yield further improvements.
• Discriminator Architecture: The discriminator remains unmodified from DCGAN, suggesting that extending attention mechanisms (USE, CMHSA) to the discriminator could enhance real/fake discrimination for subtle anime characteristics.
• Cross-Modal Attention: Despite CMHSA's designation as "cross-modal," the current instantiation attends only within the same feature channels. Future research may integrate explicit cross-modal fusion (e.g., semantic labels for controllable generation) by extending CMHSA to attend across distinct attribute spaces.

USE-CMHSA-GAN demonstrates the effectiveness of fusing channel-wise and spatial self-attention in generative modeling for anime image synthesis, with empirical evidence supporting its superiority over prevailing GAN variants in both fidelity and diversity of outputs (Lu, 2024).