U-GAT-IT: Unsupervised Image Translation

• U-GAT-IT is an unsupervised image-to-image translation framework that uses attention maps and adaptive normalization to handle diverse domain changes.
• It employs dual generator-discriminator pairs with CAM-based attention to focus on critical regions, achieving superior performance in shape and texture adaptation.
• Adaptive Layer-Instance Normalization (AdaLIN) enables precise control between instance and layer normalization, enhancing geometric transformation and overall image quality.

U-GAT-IT (Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization) is an unsupervised image-to-image translation framework designed to handle both holistic domain changes and translations that demand large geometric transformations. It integrates a novel attention module based on Class Activation Maps (CAM) and introduces Adaptive Layer-Instance Normalization (AdaLIN) to control the balance between texture and shape adaptation. The architecture achieves state-of-the-art results without task-specific architecture or hyper-parameter tuning across diverse translation domains (Kim et al., 2019).

1. Architecture and Components

U-GAT-IT employs an unpaired translation strategy using two symmetric generator-discriminator pairs:

• Generators:
  • $G_{s \to t}: X_s \rightarrow X_t$
  • $G_{t \to s}: X_t \rightarrow X_s$
• Discriminators:
  • $D_t$: discriminates real/fake in $X_t$
  • $D_s$: discriminates real/fake in $X_s$

Each generator encapsulates an encoder (downsampling + residual blocks), a bottleneck, and a decoder (residual and upsampling blocks). Critically, both generators and discriminators incorporate an auxiliary classifier $\eta$, which produces attention maps via CAM.

The discriminators are multi-scale PatchGANs with local (70×70) and global (286×286) scales. Their encoders are followed by a CAM-style classifier $\eta_D$ and a real/fake classifier $C_D$.

Core Architectural Flow

Component	Generator Pathway	Discriminator Pathway
Encoder	conv↓ → 4×ResBlock	conv↓
Attention	CAM (w, a(x)), MLP → (γ, β)	CAM
Decoder	4×AdaResBlock → up-convs (UpConv)	conv → 1 (real/fake decision)

This design enables the model to focus and adapt to domain-relevant visual semantics.

2. Attention Module and Class Activation Maps

The attention mechanism utilizes a CAM-style module applied within both G and D. For input $x$, the encoder produces feature maps $E(x) \in \mathbb{R}^{H\times W \times C}$, from which a channel-wise weight vector $w \in \mathbb{R}^C$ is learned using an auxiliary classifier $\eta$ (a single-layer MLP on pooled features).

Both global average pooling (GAP) and global max pooling (GMP) are performed on $E(x)$, concatenated, then processed by an MLP to produce a scalar logit. The output $$\eta(x) = \sigma\left( \sum_k w_k \cdot (\text{GAP}_k + \text{GMP}_k) \right)$$ is optimized to distinguish between domains for G (real $X_s$ vs. $X_t$) and authenticity for D (real vs. fake). The corresponding attention feature map is given by $a(x) = w \odot E(x)$ (channel-wise scaling).

CAM attention maps extracted by the generator and the two discriminators (local/global) spatially indicate regions critical for domain distinction and thus guide shape and texture conversion focus during translation.

3. Adaptive Layer-Instance Normalization (AdaLIN)

U-GAT-IT introduces AdaLIN in the residual blocks of the decoder, replacing traditional normalization approaches. Given an activation $a \in \mathbb{R}^{B\times C \times H \times W}$:

• $\mu_I$, $\sigma_I^2$: Per-channel mean/variance (Instance Normalization, IN)
• $\mu_L$, $\sigma_L^2$: Per-layer mean/variance (Layer Normalization, LN)
• Normalized forms:
  • $$\hat{a}_I = \frac{a - \mu_I}{\sqrt{\sigma_I^2 + \epsilon}}$$
  • $$\hat{a}_L = \frac{a - \mu_L}{\sqrt{\sigma_L^2 + \epsilon}}$$
• AdaLIN fusion:

$$\text{AdaLIN}(a; \gamma, \beta, \rho) = \gamma \odot \left[ \rho \cdot \hat{a}_I + (1 - \rho) \cdot \hat{a}_L \right] + \beta$$

$\gamma, \beta \in \mathbb{R}^C$ are affine parameters generated by an MLP from CAM features, and $\rho \in [0, 1]$ is a learnable block-wise scalar, dynamically updated during backpropagation and clipped to $[0,1]$.

This methodology allows fine-grained control over the degree of instance-level versus layer-level normalization during translation.

4. Controlling Shape and Texture Adaptation

The functionality of AdaLIN enables the model to modulate between preserving shape integrity and adapting global texture, controlled by the $\rho$ parameter:

• $\rho \rightarrow 1$: Favoring IN (preserves semantic features; shape retention)
• $\rho \rightarrow 0$: Favoring LN (emphasizes layer-wide styling; texture transformation)

The network automatically adapts $\rho$ per decoder block, depending on the dataset and the required semantic transformation. This mechanism is essential for translations involving large geometric changes, such as human faces to anime or animal morphing.

5. Loss Functions and Optimization

The complete U-GAT-IT objective for both $s \to t$ and $t \to s$ comprises a weighted sum of four losses using the Least Squares GAN (LSGAN) framework for stability:

• Adversarial Loss (LSGAN):

$$L_\text{lsgan}^{s \to t} = \mathbb{E}_{x \sim X_t}[D_t(x)^2] + \mathbb{E}_{x \sim X_s}[(1 - D_t(G_{s \to t}(x)))^2]$$

• Cycle-Consistency Loss:

$$L_\text{cycle}^{s \to t} = \mathbb{E}_{x \sim X_s}\left[\lVert x - G_{t \to s}(G_{s \to t}(x)) \rVert_1\right]$$

• Identity Loss:

$$L_\text{identity}^{s \to t} = \mathbb{E}_{x \sim X_t}\left[\lVert x - G_{s \to t}(x) \rVert_1\right]$$

• CAM Loss:
  • Generator CAM: Binary cross-entropy on auxiliary classifier output $\eta_s$,
  • Discriminator CAM: LSGAN-style loss on classifier output $\eta_{D_t}$.

The joint minimax objective with coefficients $\lambda_1=1$, $\lambda_2=10$, $\lambda_3=10$, $\lambda_4=1000$ encapsulates all four terms and is optimized for both forward and reverse directions.

6. Empirical Evaluation

Quantitative assessment utilizes Kernel Inception Distance (KID × 100), where lower values indicate better alignment with reference statistics, and a large user evaluation assesses qualitative preference.

KID Results (Lower = Better)

Dataset	U-GAT-IT	CycleGAN	UNIT	MUNIT	DRIT	AGGAN
selfie2anime	11.61±0.57	13.08±0.49	14.71±0.59	13.85±0.41	15.08±0.62	14.63±0.55
horse2zebra	7.06±0.80	8.05±0.72	10.44±0.67	11.41±0.83	9.79±0.62	7.58±0.71
cat2dog	7.07±0.65	8.92±0.69	8.15±0.48	10.13±0.27	10.92±0.33	9.84±0.79
photo2portrait	1.79±0.34	1.84±0.34	1.20±0.31	4.75±0.52	5.85±0.54	2.33±0.36
photo2vangogh	4.28±0.33	5.46±0.33	4.26±0.29	13.08±0.34	12.65±0.35	6.95±0.33

U-GAT-IT records the lowest KID on most domains, indicating superior image statistics alignment.

User Preference (Percentage Best)

Domain	U-GAT-IT	CycleGAN	UNIT	MUNIT	DRIT
selfie2anime	73.15 %	20.07 %	1.48 %	3.41 %	1.89 %
horse2zebra	73.56 %	23.07 %	0.85 %	1.04 %	1.48 %
cat2dog	58.22 %	6.19 %	18.63 %	14.48 %	2.48 %
photo2portrait	30.59 %	26.59 %	32.11 %	8.22 %	2.48 %
photo2vangogh	48.96 %	27.33 %	11.93 %	2.07 %	9.70 %

Preference is markedly skewed toward U-GAT-IT in domains requiring significant shape transformations (e.g., selfie↔anime, cat↔dog, horse↔zebra), reflecting effective semantic adaptation.

7. Implementation Strategies

The default configuration for U-GAT-IT is universal across all reported tasks:

• Input images are resized to 286×286, with a random 256×256 crop and horizontal flipping ($p=0.5$).
• Adam optimizer is used ($\beta_1=0.5$, $\beta_2=0.999$), learning rate $=10^{-4}$ (decayed linearly after 500k iterations to 0 by 1M).
• Batch size is 1; weight decay is $10^{-4}$; weight initialization $\sim \mathcal{N}(0,0.02)$.
• Encoder for G: sequential convolutions (Conv64-IN-ReLU, Conv128-IN-ReLU, Conv256-IN-ReLU) followed by 4 ResBlocks (256-IN-ReLU).
• Decoder for G: 4 AdaResBlocks (256-AdaLIN-ReLU), UpConv128-LIN-ReLU, UpConv64-LIN-ReLU, Conv3-Tanh; affine parameters from MLP (CAM features).
• Discriminators (local/global): stacked convolutions with spectral normalization and LeakyReLU, followed by CAM, then Conv1×1 for real/fake output.

This consistent architectural and optimization protocol enables cross-domain generalization from style transfer to tasks involving large structural changes without per-task calibration (Kim et al., 2019).