---
title: Adaptive Layer-Instance Normalization (AdaLIN)
url: https://www.emergentmind.com/topics/adaptive-layer-instance-normalization-adalin
type: topic
---

# Adaptive Layer-Instance Normalization (AdaLIN)

Adaptive Layer-Instance Normalization (AdaLIN) is a normalization technique introduced in U-GAT-IT for unsupervised image-to-image translation, designed to interpolate adaptively between Instance Normalization (IN) and Layer Normalization (LN) using a learned, per-layer scalar gate. This approach equips generative models with the flexibility to control the trade-off between local texture (IN) and global shape/style (LN) transformations based on the learning dynamics and specific requirements of image translation tasks [1907.10830].

## 1. Mathematical Definition

Given an activation tensor $a \in \mathbb{R}^{C \times H \times W}$ for a single sample, AdaLIN first computes both IN-style and LN-style statistics:

- **Instance Normalization (IN):**
  - Channel-wise mean: $\mu_I^c = \frac{1}{HW} \sum_{i=1}^H \sum_{j=1}^W a_{cij}$
  - Channel-wise std: $\sigma_I^c = \sqrt{\frac{1}{HW} \sum_{i,j} (a_{cij} - \mu_I^c)^2 + \epsilon}$

- **Layer Normalization (LN):**
  - Layer mean: $\mu_L = \frac{1}{CHW} \sum_{c,i,j} a_{cij}$
  - Layer std: $\sigma_L = \sqrt{\frac{1}{CHW} \sum_{c,i,j} (a_{cij} - \mu_L)^2 + \epsilon}$

Normalized activations:
- IN-style: $\hat a_I = (a - \mu_I) / \sigma_I$
- LN-style: $\hat a_L = (a - \mu_L) / \sigma_L$

The AdaLIN output is given by
$$
\mathrm{AdaLIN}(a;\gamma,\beta,\rho) = \gamma \cdot [\rho \cdot \hat{a}_I + (1-\rho) \cdot \hat{a}_L] + \beta
$$
where $\gamma, \beta \in \mathbb{R}^C$ are per-channel scale and shift, and $\rho \in [0,1]$ is a learned layerwise gate. After each gradient step, $\rho$ is updated via backpropagation and clipped to the $[0,1]$ interval.

## 2. Learnable Parameters and Their Roles

AdaLIN introduces three principal sets of learnable parameters per layer:

- **Gate parameter ($\rho$):** One scalar per AdaLIN layer, initialized to $1$ (favoring IN) in decoder residual blocks and $0$ (favoring LN) in up-sampling blocks. It is updated via gradient descent and clamped to $[0,1]$ post-update.
- **Affine transform ($\gamma, \beta$):** One scaling ($\gamma_c$) and shifting ($\beta_c$) parameter per channel. In U-GAT-IT, these are dynamically computed from attention embeddings rather than being static learnable parameters.
- **Attention-driven MLP:** Three fully connected layers with hidden size 256 and ReLU activations map global-pooled attention features to ($\gamma, \beta$). These MLPs are trained end-to-end and are initialized from $\mathcal{N}(0,0.02)$.

This arrangement allows every decoder layer to adaptively select its own normalization mode and affine transformation based on learned attention signals.

## 3. Relation to Other Normalization Techniques

AdaLIN generalizes and interpolates between different normalization approaches:

| Normalization | Statistic Scope          | Typical Effects                | Limitation                                  | AdaLIN Behavior         |
|---------------|-------------------------|-------------------------------|----------------------------------------------|------------------------|
| Instance Norm (IN) | Channel over spatial        | Style removal, local consistency | May lose global (shape) structure            | $\rho = 1.0$           |
| Layer Norm (LN)    | All channels & spatial     | Preserves global structure      | May oversmooth, weaken local details         | $\rho = 0.0$           |
| AdaIN               | Style signal to modulate   | Style transfer                  | Interpolates only style, not normalization   | -                      |
| BIN                | Batch/instance interpolation| Batch + instance, fixed blend   | Blends BN & IN, not adaptive per sample/layer| -                      |

In contrast to AdaIN, which interpolates style via external statistics, and BIN, which uses a fixed parameter between BN and IN, AdaLIN adaptively learns $\rho$ per layer to select the appropriate balance point between content and style preservation for each decoder layer [1907.10830].

## 4. Integration and Workflow within U-GAT-IT

AdaLIN is integrated exclusively in the decoder of each generator in U-GAT-IT. Its placement and usage are as follows:

- Each decoder residual block contains an AdaLIN layer immediately after its $3 \times 3$ convolution ("AdaResBlock").
- In decoder up-sampling convolutions, AdaLIN is applied before the activation function.
- Encoders use standard Instance Normalization to assist the auxiliary classifier.
- No AdaLIN is employed in discriminators, which instead utilize spectral normalization and a class activation map (CAM)-based attention mechanism.

Key hyperparameters include batch size $= 1$, learning rate $= 10^{-4}$ (constant for 500k iterations, linearly decayed to zero by 1M iterations), $\gamma,\beta$ MLP hidden dimension $= 256$, overall Adam optimizer settings $(\beta_1=0.5, \beta_2=0.999)$, and weight decay $= 10^{-4}$.

## 5. Empirical Results and Ablation

Ablation studies on the selfie2anime dataset demonstrate the efficacy of AdaLIN. Measured by Kernel Inception Distance (KID $\times100$, lower is better):

| Model                | KID (selfie2anime) $\downarrow$ |
|----------------------|-------------------------------|
| U-GAT-IT (AdaLIN)    | $11.61 \pm 0.57$              |
| Only IN              | $13.64 \pm 0.76$              |
| Only LN              | $12.39 \pm 0.61$              |
| AdaIN                | $12.29 \pm 0.78$              |
| GN                   | $12.76 \pm 0.64$              |

AdaLIN achieves the lowest KID, indicating superior preservation of identity while enabling appropriate stylization and shape transformation. Qualitative observations report that "only-IN" configurations retain facial accessories but provide insufficient stylization, while "only-LN" results give strong stylization at the cost of identity preservation. In practice, AdaLIN learns $\rho \to 1$ in residual blocks (favoring IN, preserving content) and $\rho \to 0$ in up-sampling blocks (favoring LN, imposing style/shape).

## 6. Algorithmic Implementation

A PyTorch-style implementation sketch for AdaLIN, based on a single sample input $x \in \mathbb{R}^{C \times H \times W}$, is as follows:

```python
def AdaLIN(x, rho, gamma, beta, eps=1e-5):
    # x: [C, H, W]
    C, H, W = x.shape
    mu_I = x.view(C, -1).mean(dim=1, keepdim=True).view(C, 1, 1)
    var_I = x.view(C, -1).var(dim=1, keepdim=True, unbiased=False).view(C, 1, 1)
    sigma_I = torch.sqrt(var_I + eps)
    mu_L = x.mean()
    var_L = x.var(unbiased=False)
    sigma_L = torch.sqrt(var_L + eps)
    x_I = (x - mu_I) / sigma_I
    x_L = (x - mu_L) / sigma_L
    out = gamma.view(C,1,1) * (rho * x_I + (1 - rho) * x_L) + beta.view(C,1,1)
    return out
```

In deployed models, $\rho$ is updated by backpropagation and clamped to $[0,1]$ after each training step, while $\gamma, \beta$ are produced by the attention-driven MLP.

## 7. Significance for Image-to-Image Translation

AdaLIN provides a principled, data-driven method for controlling the balance between content preservation and stylization in image translation. By dynamically blending IN and LN through $\rho$, AdaLIN enables a fixed-architecture generator to handle diverse translation regimes—from fine texture transfer (e.g., photo to painting) to substantial geometric changes (e.g., selfie to anime)—without manual tuning or architectural modifications. Its effectiveness is empirically demonstrated via improved quantitative metrics and qualitative results on tasks requiring both local and global style adaptation [1907.10830].

Source: https://www.emergentmind.com/topics/adaptive-layer-instance-normalization-adalin