Adaptive Whitening & Coloring (AdaWCT)

• Adaptive Whitening and Coloring Transformation (AdaWCT) is a technique that performs group-wise whitening and coloring of neural activations to precisely match both mean and covariance statistics of a target style.
• Its innovation lies in replacing standard AdaIN with a learned block-diagonal mapping that uses efficient Newton-Schulz iterations and structured parameterization for scalable style injection.
• Empirical results show that AdaWCT improves image quality metrics like FID and LPIPS over AdaIN, proving its effectiveness in style-conditional image-to-image translation tasks while keeping computational costs low.

Adaptive Whitening and Coloring Transformation (AdaWCT) generalizes feature normalization in deep neural networks by performing channel-wise whitening and coloring to enable expressive, group-wise, style-conditional transformations. AdaWCT explicitly matches both the mean and full (or block-diagonal/grouped) covariance statistics of network activations to those of a target style, surpassing the limited per-channel control of methods such as Adaptive Instance Normalization (AdaIN). This yields richer style injection while maintaining computational and memory efficiency via structured parameterization and group-wise operations (Cho et al., 2018, Dufour et al., 2022).

1. Mathematical Formulation

Let $F\in\mathbb{R}^{C\times H\times W}$ denote the feature activations at a given layer, where $C$ is the number of channels and $N=H\cdot W$ the number of spatial positions. The standard whitening and coloring transformation (WCT) proceeds as follows:

• Whitening:
  • Compute the mean vector:

  $\mu = \frac{1}{N} X \mathbf{1}$

  where $X$ is $F$ flattened to $\mathbb{R}^{C\times N}$. - Compute the covariance:

  $$\Sigma = \frac{1}{N-1} (X - \mu \mathbf{1}^{T})(X - \mu \mathbf{1}^{T})^T$$ - Obtain the whitening transform:

  $$W = \Sigma^{-\frac{1}{2}},\quad X_w = W(X - \mu \mathbf{1}^{T})$$

• Coloring:

  • Given style covariance $\Sigma_s$ and mean $\mu_s$, form coloring matrix $C = \Sigma_s^{\frac{1}{2}}$.
  • Color and shift:

  $X_c = C X_w + \mu_s \mathbf{1}^{T}$

AdaWCT replaces the coloring operation with a learned, style-conditional block-diagonal matrix $\Gamma(z)$ and mean $\mu_s(z)$, with mappings produced by a lightweight network (typically an MLP or affine layer) from style code $z \in \mathbb{R}^d$:

$$\tilde{X} = \Gamma \left[\Sigma^{-\frac{1}{2}}(X-\mu \mathbf{1}^{T})\right] + \mu_s \mathbf{1}^{T}$$

Group-wise operation splits the channels into $n$ groups of size $G = C/n$, and all whitening/coloring operations are performed independently per group, leveraging block-diagonal structures for memory and compute efficiency (Dufour et al., 2022).

2. Algorithmic Details and Implementation

The AdaWCT layer is constructed as a group-wise operation. For each group $j=1\dots n$:

1. Extract group $G_j\in\mathbb{R}^{G\times N}$ and compute mean $\mu_j$ and covariance $\Sigma_j$.

2. Estimate $W_j = \Sigma_j^{-1/2}$, frequently using the Newton-Schulz iteration to avoid explicit eigendecomposition.

3. Transform features: $Z_j = W_j (G_j - \mu_j \mathbf{1}^{T})$.

4. Obtain the group coloring matrix $\Gamma_j$ and mean $\mu_{s, j}$ via a mapping from style code $z$.

5. Color and shift: $$\tilde{G}_j = \Gamma_j Z_j + \mu_{s, j} \mathbf{1}^{T}$$.

6. Concatenate all groups to yield the output.

This approach generalizes AdaIN: setting $G=1$ reduces AdaWCT to scaling and shifting with diagonal covariance. Conversely, $G=C$ recovers full whitening/coloring at greater computational cost (Dufour et al., 2022). In methods such as GDWCT, the procedure is embedded into generator architectures by replacing normalization or AdaIN instances with AdaWCT modules, and regularization terms encourage proper whitening and orthogonalization (Cho et al., 2018).

3. Integration with Generator Architectures

AdaWCT has been integrated into both exemplar-based image-to-image translation frameworks (e.g., GDWCT) and reference- or latent-guided GANs (e.g., StarGANv2).

• GDWCT pipeline (Cho et al., 2018):

  • Two encoders extract content $c\in\mathbb{R}^{C\times H\times W}$ and style $s\in\mathbb{R}^{C\times h\times w}$.
  • The generator $G$ comprises several residual blocks with AdaWCT modules applied to feature activations.
  • Inference and training interplay multiple encodings, translations, re-encodings, and loss computation (adversarial, cycle, identity, latent consistency, and regularization terms).
• StarGANv2 with AdaWCT (Dufour et al., 2022):
  • Each AdaIN instance in the residual blocks is replaced with a group-wise AdaWCT layer.
  • Style codes $z$ are provided by either a reference style encoder or a mapping network.
  • All architectural and loss functions from StarGANv2 remain unchanged other than the style injection module.

4. Regularization and Training Procedures

• Whitening regularization: $$R_w = \mathbb{E}_x \lVert \Sigma_c - I \rVert_{1,1}$$, penalizes deviations of the content covariance from identity, ensuring that subtracting the mean is sufficient for whitening.
• Coloring regularization: $R_c = \mathbb{E}_s \lVert U^T U - I \rVert_{1,1}$, encourages the coloring matrix to have approximate orthogonality akin to the eigenspace of the style covariance.
• Total loss: Combined with adversarial, cycle, identity, and content/style consistency losses.

Pseudocode, as detailed in both (Cho et al., 2018) and (Dufour et al., 2022), describes batchwise processing of content and style images—extracting feature statistics, applying AdaWCT in generators, re-encoding, and evaluating the composite loss objective.

5. Empirical Results and Evaluation Metrics

Extensive benchmarks reveal the effects of AdaWCT relative to other style-injection mechanisms:

• Image Quality (StarGANv2/AFHQ at 256×256) (Dufour et al., 2022):
  • AdaWCT achieves lower Fréchet Inception Distance (FID) and higher LPIPS diversity than AdaIN:

  Method	FID↓ (ref)	LPIPS↑ (ref)	FID↓ (latent)	LPIPS↑ (latent)
  AdaIN (StarGANv2)	19.78	0.431	16.18	0.450
  AdaWCT	16.20	0.434	13.07	0.476

• Ablations (Dufour et al., 2022):

  • Incremental group size $G$ from $1$ (AdaIN) to $64$ improves FID and LPIPS, with diminishing returns beyond $G=16$.
  • Disabling either whitening or coloring halves the gains; the full AdaWCT mechanism is necessary for optimal results.
• Computational Efficiency:
  • Newton-Schulz-based group-wise whitening adds approximately 0.5ms per AdaWCT block on GPU, with minimal overall impact.
  • The parameter count is significantly reduced by group-wise and block-diagonal parameterization ($O(C\cdot G)$ per block).
• Unsupervised Image Translation (CelebA, Artworks, Yosemite, BAM, cat2dog) (Cho et al., 2018):
  • GDWCT/AdaWCT is preferred in user studies and achieves higher class-accuracy in attribute translation versus AdaIN, DRIT, and classical WCT.

6. Comparison with Existing Methods

• AdaIN: Matches only per-channel mean and variance; no modeling of inter-channel correlations (covariances).
• Exact WCT (Li et al., 2017): Performs full-channel whitening/coloring via eigendecomposition ($O(C^3)$ time, expensive backpropagation), intractable for modern architectures.
• AdaWCT: Approximates full or group-wise WCT with lightweight, regularized, and learnable transforms, end-to-end differentiable and much faster (C²/G parameters, no SVD in forward/backward).

AdaWCT bridges the gap by controlling the expressiveness through the group size parameter $G$, with $G=1$ degenerating to AdaIN and $G=C$ approximating full WCT. This tunable design allows efficient deployment in high-dimensional architectures (Cho et al., 2018, Dufour et al., 2022).

7. Limitations and Future Directions

AdaWCT has been validated primarily in image-to-image translation and style-conditional GAN settings. Known limitations include:

• Thus far, evaluations are limited to architectures such as StarGANv2; tests on unconditional generators (e.g., StyleGAN3) remain open.
• The style-to-parameter mapping network can be further compressed (e.g., replacing the MLP with affine projections) to reduce complexity.
• The explicit block-diagonal/group-wise structure may restrict modeling of global channel correlations compared to unrestricted full covariance operations.

A plausible implication is that further generalization or hybridization with efficient structured matrix parameterizations (e.g., low-rank, Toeplitz) could enhance expressivity while retaining scalability.

• "Image-to-Image Translation via Group-wise Deep Whitening-and-Coloring Transformation" (Cho et al., 2018)
• "AdaWCT: Adaptive Whitening and Coloring Style Injection" (Dufour et al., 2022)