Whitening-Coloring Transform for Style Transfer

• Whitening-Coloring Transform (WCT) is a technique that decorrelates content features and re-correlates them using style covariance, achieving universal style transfer.
• It employs analytic whitening, eigendecomposition, and coloring to align CNN activations, enabling style transfer without style-specific retraining.
• WCT’s framework has been extended to photorealistic transfer, GANs, and image translation, balancing computational efficiency with high-quality visual results.

Searching arXiv for core and related WCT papers to ground the article in the cited literature. Whitening-Coloring Transform (WCT) is a feature-space transformation for universal style transfer in which deep content features are first decorrelated and then re-correlated to match the covariance structure of style features. In its canonical form, WCT operates on CNN activations extracted from a fixed pretrained VGG network and transfers arbitrary unseen styles without style-specific retraining: whitening removes the original second-order statistics of the content representation, while coloring imposes those of the style representation. The method was introduced as the analytic core of a reconstruction-based universal style transfer pipeline (Li et al., 2017), and was later reinterpreted as one particular solution of a broader covariance-matching and optimal-transport formulation (Lu et al., 2019).

1. Statistical definition and core equations

In the standard formulation, content and style feature maps at a chosen CNN layer are vectorized as

$$f_c \in \mathbb{R}^{C \times H_c W_c}, \qquad f_s \in \mathbb{R}^{C \times H_s W_s},$$

or, in the Gaussian interpretation, as samples from multivariate distributions

$$u \sim \mathcal{N}(\mu_c,\Sigma_c), \qquad v \sim \mathcal{N}(\mu_s,\Sigma_s).$$

WCT is a linear transform that maps centered content features to the style distribution by whitening with the inverse square root of the content covariance and coloring with the square root of the style covariance. In matrix form, the transform is

$t(u) = T(u-\mu_c) + \mu_s,$

with the covariance-matching constraint

$T\Sigma_c T^T = \Sigma_s.$

The canonical WCT choice is

$T = \Sigma_s^{1/2}\Sigma_c^{-1/2},$

which first removes content feature correlations and then imposes style correlations (Lu et al., 2019).

In the eigendecomposition form used in universal style transfer, if

$$f_c f_c^\top = E_c D_c E_c^\top,\qquad f_s f_s^\top = E_s D_s E_s^\top,$$

then whitening and coloring are written as

$$\hat{f}_c = E_c D_c^{-\frac12} E_c^\top f_c, \qquad \hat{f}_{cs} = E_s D_s^{\frac12} E_s^\top \hat{f}_c,$$

followed by restoration of the style mean. The resulting transformed features satisfy

$\hat{f}_{cs}\hat{f}_{cs}^\top = f_s f_s^\top,$

so WCT matches the full covariance matrix, not merely diagonal variances. This is the explicit covariance-matching analogue of Gram-matrix style transfer, since the Gram matrix $FF^\top$ encodes channel correlations in feature space (Li et al., 2017).

2. Canonical universal style transfer pipeline

The original WCT system combines a fixed encoder, learned decoders, and an analytic feature transform. A pretrained VGG-19 network serves as encoder; a symmetric reconstruction network is trained to invert features back to RGB images; and WCT is inserted between them as a closed-form style operator. The decoder is trained with

$$L = \|I_{o}-I_{i}\|^{2}_{2} + \lambda\|\Phi(I_{o}) - \Phi(I_{i})\|^{2}_{2},$$

where $$u \sim \mathcal{N}(\mu_c,\Sigma_c), \qquad v \sim \mathcal{N}(\mu_s,\Sigma_s).$$0 is the fixed VGG encoder. After training, the encoder and decoders are frozen, and style transfer is performed entirely by feature extraction, covariance computation, WCT, and decoding (Li et al., 2017).

The original system trains five decoders, one for each VGG layer $$u \sim \mathcal{N}(\mu_c,\Sigma_c), \qquad v \sim \mathcal{N}(\mu_s,\Sigma_s).$$1 with $$u \sim \mathcal{N}(\mu_c,\Sigma_c), \qquad v \sim \mathcal{N}(\mu_s,\Sigma_s).$$2. This enables both single-level stylization and a multi-level coarse-to-fine procedure. In the multi-level version, WCT is applied sequentially from $$u \sim \mathcal{N}(\mu_c,\Sigma_c), \qquad v \sim \mathcal{N}(\mu_s,\Sigma_s).$$3 down to $$u \sim \mathcal{N}(\mu_c,\Sigma_c), \qquad v \sim \mathcal{N}(\mu_s,\Sigma_s).$$4: higher layers introduce coarse semantic style patterns, and lower layers refine colors and fine textures. The reported motivation is that reversing this order destroys low-level information once higher-level transformations are applied, whereas coarse-to-fine stylization better preserves visual quality and style statistics across levels (Li et al., 2017).

A style-strength control parameter is introduced by interpolating between transformed and original content features: $$u \sim \mathcal{N}(\mu_c,\Sigma_c), \qquad v \sim \mathcal{N}(\mu_s,\Sigma_s).$$5 The same analytic mechanism is also used for texture synthesis by replacing content with random noise, and for texture interpolation via feature blending: $$u \sim \mathcal{N}(\mu_c,\Sigma_c), \qquad v \sim \mathcal{N}(\mu_s,\Sigma_s).$$6 These extensions follow directly from WCT’s interpretation as an explicit manipulation of feature statistics rather than a learned style-specific mapping (Li et al., 2017).

3. Expressivity, relation to AdaIN, and non-uniqueness

A central distinction in the literature is between WCT and Adaptive Instance Normalization (AdaIN). AdaIN aligns per-channel mean and variance and therefore uses only diagonal covariance information; in matrix terms,

$$u \sim \mathcal{N}(\mu_c,\Sigma_c), \qquad v \sim \mathcal{N}(\mu_s,\Sigma_s).$$7

with $$u \sim \mathcal{N}(\mu_c,\Sigma_c), \qquad v \sim \mathcal{N}(\mu_s,\Sigma_s).$$8 and $$u \sim \mathcal{N}(\mu_c,\Sigma_c), \qquad v \sim \mathcal{N}(\mu_s,\Sigma_s).$$9 the diagonal matrices of $t(u) = T(u-\mu_c) + \mu_s,$0 and $t(u) = T(u-\mu_c) + \mu_s,$1. WCT, by contrast, matches the entire covariance matrix, including inter-channel correlations. The stated argument is that CNN channels are correlated, so a full covariance-aware transform is more expressive and generally more stylized than a diagonal one (Lu et al., 2019). The same contrast reappears in later image translation and GAN literature, where WCT is treated as a strict generalization of channel-wise normalization because it models cross-channel interactions rather than independent scale-and-shift parameters (Cho et al., 2018, Dufour et al., 2022).

Theoretical analysis also shows that covariance matching alone is underconstrained. If

$t(u) = T(u-\mu_c) + \mu_s,$2

then

$t(u) = T(u-\mu_c) + \mu_s,$3

is also a valid solution for any unit orthogonal matrix $t(u) = T(u-\mu_c) + \mu_s,$4. WCT is therefore only one member of an infinite family of covariance-matching transforms. This result formalizes a common but often implicit point: matching style statistics does not define a unique stylized feature, and different orthogonal choices can yield different visual outputs even when the covariance constraint is exactly satisfied (Lu et al., 2019).

Within the optimal-transport view, WCT satisfies the style-distribution constraint but does not explicitly optimize content preservation. The proposed objective is

$t(u) = T(u-\mu_c) + \mu_s,$5

which selects, among all admissible covariance-matching transforms, the one closest in expected squared distance to the original content feature. The resulting closed-form Optimal Style Transfer (OST) solution is

$t(u) = T(u-\mu_c) + \mu_s,$6

In this interpretation, WCT is “covariance matching,” whereas OST is “covariance matching plus optimal preservation of content.” The same analysis emphasizes that WCT can preserve some structure due to whitening, but that artifacts or structural distortions remain because no explicit content-preserving criterion is enforced (Lu et al., 2019).

4. Variants within style transfer: diversity and photorealism

The non-uniqueness of covariance or Gram matching motivated diversified arbitrary style transfer based on deep feature perturbation (DFP). In that formulation, the standard WCT coloring stage

$t(u) = T(u-\mu_c) + \mu_s,$7

is modified by inserting an orthogonal random matrix $t(u) = T(u-\mu_c) + \mu_s,$8: $t(u) = T(u-\mu_c) + \mu_s,$9 Because $T\Sigma_c T^T = \Sigma_s.$0, the Gram matrix is preserved exactly: $T\Sigma_c T^T = \Sigma_s.$1 The reported interpretation is that DFP traverses the style-specific feature space of all feature maps sharing the same Gram matrix, thereby producing diverse stylizations without retraining (Wang et al., 2019).

DFP is integrated into multi-level WCT-style pipelines as perturbed WCT (PWCT). Diversity is controlled by mixing the perturbed and unperturbed outputs,

$T\Sigma_c T^T = \Sigma_s.$2

and then blending with content via the usual $T\Sigma_c T^T = \Sigma_s.$3 parameter. The paper reports that deterministic WCT-based baselines have zero diversity under the paper’s diversity metrics, whereas Li et al. + DFP reaches Pixel distance $T\Sigma_c T^T = \Sigma_s.$4 and LPIPS $T\Sigma_c T^T = \Sigma_s.$5, Sheng et al. + DFP $T\Sigma_c T^T = \Sigma_s.$6, and Li et al. + DFP $T\Sigma_c T^T = \Sigma_s.$7. It also reports that arbitrary random matrices severely degrade quality, indicating that orthogonality, rather than the specific sampling distribution, is the property preserving style statistics (Wang et al., 2019).

A distinct line of work targets photorealistic transfer. WCT$T\Sigma_c T^T = \Sigma_s.$8 retains the whitening-coloring mechanism

$T\Sigma_c T^T = \Sigma_s.$9

but replaces lossy max-pooling and unpooling with Haar wavelet pooling and wavelet unpooling, which satisfy exact reconstruction conditions. The stated motivation is that ordinary WCT and PhotoWCT can distort object boundaries and amplify reconstruction errors when stylization is applied recursively across scales. WCT$T = \Sigma_s^{1/2}\Sigma_c^{-1/2},$0 instead performs progressive stylization within a single forward pass at layers such as conv1_1, conv2_1, conv3_1, and conv4_1, while preserving high-frequency structure through wavelet subbands $T = \Sigma_s^{1/2}\Sigma_c^{-1/2},$1, $T = \Sigma_s^{1/2}\Sigma_c^{-1/2},$2, $T = \Sigma_s^{1/2}\Sigma_c^{-1/2},$3, and $T = \Sigma_s^{1/2}\Sigma_c^{-1/2},$4 (Yoo et al., 2019).

The reported consequences are architectural and empirical. WCT$T = \Sigma_s^{1/2}\Sigma_c^{-1/2},$5 is described as the first and the only end-to-end model in that discussion that can stylize a $T = \Sigma_s^{1/2}\Sigma_c^{-1/2},$6 image in $T = \Sigma_s^{1/2}\Sigma_c^{-1/2},$7 seconds without post-processing, uses about $T = \Sigma_s^{1/2}\Sigma_c^{-1/2},$8 of the GPU memory of PhotoWCT, and yields a stable video stylization without temporal constraints. Human evaluation preferred it in $T = \Sigma_s^{1/2}\Sigma_c^{-1/2},$9 for fewest artifacts, $$f_c f_c^\top = E_c D_c E_c^\top,\qquad f_s f_s^\top = E_s D_s E_s^\top,$$0 for best stylization, and $$f_c f_c^\top = E_c D_c E_c^\top,\qquad f_s f_s^\top = E_s D_s E_s^\top,$$1 overall preference (Yoo et al., 2019).

5. Extensions to image translation, GANs, and normalization

WCT has been adapted to exemplar-based image-to-image translation through Group-wise Deep Whitening-and-Coloring Transformation (GDWCT). The motivation is that plain WCT uses explicit eigendecomposition, which is computationally intensive and difficult to differentiate through. GDWCT therefore replaces exact whitening with a learned approximation in which the encoder is regularized so that its output covariance approaches identity,

$$f_c f_c^\top = E_c D_c E_c^\top,\qquad f_s f_s^\top = E_s D_s E_s^\top,$$2

making mean subtraction a surrogate whitening step. Coloring is likewise approximated by a learned matrix $$f_c f_c^\top = E_c D_c E_c^\top,\qquad f_s f_s^\top = E_s D_s E_s^\top,$$3, with orthogonality encouraged by

$$f_c f_c^\top = E_c D_c E_c^\top,\qquad f_s f_s^\top = E_s D_s E_s^\top,$$4

The transform is made group-wise by splitting channels into $$f_c f_c^\top = E_c D_c E_c^\top,\qquad f_s f_s^\top = E_s D_s E_s^\top,$$5 groups, which reduces parameter count from about

$$f_c f_c^\top = E_c D_c E_c^\top,\qquad f_s f_s^\top = E_s D_s E_s^\top,$$6

The reported interpretation is that GDWCT is a trainable, group-wise, end-to-end approximation of WCT suitable for image translation (Cho et al., 2018).

In GAN generators, AdaWCT generalizes AdaIN by replacing per-channel normalization and affine modulation with a whitening matrix $$f_c f_c^\top = E_c D_c E_c^\top,\qquad f_s f_s^\top = E_s D_s E_s^\top,$$7 and a learned coloring matrix $$f_c f_c^\top = E_c D_c E_c^\top,\qquad f_s f_s^\top = E_s D_s E_s^\top,$$8: $$f_c f_c^\top = E_c D_c E_c^\top,\qquad f_s f_s^\top = E_s D_s E_s^\top,$$9 To avoid unstable exact SVD, the whitening matrix is approximated with the Newton-Schulz iterative method after shrinkage and normalization. Group-wise whitening and coloring are implemented with block-diagonal matrices; $$\hat{f}_c = E_c D_c^{-\frac12} E_c^\top f_c, \qquad \hat{f}_{cs} = E_s D_s^{\frac12} E_s^\top \hat{f}_c,$$0 degenerates to AdaIN, and $$\hat{f}_c = E_c D_c^{-\frac12} E_c^\top f_c, \qquad \hat{f}_{cs} = E_s D_s^{\frac12} E_s^\top \hat{f}_c,$$1 gives full WCT. In StarGANv2 on AFHQ, reported averages over all domains are: reference-guided AdaIN $$\hat{f}_c = E_c D_c^{-\frac12} E_c^\top f_c, \qquad \hat{f}_{cs} = E_s D_s^{\frac12} E_s^\top \hat{f}_c,$$2 FID and $$\hat{f}_c = E_c D_c^{-\frac12} E_c^\top f_c, \qquad \hat{f}_{cs} = E_s D_s^{\frac12} E_s^\top \hat{f}_c,$$3 LPIPS versus AdaWCT $$\hat{f}_c = E_c D_c^{-\frac12} E_c^\top f_c, \qquad \hat{f}_{cs} = E_s D_s^{\frac12} E_s^\top \hat{f}_c,$$4 and $$\hat{f}_c = E_c D_c^{-\frac12} E_c^\top f_c, \qquad \hat{f}_{cs} = E_s D_s^{\frac12} E_s^\top \hat{f}_c,$$5; latent-guided AdaIN $$\hat{f}_c = E_c D_c^{-\frac12} E_c^\top f_c, \qquad \hat{f}_{cs} = E_s D_s^{\frac12} E_s^\top \hat{f}_c,$$6 and $$\hat{f}_c = E_c D_c^{-\frac12} E_c^\top f_c, \qquad \hat{f}_{cs} = E_s D_s^{\frac12} E_s^\top \hat{f}_c,$$7 versus AdaWCT $$\hat{f}_c = E_c D_c^{-\frac12} E_c^\top f_c, \qquad \hat{f}_{cs} = E_s D_s^{\frac12} E_s^\top \hat{f}_c,$$8 and $$\hat{f}_c = E_c D_c^{-\frac12} E_c^\top f_c, \qquad \hat{f}_{cs} = E_s D_s^{\frac12} E_s^\top \hat{f}_c,$$9. The same study reports the best FID at group size $\hat{f}_{cs}\hat{f}_{cs}^\top = f_s f_s^\top,$0 and states that this is the first use of WCT for style injection in GANs (Dufour et al., 2022).

A related but distinct adaptation is Whitening and Coloring batch transform for GANs. There, whitening is applied to batch features using a matrix $\hat{f}_{cs}\hat{f}_{cs}^\top = f_s f_s^\top,$1 satisfying

$\hat{f}_{cs}\hat{f}_{cs}^\top = f_s f_s^\top,$2

followed by a learned coloring transform

$\hat{f}_{cs}\hat{f}_{cs}^\top = f_s f_s^\top,$3

Conditional versions use class-specific coloring filters $\hat{f}_{cs}\hat{f}_{cs}^\top = f_s f_s^\top,$4 and $\hat{f}_{cs}\hat{f}_{cs}^\top = f_s f_s^\top,$5, optionally with a shared class-agnostic branch. The paper frames this as a GAN-oriented instantiation of the same whiten-then-color principle: whitening improves optimization conditioning, while coloring restores representational flexibility and encodes conditioning information more expressively than conditional Batch Normalization (Siarohin et al., 2018).

6. Domain-specific adaptations, limitations, and recurring misconceptions

A recent domain-specific extension is UStyle, which reformulates WCT for underwater imagery through depth-aware whitening and coloring transform (DA-WCT). Standard WCT is characterized there as a global covariance-alignment operator,

$\hat{f}_{cs}\hat{f}_{cs}^\top = f_s f_s^\top,$6

or equivalently

$\hat{f}_{cs}\hat{f}_{cs}^\top = f_s f_s^\top,$7

The stated problem is that underwater “style” is tied to waterbody optics, depth-dependent attenuation, and backscatter; direct global WCT can therefore cause global averaging, oversaturation, or loss of object detail. DA-WCT first extracts a style waterbody component $\hat{f}_{cs}\hat{f}_{cs}^\top = f_s f_s^\top,$8 using a physics-based procedure based on the style depth map and the $\hat{f}_{cs}\hat{f}_{cs}^\top = f_s f_s^\top,$9 farthest pixels, then fuses stylized and original features with a depth-dependent gate

$FF^\top$0

followed by a global style-strength parameter

$FF^\top$1

The paper reports that, in every ablation configuration, DA-WCT outperforms plain WCT on PSNR, SSIM, and RMSE; for example, in the full model, $FF^\top$2 improves from PSNR $FF^\top$3 to $FF^\top$4, SSIM $FF^\top$5 to $FF^\top$6, and RMSE $FF^\top$7 to $FF^\top$8, while $FF^\top$9 improves from PSNR $$L = \|I_{o}-I_{i}\|^{2}_{2} + \lambda\|\Phi(I_{o}) - \Phi(I_{i})\|^{2}_{2},$$0 to $$L = \|I_{o}-I_{i}\|^{2}_{2} + \lambda\|\Phi(I_{o}) - \Phi(I_{i})\|^{2}_{2},$$1, SSIM $$L = \|I_{o}-I_{i}\|^{2}_{2} + \lambda\|\Phi(I_{o}) - \Phi(I_{i})\|^{2}_{2},$$2 to $$L = \|I_{o}-I_{i}\|^{2}_{2} + \lambda\|\Phi(I_{o}) - \Phi(I_{i})\|^{2}_{2},$$3, and RMSE $$L = \|I_{o}-I_{i}\|^{2}_{2} + \lambda\|\Phi(I_{o}) - \Phi(I_{i})\|^{2}_{2},$$4 to $$L = \|I_{o}-I_{i}\|^{2}_{2} + \lambda\|\Phi(I_{o}) - \Phi(I_{i})\|^{2}_{2},$$5 (Siddique et al., 14 Mar 2025).

Several limitations recur across the WCT literature. Exact WCT requires eigendecomposition or SVD of covariance matrices, with computational complexity described as roughly $$L = \|I_{o}-I_{i}\|^{2}_{2} + \lambda\|\Phi(I_{o}) - \Phi(I_{i})\|^{2}_{2},$$6 for an $$L = \|I_{o}-I_{i}\|^{2}_{2} + \lambda\|\Phi(I_{o}) - \Phi(I_{i})\|^{2}_{2},$$7 covariance matrix, and its backpropagation is repeatedly characterized as non-trivial or unstable (Cho et al., 2018). Full coloring matrices also incur parameter cost: in AdaWCT, for $$L = \|I_{o}-I_{i}\|^{2}_{2} + \lambda\|\Phi(I_{o}) - \Phi(I_{i})\|^{2}_{2},$$8 and a 512-dimensional style vector, the projection network would require at least

$$L = \|I_{o}-I_{i}\|^{2}_{2} + \lambda\|\Phi(I_{o}) - \Phi(I_{i})\|^{2}_{2},$$9

parameters per style injection block, motivating group-wise approximations (Dufour et al., 2022). Within style transfer proper, a frequent misconception is that exact covariance matching is sufficient for good stylization. The optimal-transport analysis argues against this: covariance matching is underconstrained, WCT is not unique up to orthogonal rotations, and explicit content preservation changes the solution class (Lu et al., 2019). Another recurring misconception is that WCT and AdaIN differ only in implementation detail; the literature consistently treats them as different statistical models, with AdaIN matching only channel-wise moments and WCT matching full inter-channel covariance (Lu et al., 2019, Dufour et al., 2022).