Controllable Feature Whitening

• Controllable Feature Whitening is a method that transforms features to remove or attenuate second-order dependencies while exposing control variables (e.g., interpolation, grouping, masking) for tailored applications.
• It is applied across domains such as style transfer, self-supervised learning, and bias mitigation by regulating covariance structure to balance performance and interpretability.
• Integrating tunable whitening operators into models improves optimization, interpretability, and fairness by dynamically managing feature correlations and domain-sensitive structures.

Searching arXiv for papers on controllable feature whitening and closely related whitening-based methods across style transfer, fairness, SSL, and domain adaptation. Controllable feature whitening denotes a family of methods in which whitening is not treated as a fixed preprocessing primitive, but as an explicit mechanism for regulating feature covariance, correlation, and domain- or bias-sensitive structure. Across the literature, the term covers several distinct but related uses: closed-form whitening and coloring in visual style transfer, regularized or partial whitening for linear-model interpretability, spectrum modulation in self-supervised learning, selective whitening for domain generalization, and last-layer whitening for fairness and bias mitigation. In all cases, the central operation is a transformation that removes or attenuates second-order dependencies—typically via covariance or correlation normalization—while exposing one or more control variables such as interpolation parameters, group structure, masking, layer placement, or spectral weighting (Li et al., 2017, Petiton et al., 22 Apr 2026, Cho et al., 27 Jul 2025).

1. Definition and mathematical basis

At its most general, feature whitening transforms a feature representation so that the transformed variables are zero-mean, decorrelated, and normalized in variance. In the canonical covariance form, a feature matrix is centered, a covariance or correlation matrix is estimated, and an inverse square root transform is applied. In "Universal Style Transfer via Feature Transforms" (Li et al., 2017), a VGG-19 feature tensor is flattened to $f \in \mathbb{R}^{C \times N}$, centered, and whitened by

$$\hat{f}_c = E_c D_c^{-\frac{1}{2}} E_c^\top \tilde{f}_c,$$

with $E_c$ and $D_c$ obtained from the eigendecomposition of the channel covariance. The paper explicitly states that this yields whitened feature maps satisfying $\hat{f}_c \hat{f}_c^\top = I$ (Li et al., 2017).

A related but distinct formulation appears in "Improving clinical interpretability of linear neuroimaging models through feature whitening" (Petiton et al., 22 Apr 2026). There, whitening is applied not globally but to anatomically motivated feature pairs using ZCA-cor. For a standardized pair $X_p \in \mathbb{R}^{n \times 2}$ with correlation matrix $R_p = U_p \Lambda_p U_p^\top$, the whitening matrix is

$W_p = U_p \Lambda_p^{-1/2} U_p^\top,$

and the whitened features are $Z_p = X_p W_p$, with $\mathrm{Corr}(Z_p) = I_2$ (Petiton et al., 22 Apr 2026). Unlike PCA or ICA whitening used for dimensionality reduction, this approach decorrelates anatomically informed pairs while retaining the full input signal (Petiton et al., 22 Apr 2026).

The same second-order logic reappears in later work, but with different objectives. In self-supervised learning, whitening is often defined as forcing a covariance or cross-correlation matrix toward the identity to avoid dimensional collapse (Weng et al., 2022, Mohamadi et al., 2024, Kalapos et al., 2024). In optimization-oriented work, whitening is defined as applying a matrix $$\hat{f}_c = E_c D_c^{-\frac{1}{2}} E_c^\top \tilde{f}_c,$$0 satisfying

$$\hat{f}_c = E_c D_c^{-\frac{1}{2}} E_c^\top \tilde{f}_c,$$1

to layer inputs or, equivalently, to weight gradients, in order to improve conditioning and approximate natural-gradient behavior (Markovich-Golan et al., 2020). In fairness-oriented work, whitening is applied to concatenated target and bias features to eliminate linear correlation between the subspaces fed to the final classifier (Cho et al., 27 Jul 2025).

These variants share a common invariant: whitening is a covariance-control operation. What makes it controllable is the introduction of explicit parameters or structural restrictions that determine which dependencies are removed, how strongly they are removed, and at what representational level.

2. Modes of controllability

Several control mechanisms recur across the literature. The earliest explicit instance in the supplied corpus is the interpolation parameter $$\hat{f}_c = E_c D_c^{-\frac{1}{2}} E_c^\top \tilde{f}_c,$$2 in WCT-style transfer: $$\hat{f}_c = E_c D_c^{-\frac{1}{2}} E_c^\top \tilde{f}_c,$$3 Here, $$\hat{f}_c = E_c D_c^{-\frac{1}{2}} E_c^\top \tilde{f}_c,$$4 gives full style application, $$\hat{f}_c = E_c D_c^{-\frac{1}{2}} E_c^\top \tilde{f}_c,$$5 reconstructs the original content image, and intermediate values yield partial transfer in feature space (Li et al., 2017). The same paper also introduces layer-wise control and coarse-to-fine composition across VGG layers, so controllability is realized not only through interpolation but also through where whitening is applied in the hierarchy (Li et al., 2017).

A more direct regularization view appears in neuroimaging. "Improving clinical interpretability of linear neuroimaging models through feature whitening" defines a regularized whitening operator

$$\hat{f}_c = E_c D_c^{-\frac{1}{2}} E_c^\top \tilde{f}_c,$$6

which interpolates linearly between no whitening and full ZCA-cor whitening (Petiton et al., 22 Apr 2026). This makes the degree of decorrelation explicitly tunable. The limiting cases are exact: $$\hat{f}_c = E_c D_c^{-\frac{1}{2}} E_c^\top \tilde{f}_c,$$7 gives the identity transform and $$\hat{f}_c = E_c D_c^{-\frac{1}{2}} E_c^\top \tilde{f}_c,$$8 gives full whitening (Petiton et al., 22 Apr 2026). The paper uses $$\hat{f}_c = E_c D_c^{-\frac{1}{2}} E_c^\top \tilde{f}_c,$$9 for left–right pairs and $E_c$0 for GM–CSF pairs, because some left–right correlations may be diagnosis-dependent whereas GM–CSF relations are expected to be stable (Petiton et al., 22 Apr 2026).

In style-conditioned generation, controllability is tied to grouping. "AdaWCT: Adaptive Whitening and Coloring Style Injection" generalizes AdaIN by replacing diagonal scale-and-shift with whitening and coloring matrices. It introduces group-wise whitening and coloring, where channels are partitioned into groups of size $E_c$1. This yields a continuum:

• $E_c$2: the method degrades naturally to AdaIN;
• $E_c$3: one obtains full WCT;
• intermediate $E_c$4: covariance is controlled within groups but not across groups (Dufour et al., 2022).

In domain-sensitive medical segmentation, controllability takes the form of masking rather than interpolation. "Domain-incremental Cardiac Image Segmentation with Style-oriented Replay and Domain-sensitive Feature Whitening" estimates a variance matrix $E_c$5 over covariance entries across same-content, different-style feature pairs, clusters those entries with $E_c$6-means, and constructs a binary mask $E_c$7 that selects only the most domain-sensitive interactions for whitening-style suppression (Li et al., 2022). The clustering parameter $E_c$8 controls the proportion of covariance entries that are penalized; the paper reports that $E_c$9 works best in its setting (Li et al., 2022).

Fairness-oriented whitening introduces a different control variable. "Controllable Feature Whitening for Hyperparameter-Free Bias Mitigation" defines a re-weighted covariance

$D_c$0

where $D_c$1 is the covariance under the biased training distribution and $D_c$2 is a synthetic unbiased covariance (Cho et al., 27 Jul 2025). According to the paper, $D_c$3 tends toward demographic parity and $D_c$4 tends toward equalized odds, so $D_c$5 controls the fairness–utility trade-off through the whitening transform itself (Cho et al., 27 Jul 2025).

A plausible synthesis is that controllability is realized through four recurring design variables: interpolation strength, grouping granularity, spatial or covariance masking, and choice of target covariance.

3. Closed-form whitening, coloring, and covariance matching

The best-known closed-form formulation is whitening–coloring transform (WCT) in arbitrary style transfer. Given content covariance $D_c$6 and style covariance $D_c$7, the method first whitens content features and then colors them with the style covariance: $D_c$8 The transformed feature matches the style covariance in one shot, and the paper notes that this shares similar spirit with Gram-matrix-based neural style transfer (Li et al., 2017). Because the decoder is trained only for reconstruction and not on any pre-defined styles, the method is universal with respect to styles (Li et al., 2017).

The same whitening–coloring principle is adapted to GAN style injection in AdaWCT. There, activations $D_c$9 are centered, whitened by a data-dependent matrix $\hat{f}_c \hat{f}_c^\top = I$0, then recolored by a style-dependent matrix $\hat{f}_c \hat{f}_c^\top = I$1, producing

$\hat{f}_c \hat{f}_c^\top = I$2

The whitening matrix is approximated with Newton–Schulz iterations rather than eigendecomposition, and grouping is used to make the coloring matrices tractable (Dufour et al., 2022). Quantitatively, the paper reports that AdaWCT improves both FID and LPIPS over AdaIN on AFHQ, for both reference-guided and latent-guided translation (Dufour et al., 2022).

Closed-form covariance control is also central to adversarial defense. "Diminishing the Effect of Adversarial Perturbations via Refining Feature Representation" uses whitening and coloring at arbitrary layers of pretrained classifiers. An adversarial representation is whitened using its own covariance, then recolored using the statistics of a reference image $\hat{f}_c \hat{f}_c^\top = I$3, often chosen by nearest-neighbor search in image space (Asadi et al., 2019). The method is training-free and can be inserted into any desirable layer of any arbitrary model (Asadi et al., 2019). The paper reports strong gains on MNIST-like benchmarks under transferred attacks when WCT is applied to later VGG layers, and it argues that deeper application generally yields better results if only a single WCT is used (Asadi et al., 2019).

A unifying interpretation is that closed-form whitening and coloring turns feature control into an operator-design problem: remove second-order structure from one representation, then inject second-order structure from another. Controllability arises because the injected structure can be selected, blended, grouped, or spatially restricted.

4. Whitening as spectrum control in self-supervised learning

In self-supervised learning, whitening is closely tied to collapse avoidance. "An Investigation into Whitening Loss for Self-supervised Learning" shows that batch-whitening-based methods do not actually impose whitening constraints on the embeddings; instead, they require the embeddings to be full-rank, and that full-rank constraint is sufficient to avoid dimensional collapse (Weng et al., 2022). The paper introduces normalized rank and stable rank as indicators and proposes channel whitening with random group partition (CW-RGP), which replaces large-batch covariance whitening with whitening across samples inside random channel groups (Weng et al., 2022). Group count and random partitioning act as control variables over whitening strength and robustness to small batch sizes (Weng et al., 2022).

"Modulate Your Spectrum in Self-Supervised Learning" generalizes whitening into a broader spectral-transformation framework. For covariance $\hat{f}_c \hat{f}_c^\top = I$4, a spectral transformation is defined by a unary function $\hat{f}_c \hat{f}_c^\top = I$5, giving transformed features

$\hat{f}_c \hat{f}_c^\top = I$6

and transformed eigenvalues $\hat{f}_c \hat{f}_c^\top = I$7 (Weng et al., 2023). Whitening is the special case $\hat{f}_c \hat{f}_c^\top = I$8 (Weng et al., 2023). The paper then studies the power family $\hat{f}_c \hat{f}_c^\top = I$9, where $X_p \in \mathbb{R}^{n \times 2}$0 gives identity, $X_p \in \mathbb{R}^{n \times 2}$1 gives whitening, and nearby values can also avoid collapse when the transformed spectrum remains well-conditioned (Weng et al., 2023). Its IterNorm with trace loss (INTL) uses iteration count $X_p \in \mathbb{R}^{n \times 2}$2 and trace-loss weight $X_p \in \mathbb{R}^{n \times 2}$3 as explicit spectrum-control parameters, and the paper proves that INTL drives the normalized eigenvalues toward equality (Weng et al., 2023).

A different line of work argues that whitening should be complemented, not merely strengthened. "Direct Coloring for Self-Supervised Enhanced Feature Decoupling" introduces a hybrid scheme in which an early coloring projector is trained to match a target cross-correlation matrix $X_p \in \mathbb{R}^{n \times 2}$4, while a final whitening head enforces decorrelation at the output (Mohamadi et al., 2024). The overall loss is

$X_p \in \mathbb{R}^{n \times 2}$5

so $X_p \in \mathbb{R}^{n \times 2}$6 controls the balance between covariance shaping and whitening (Mohamadi et al., 2024). The paper reports that coloring improves variance and helps avoid complete collapse, while whitening still handles dimensional collapse (Mohamadi et al., 2024).

The most deployment-oriented claim in this area comes from "Whitening Consistently Improves Self-Supervised Learning", which adds ZCA-style whitening, implemented with IterNorm, as the final layer of the encoder and reports consistent linear-probe and k-NN gains across BYOL, VICReg, SimCLR, Barlow Twins, DINO, and SwAV, over ResNet-18 and ConvNeXtV2-Pico, on CIFAR10, STL10, and Tiny-ImageNet (Kalapos et al., 2024). The paper also proposes anisotropy, mean absolute feature correlation, and mean feature standard deviation as diagnostics for representation quality and collapse (Kalapos et al., 2024).

Together, these works broaden controllable feature whitening from covariance normalization to spectral design. The control variables are no longer only means and covariances, but also eigenvalue profiles, grouping structure, iteration counts, and auxiliary covariance targets.

5. Bias mitigation, fairness, and interpretability

In fairness and bias mitigation, controllable whitening is used to suppress spurious correlations without adversarial learning. "Controllable Feature Whitening for Hyperparameter-Free Bias Mitigation" freezes a biased target encoder $X_p \in \mathbb{R}^{n \times 2}$7, trains a bias encoder $X_p \in \mathbb{R}^{n \times 2}$8, concatenates the resulting features $X_p \in \mathbb{R}^{n \times 2}$9, and applies a whitening module $R_p = U_p \Lambda_p U_p^\top$0 so that the whitened target subspace $R_p = U_p \Lambda_p U_p^\top$1 and whitened bias subspace $R_p = U_p \Lambda_p U_p^\top$2 are linearly uncorrelated (Cho et al., 27 Jul 2025). The first half is used for target prediction and the second half for bias prediction. Because the covariance after whitening is approximately identity, the cross-covariance between the two subspaces is approximately zero, which prevents a linear last-layer classifier from exploiting bias information in $R_p = U_p \Lambda_p U_p^\top$3 (Cho et al., 27 Jul 2025).

The same paper connects whitening to two fairness criteria. Demographic parity is associated with whitening under the biased covariance $R_p = U_p \Lambda_p U_p^\top$4, while equalized odds is associated with whitening under the synthetic unbiased covariance $R_p = U_p \Lambda_p U_p^\top$5; interpolating with $R_p = U_p \Lambda_p U_p^\top$6 controls the trade-off (Cho et al., 27 Jul 2025). The paper states that $R_p = U_p \Lambda_p U_p^\top$7 works robustly across Corrupted CIFAR-10, Biased FFHQ, WaterBirds, and Celeb-A (Cho et al., 27 Jul 2025).

A conceptually related but distinct application appears in neuroimaging interpretability. Whitening is used there not to remove socially sensitive information, but to disentangle overlapping anatomical contributions in linear models. The paper emphasizes that linear weights reflect conditional effects and are unstable under strong feature correlation; pairwise ZCA-cor whitening re-parameterizes the problem so that weights in whitened space more directly reflect independent axes of variation (Petiton et al., 22 Apr 2026). It then maps them back by

$R_p = U_p \Lambda_p U_p^\top$8

preserving the ordering of feature importance within each whitened pair (Petiton et al., 22 Apr 2026). Empirically, the paper reports that whitening preserves predictive performance while improving the alignment of weights with clinically plausible regions, and that a Student’s $R_p = U_p \Lambda_p U_p^\top$9-test across CV folds found no significant performance difference ($W_p = U_p \Lambda_p^{-1/2} U_p^\top,$0) (Petiton et al., 22 Apr 2026).

Whitening is also used for domain generalization in medical segmentation. "Domain-incremental Cardiac Image Segmentation with Style-oriented Replay and Domain-sensitive Feature Whitening" computes current and replayed features for same-content images with different styles, standardizes them, estimates channel-wise covariance matrices, and then identifies domain-sensitive covariance entries by their variance across domains (Li et al., 2022). A mask $W_p = U_p \Lambda_p^{-1/2} U_p^\top,$1 selects high-variance entries, and the whitening loss penalizes only those entries: $W_p = U_p \Lambda_p^{-1/2} U_p^\top,$2 This selective whitening improves past-domain retention, current-domain adaptation, and unseen-domain generalization on the M&Ms dataset (Li et al., 2022).

A plausible implication is that controllable whitening is especially attractive when the nuisance variable is known or partially observable. In such cases, the covariance itself becomes a mechanism for encoding a fairness or invariance criterion.

6. Structured, grouped, and geometry-aware whitening

A recurrent design principle is that full whitening is often unnecessary, unstable, or too destructive, so structure is imposed on the whitening operator itself. Grouping is the simplest form. In unsupervised domain adaptation, "Unsupervised Domain Adaptation using Feature-Whitening and Consensus Loss" introduces domain-specific whitening transform (DWT) layers that whiten source and target batches separately but map both to an identity-covariance reference distribution (Roy et al., 2019). Feature grouping controls decorrelation strength: $W_p = U_p \Lambda_p^{-1/2} U_p^\top,$3 degenerates to BatchNorm-style standardization, larger $W_p = U_p \Lambda_p^{-1/2} U_p^\top,$4 performs stronger decorrelation, and overly large groups can destabilize covariance estimation (Roy et al., 2019). The paper reports that moderate group size, such as $W_p = U_p \Lambda_p^{-1/2} U_p^\top,$5, works best in its SVHN→MNIST ablation, and that increasing the number of DWT layers improves performance (Roy et al., 2019).

Grouping also appears in sentence representation learning. "Whitening-based Contrastive Learning of Sentence Embeddings" defines shuffled group whitening (SGW), which randomly permutes feature dimensions, partitions them into groups, whitens each group independently with ZCA, then inverts the permutation (Zhuo et al., 2023). Repeating SGW with multiple random shuffles produces multiple positive views for the same sample. The paper reports that group whitening without shuffling improves over SimCSE, while shuffled group whitening yields the best STS results and improves both alignment and uniformity (Zhuo et al., 2023).

More elaborate structure appears in hyperbolic representation learning. "Generalizable Audio Deepfake Detection via Hierarchical Structure Learning and Feature Whitening in Poincaré sphere" defines Poincaré Feature Whitening (PFW), where Euclidean covariance is replaced by a similarity matrix computed with hyperbolic distance between Poincaré features (Yang et al., 3 Aug 2025). Original and augmented samples are used to estimate which inter-dimension similarities vary most under domain perturbation, then a mask selects the top $W_p = U_p \Lambda_p^{-1/2} U_p^\top,$6 fraction of domain-sensitive entries, with separate ratios $W_p = U_p \Lambda_p^{-1/2} U_p^\top,$7 and $W_p = U_p \Lambda_p^{-1/2} U_p^\top,$8 for bonafide and spoof classes (Yang et al., 3 Aug 2025). The resulting PFW loss suppresses those entries and improves EER on ASVspoof and In-The-Wild benchmarks according to the paper’s ablation (Yang et al., 3 Aug 2025).

In optimization, structure appears not in the covariance target but in the update mechanism. "Feature Whitening via Gradient Transformation for Improved Convergence" proves that whitening activations is equivalent to transforming weight gradients with a matrix

$W_p = U_p \Lambda_p^{-1/2} U_p^\top,$9

and then introduces a recursive alternative that reduces the condition number of the sample covariance without full eigendecomposition (Markovich-Golan et al., 2020). The parameters $Z_p = X_p W_p$0, $Z_p = X_p W_p$1, $Z_p = X_p W_p$2, $Z_p = X_p W_p$3, and $Z_p = X_p W_p$4 control adaptation speed, smoothing, whitening gain, dominant-eigenvalue reduction, and leakage, respectively (Markovich-Golan et al., 2020). This makes whitening strength a practical optimization knob rather than a fixed normalization step.

These examples show that controllable whitening increasingly means structured whitening: the operator is constrained by groups, masks, manifold geometry, or update rules so that only selected correlations are removed.

7. Trade-offs, misconceptions, and research directions

A common misconception is that whitening always means full covariance identity everywhere. Several papers explicitly contradict this. In WCT-style transfer, the original paper does not define partial whitening in the covariance domain, but controllability is still realized indirectly through $Z_p = X_p W_p$5, layer choice, and the number of WCT stages (Li et al., 2017). In neuroimaging, the regularized operator $Z_p = X_p W_p$6 is not a weaker approximation of ZCA-cor in a numerical sense; it is a deliberate interpolation between preserving and removing correlation (Petiton et al., 22 Apr 2026). In SSL, whitening losses may only enforce full rank rather than exact identity covariance, depending on how whitening is implemented (Weng et al., 2022).

Another misconception is that stronger whitening is always better. Several supplied papers show explicit trade-offs. Full whitening or large-group whitening can be numerically unstable or harmful when covariance estimation is poor (Roy et al., 2019, Zhuo et al., 2023). In sparse autoencoder learning, PCA whitening improves sparse probing, feature disentanglement, and related interpretability metrics, but causes minor drops in reconstruction quality and explained variance (Saraswatula et al., 17 Nov 2025). In fairness mitigation, $Z_p = X_p W_p$7 may reduce demographic-parity violations more strongly but can damage utility, whereas $Z_p = X_p W_p$8 favors equalized odds; the best practical choice is an intermediate value (Cho et al., 27 Jul 2025).

A third misconception is that whitening is only a preprocessing trick. The corpus shows it functioning as a train-time loss, a final encoder layer, a style-injection block, a domain-alignment layer, a gradient preconditioner, and a selective regularizer (Dufour et al., 2022, Kalapos et al., 2024, Roy et al., 2019, Markovich-Golan et al., 2020, Li et al., 2022). This suggests that whitening has become less a single algorithm than a design pattern centered on second-order control.

Several directions are suggested directly by the supplied papers. The WCT framework naturally supports whitening exponents, rank truncation, and spatially varying controls, though these are presented as straightforward extensions rather than implemented components (Li et al., 2017). The neuroimaging paper notes that group-specific $Z_p = X_p W_p$9 parameters generalize immediately to arbitrary grouped features (Petiton et al., 22 Apr 2026). Spectral-transformation SSL explicitly invites functions beyond whitening that maintain a well-conditioned spectrum (Weng et al., 2023). PFW in hyperbolic space suggests that covariance surrogates need not be Euclidean if the representation geometry is non-Euclidean (Yang et al., 3 Aug 2025). Finally, the sparse-autoencoder results imply that interpretability and reconstruction may lie on different points of a whitening-controlled Pareto frontier (Saraswatula et al., 17 Nov 2025).

Controllable feature whitening is therefore best understood not as a single method, but as a technical program: use whitening or whitening-like operators to manipulate second-order structure, and expose those manipulations through explicit controls that encode the desired balance among style fidelity, invariance, interpretability, optimization, fairness, and generalization (Li et al., 2017, Petiton et al., 22 Apr 2026, Cho et al., 27 Jul 2025).