In-Batch Whitening for Deep Learning

• In-batch whitening is a transformation that uses mini-batch statistics to center, decorrelate, and rescale vector representations to achieve near-identity covariance.
• It utilizes operators such as ZCA, PCA, and Cholesky whitening to improve optimization conditioning and control gradient variance effectively.
• Its applications span contrastive learning, GANs, and imbalanced classification, though its performance hinges on factors like batch size and transform stability.

In-batch whitening denotes a batch-dependent transformation that uses statistics of the current mini-batch to center, decorrelate, and rescale vector representations so that their covariance is approximately identity or, in spectral terms, their second moment is driven toward isotropy. In the literature, the term covers both whitening layers inserted into neural networks and whitening applied to the current batch of embeddings inside a contrastive or non-contrastive objective. The shared template is to compute a batch mean, covariance, or second moment, apply an inverse square root transform, and continue the forward computation with the whitened outputs rather than treating whitening as a global preprocessing step (Huang et al., 2018, Ochieng, 7 Oct 2025).

1. Origins and conceptual scope

The modern formulation of in-batch whitening emerged as a direct extension of Batch Normalization. Batch Normalization centers and scales each feature dimension independently, but it does not remove cross-dimension correlations. Decorrelated Batch Normalization (DBN) made the stronger proposal that activations within each mini-batch should be whitened, not merely standardized, with the stated goal of improving conditioning, making gradient descent behave more like a second-order method, and supporting approximate dynamical isometry (Huang et al., 2018).

Subsequent work split the idea into several lines. One line kept the normalization-layer viewpoint and focused on more efficient or more stable whitening operators, including Newton-iteration-based Iterative Normalization (IterNorm) and online Stochastic Whitening Batch Normalization (SWBN) (Huang et al., 2019, Zhang et al., 2021). A second line moved whitening into representation learning objectives, where whitening acts as an anti-collapse constraint in self-supervised learning, as in W-MSE and later analyses of whitening loss (Ermolov et al., 2020, Weng et al., 2022). A third line specialized whitening to task structure, for example before the classifier in imbalanced recognition, in generator blocks for GANs, or in concept-aligned latent spaces for interpretability (Zhang, 2024, Siarohin et al., 2018, Chen et al., 2020).

This broader usage makes the term context-sensitive. In normalization papers, in-batch whitening usually means whitening hidden activations inside the network. In contrastive learning, it can mean whitening the current mini-batch embeddings before computing similarities. The 2025 spectral analysis of InfoNCE is explicit that its “in-batch whitening” is performed within the current batch of $\ell_2$-normalized embeddings and is not a global preprocessing step (Ochieng, 7 Oct 2025).

2. Core operators and implementation patterns

The canonical operator computes a batch mean and covariance, then applies an inverse square root. For vector-valued activations $\mathbf{X}\in\mathbb{R}^{d\times m}$, DBN writes

$$\mathbf{\mu} = \frac{1}{m}\mathbf{X}\mathbf{1}, \qquad \Sigma = \frac{1}{m}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)^T+\epsilon\mathbf{I},$$

followed by

$$\phi(\mathbf{X})=\Sigma^{-1/2}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T).$$

This gives transformed activations with approximately identity covariance (Huang et al., 2018).

For contrastive embeddings, the same template is expressed with a batch second moment. Given $\ell_2$-normalized embeddings $z_1,\dots,z_n\in\mathbb S^{d-1}$, the 2025 spectral work forms

$$\hat{\Sigma}_t=\frac{1}{n}\sum_{i=1}^n z_i z_i^\top, \qquad \Sigma_\varepsilon=\hat{\Sigma}_t+\varepsilon I, \qquad \hat z_i=\Sigma_\varepsilon^{-1/2} z_i, \qquad \tilde z_i=\frac{\hat z_i}{\|\hat z_i\|},$$

with $\varepsilon=10^{-5}$, and uses $\tilde z_i$ in the contrastive computation. The stated purpose is to push the batch spectrum toward isotropy (Ochieng, 7 Oct 2025).

Several matrix square-root constructions recur across the literature:

Whitening form	Representative expression	Salient property
ZCA whitening	$D\Lambda^{-1/2}D^T$	rotates back to the original axis system
PCA whitening	$\mathbf{X}\in\mathbb{R}^{d\times m}$0	changes the coordinate system
Cholesky whitening	$\mathbf{X}\in\mathbb{R}^{d\times m}$1	efficient triangular form
Iterative / stochastic whitening	Newton iterations or online $\mathbf{X}\in\mathbb{R}^{d\times m}$2 updates	avoids exact eigendecomposition at each step

ZCA whitening is the preferred exact form in several discriminative settings. DBN reports that PCA whitening causes stochastic axis swapping, whereas ZCA whitening does not suffer from this problem and therefore permits successful learning (Huang et al., 2018). Whitening-Net also uses ZCA whitening before the linear classifier in imbalanced classification (Zhang, 2024). By contrast, W-MSE and Whitening and Coloring for GANs use Cholesky-based whitening, with $\mathbf{X}\in\mathbb{R}^{d\times m}$3 in the self-supervised setting and $\mathbf{X}\in\mathbb{R}^{d\times m}$4 in the GAN setting (Ermolov et al., 2020, Siarohin et al., 2018).

Approximate operators were introduced mainly for efficiency and stability. IterNorm computes the inverse square root using Newton iterations after trace normalization of the covariance, thereby avoiding eigendecomposition and relying only on matrix multiplications (Huang et al., 2019). SWBN maintains a whitening matrix $\mathbf{X}\in\mathbb{R}^{d\times m}$5 online, applies

$\mathbf{X}\in\mathbb{R}^{d\times m}$6

and updates $\mathbf{X}\in\mathbb{R}^{d\times m}$7 recursively with a small step size instead of recomputing exact whitening from scratch (Zhang et al., 2021).

3. Conditioning, stochasticity, and spectral control

The standard rationale for in-batch whitening is that decorrelation improves optimization beyond what per-channel standardization can provide. DBN states that whitening improves the conditioning of the covariance matrix, which improves the conditioning of the Hessian or Fisher information and can lead to faster optimization, more stable gradients, approximate dynamical isometry, and improved generalization (Huang et al., 2018). SWBN adopts the same motivation, explicitly framing whitening as a way to reduce dependence among features, make the optimization landscape better conditioned, accelerate convergence, and improve generalization (Zhang et al., 2021).

A central complication is that whitening is itself stochastic because the transform depends on the sampled mini-batch. “An Investigation into the Stochasticity of Batch Whitening” formalizes this with Stochastic Normalization Disturbance (SND), treating the whitened output of a fixed sample as a random variable induced by the random batch. That study reports that PCA whitening has the largest SND, BN the smallest, and ZCA smaller SND than CD, and argues that this stochasticity correlates well with optimization behavior during training (Huang et al., 2020). This provides a mechanism for why whitening transforms with comparable conditioning can behave very differently in practice.

IterNorm develops the same point into an optimization–generalization trade-off. It argues that exact whitening is not always optimal because full whitening can amplify tiny eigen-directions and raise stochastic normalization disturbance, especially in high dimension. This is the basis for its claim that controlled or partial whitening can outperform exact whitening, and for its explanation of why group-wise whitening often performs better than full whitening (Huang et al., 2019). The same trade-off appears in group-based analyses of representational capacity: stronger normalization constraints can improve conditioning while simultaneously restricting feature diversity (Huang et al., 2020).

The 2025 spectral analysis of InfoNCE shifts the emphasis from covariance conditioning to the batch spectrum. There the relevant anisotropy proxy is the top eigenvalue of the batch second moment,

$\mathbf{X}\in\mathbb{R}^{d\times m}$8

with perfect isotropy corresponding to $\mathbf{X}\in\mathbb{R}^{d\times m}$9. The paper further uses the negatives-only top eigenvalue

$$\mathbf{\mu} = \frac{1}{m}\mathbf{X}\mathbf{1}, \qquad \Sigma = \frac{1}{m}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)^T+\epsilon\mathbf{I},$$0

and shows a deterministic batch-level ceiling

$$\mathbf{\mu} = \frac{1}{m}\mathbf{X}\mathbf{1}, \qquad \Sigma = \frac{1}{m}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)^T+\epsilon\mathbf{I},$$1

Its theoretical message is that anisotropy enters the gradient norm through the spectrum of the negatives, so whitening matters because it directly lowers the spectral term that controls gradient variability (Ochieng, 7 Oct 2025).

4. Self-supervised and contrastive learning

In self-supervised learning, in-batch whitening was used to replace or weaken the role of negatives. W-MSE proposes a whitening-based objective in which projected features in a mini-batch are whitened so that the batch has zero mean and identity covariance, and then only positive pairs are pulled together by an MSE-equivalent cosine distance. The method describes whitening as having a “scattering” effect on the batch samples and explicitly positions this as a non-collapse mechanism that removes the need for explicit negatives, momentum encoders, stop-gradient, or asymmetric twin-network setups (Ermolov et al., 2020).

The corresponding whitening operator is

$$\mathbf{\mu} = \frac{1}{m}\mathbf{X}\mathbf{1}, \qquad \Sigma = \frac{1}{m}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)^T+\epsilon\mathbf{I},$$2

with $$\mathbf{\mu} = \frac{1}{m}\mathbf{X}\mathbf{1}, \qquad \Sigma = \frac{1}{m}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)^T+\epsilon\mathbf{I},$$3 computed from the current mini-batch and implemented by Cholesky factorization. W-MSE applies whitening only to the last layer features, not to intermediate network layers, and introduces batch slicing because whitening estimated on a batch can be noisy. The paper recommends a sub-batch size around $$\mathbf{\mu} = \frac{1}{m}\mathbf{X}\mathbf{1}, \qquad \Sigma = \frac{1}{m}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)^T+\epsilon\mathbf{I},$$4 embedding dimension to avoid covariance instability (Ermolov et al., 2020).

Later analysis substantially revised the interpretation of whitening-based SSL. “An Investigation into Whitening Loss for Self-supervised Learning” argues that batch whitening methods such as W-MSE and Shuffled-DBN do not impose whitening constraints on the embedding itself; they only require the embedding to be full-rank. In that account, full-rankness is already sufficient to avoid dimensional collapse, while true whitening is a stronger condition. This reinterpretation motivates Channel Whitening with Random Group Partition (CW-RGP), which whitens along the channel dimension, uses random group partition, and is reported to remain robust when batch size is reduced from 256 down to 32 (Weng et al., 2022).

In contrastive learning proper, the spectral role of whitening is made explicit in the 2025 InfoNCE study. For anchor $$\mathbf{\mu} = \frac{1}{m}\mathbf{X}\mathbf{1}, \qquad \Sigma = \frac{1}{m}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)^T+\epsilon\mathbf{I},$$5, the paper writes

$$\mathbf{\mu} = \frac{1}{m}\mathbf{X}\mathbf{1}, \qquad \Sigma = \frac{1}{m}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)^T+\epsilon\mathbf{I},$$6

with per-sample gradient

$$\mathbf{\mu} = \frac{1}{m}\mathbf{X}\mathbf{1}, \qquad \Sigma = \frac{1}{m}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)^T+\epsilon\mathbf{I},$$7

The squared gradient variance $$\mathbf{\mu} = \frac{1}{m}\mathbf{X}\mathbf{1}, \qquad \Sigma = \frac{1}{m}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)^T+\epsilon\mathbf{I},$$8 is then bounded as

$$\mathbf{\mu} = \frac{1}{m}\mathbf{X}\mathbf{1}, \qquad \Sigma = \frac{1}{m}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T)^T+\epsilon\mathbf{I},$$9

where the paper interprets $$\phi(\mathbf{X})=\Sigma^{-1/2}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T).$$0 as the anisotropy-driven term and $$\phi(\mathbf{X})=\Sigma^{-1/2}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T).$$1 as a baseline term from softmax or positive-miss error. The whitening section states that whitening “suppresses anisotropy-driven fluctuations” by shrinking $$\phi(\mathbf{X})=\Sigma^{-1/2}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T).$$2 (Ochieng, 7 Oct 2025).

The accompanying experiment uses SimCLR on ImageNet-1k with ResNet-50, batch size $$\phi(\mathbf{X})=\Sigma^{-1/2}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T).$$3, temperature $$\phi(\mathbf{X})=\Sigma^{-1/2}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T).$$4, and a protocol that alternates 100 raw batches and 100 whitened batches starting at epoch 70. The paper reports that whitening keeps $$\phi(\mathbf{X})=\Sigma^{-1/2}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T).$$5 and reduces the 50-step rolling variance of the batch-mean squared gradient to about $$\phi(\mathbf{X})=\Sigma^{-1/2}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T).$$6 the raw level, i.e. raw/whitened $$\phi(\mathbf{X})=\Sigma^{-1/2}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T).$$7, matching the theoretical prediction that the leading variance term is proportional to spectral anisotropy (Ochieng, 7 Oct 2025).

5. Task-specific adaptations beyond self-supervision

In imbalanced classification, whitening was deployed to counter feature degeneracy at the classifier input. Whitening-Net diagnoses a failure mode in which the last hidden-layer features become highly linearly dependent, with many singular values nearly zero and elevated Pearson product-moment correlation coefficients between channels. Its remedy is a ZCA whitening transform applied only to the final hidden layer before the linear classifier, with moving averages of $$\phi(\mathbf{X})=\Sigma^{-1/2}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T).$$8 and $$\phi(\mathbf{X})=\Sigma^{-1/2}(\mathbf{X}-\mathbf{\mu}\mathbf{1}^T).$$9 used at inference. Because batch covariance is unstable under severe class imbalance, the framework adds Group-based Relatively Balanced Batch Sampler (GRBS) and Batch Embedded Training (BET) to stabilize covariance estimates (Zhang, 2024).

The mechanism is explicitly selective: whitening is inserted only before the classifier, not throughout the network. The paper reports that on CIFAR-10-LT with imbalance factor 200, the comparison ERM 66.4, DBN 67.1, DBN w/ Last Layer - Group Whitening 66.6, Ours w/ Last Layer - Channel Whitening 72.3, and Ours - WhiteningNet 76.4 supports the claim that last-layer channel whitening rather than group whitening is the mechanism that alleviates degeneration (Zhang, 2024).

In GANs, whitening is coupled to a learned coloring transform. “Whitening and Coloring batch transform for GANs” proposes Whitening and Coloring (WC), which first whitens activations using batch statistics and then applies a learnable multivariate coloring transform; the conditional version cWC replaces class-specific scalar BN parameters with class-specific coloring matrices. The whitening step uses Cholesky decomposition,

$\ell_2$0

and is inserted only in the generator, before each convolutional layer. The paper argues that full-feature whitening is particularly useful in GAN optimization because GAN training is highly unstable and benefits from better-conditioned feature transforms than standard BN can offer (Siarohin et al., 2018).

That study also emphasizes that whitening alone is insufficient and that coloring restores representational power. On CIFAR-10 with a projection discriminator, the reported conditional result for cWC SN + Projection Discriminator is $\ell_2$1 in Inception Score, and the paper states that Cholesky whitening is more stable and faster than a ZCA-based variant in this setting (Siarohin et al., 2018).

A distinct specialization appears in interpretability. Concept Whitening (CW) replaces a BN layer with a whitening module followed by an orthogonal rotation that aligns designated latent axes with chosen concepts. Its whitening stage uses the standard in-batch requirement of zero mean and identity covariance, and its rotation exploits the non-uniqueness of whitening to make concept directions axis-aligned. The paper reports much lower inter-concept similarity than standard BN networks and gives an average inter/intra ratio of about 0.35 for CW versus 0.94 for standard CNNs, presenting whitening here not as an optimizer but as a structural intervention for concept disentanglement (Chen et al., 2020).

6. Limitations, misconceptions, and related alternatives

A recurrent misconception is that greater isotropy necessarily improves downstream performance. The LLM embedding study “Whitening Not Recommended for Classification Tasks in LLMs” is explicit that whitening improves isotropy, often pushing IsoScore close to 1, but does not improve classification. Across eight embedding sources and seven SentEval classification datasets, it reports that whitening hurts classification accuracy for every model on every dataset. Representative average drops include BERT 79.47 $\ell_2$2 76.28, ChatGPT 84.45 $\ell_2$3 77.29, and LLaMA 81.45 $\ell_2$4 66.24 (Forooghi et al., 2024).

That paper also clarifies an important terminological boundary. Its whitening is a linear post-processing transform computed from a set of sentence embeddings, and it explicitly states that it does not describe any “in-batch whitening” in the contrastive-learning or batch-normalization sense. This distinction matters because batch-wise whitening used for evaluation-time post-processing is not equivalent to a differentiable whitening layer inside end-to-end training (Forooghi et al., 2024).

A second limitation is statistical reliability. DBN and BW-style methods depend heavily on sufficiently large batches because covariance estimation becomes noisy or rank-deficient when the batch is too small (Huang et al., 2019). Group Whitening (GW) was proposed partly to avoid normalization along the batch dimension altogether by whitening within channel groups of each sample, combining the decorrelation benefits of whitening with the batch-size robustness of Group Normalization. At the same time, GW develops a general caution: stronger normalization constraints can reduce representational capacity, and its analysis shows that the constraint number for GW scales quadratically in the group number $\ell_2$5, whereas for GN it scales linearly (Huang et al., 2020).

A third limitation is transform stability. PCA whitening is repeatedly identified as problematic in learned systems: DBN attributes its failure to stochastic axis swapping, and the stochasticity analysis reports that PCA retains large SND even when batch size increases (Huang et al., 2018, Huang et al., 2020). The practical consequence is that successful in-batch whitening systems are usually built around ZCA, Cholesky, IterNorm-style approximations, or online stochastic updates rather than raw PCA whitening.

Taken together, these results support a narrow but technically precise conclusion. In-batch whitening is not a single method but a family of batch-dependent decorrelation operators whose effects depend on where the transform is inserted, which whitening matrix is used, how batch statistics are estimated, and what objective the transformed features serve. It can improve conditioning, suppress anisotropy-driven gradient fluctuations, prevent collapse, stabilize specific architectures, or expose concept axes; but it can also increase stochasticity, over-constrain representations, or remove task-relevant geometry. The literature therefore treats in-batch whitening less as a universally beneficial normalization primitive than as a spectrum-shaping tool whose utility is highly architecture- and objective-dependent (Ochieng, 7 Oct 2025, Huang et al., 2019).