Cluster-Based Whitening Techniques

• Cluster-based whitening is an unsupervised learning technique that transforms latent embeddings to normalize covariance and reveal interpretable data clusters.
• It utilizes PCA-based whitening and spectral band decomposition to mitigate anisotropic scaling and feature redundancy for robust clustering outcomes.
• The method supports high-dimensional applications by incorporating corrected whitening, which restores asymptotic orthogonality and improves Gaussian mixture model estimation.

Cluster-based whitening is a family of techniques in unsupervised learning that employs linear transformations, notably principal component analysis (PCA)-based whitening, to standardize latent representations so that the structure of data clusters—often corresponding to interpretable semantic or generative classes—is revealed more transparently. This preprocessing enables the use of spectral and clustering methods for robust decomposition of data into constituent components and supports fine-grained analysis of geometric phenomena such as model hallucination, embedding space concentration, and large-dimensional estimation artifacts (Korun, 8 Mar 2026, Boudjemaa et al., 22 Sep 2025).

1. Foundations and Motivation

Whitening refers to affinely transforming a collection of vectors so that their covariance becomes the identity matrix. Formally, given a set $H = \{h_1, ..., h_N\} \subset \mathbb{R}^D$ of contextual or latent-space embeddings, whitening seeks a transformation $W$ such that, after mapping $w_i = W (h_i - \mu)$ (with $\mu$ the empirical mean), $\mathrm{Cov}(w) \approx I_M$ for retained dimensionality $M \leq D$.

In latent variable models, especially Gaussian mixture models (GMMs) and neural network embedding spaces, vanilla clustering is often hindered by dominant variance directions, anisotropic scaling, and feature redundancy. Whitening not only standardizes variance but, critically, decorrelates features so that geometric relationships among cluster centroids become amenable to isotropic clustering methods and downstream analysis. In the context of GMMs, it grants orthogonality among cluster means in the transformed space, which is essential for algebraic decompositions of higher-order moments (Boudjemaa et al., 22 Sep 2025). For deep contextual representations, whitening exposes fine “micro-signal” variations for delineating semantic or failure-driven cluster structure (Korun, 8 Mar 2026).

2. Algorithms: PCA-Whitening and Spectral Band Decomposition

The canonical whitening strategy proceeds via eigendecomposition of the empirical covariance:

• Given $h_i \in \mathbb{R}^D$, compute $\mu = (1/N)\sum_i h_i$ and covariance $\Sigma = (1/N)\sum_i (h_i - \mu)(h_i - \mu)^T$.
• Perform $\Sigma = U \Lambda U^T$, where $W$0 is orthogonal and $W$1 (sorted descending).
• Apply a small ridge $W$2 ($W$3) to each eigenvalue for numerical stability.
• Construct the whitening matrix using the top $W$4 eigenpairs:

$W$5

• Whitened representations: $W$6.

Variance preservation is tuned by $W$7 (e.g., $W$8 to retain $W$9 variance in GPT-2-small, $w_i = W (h_i - \mu)$0) (Korun, 8 Mar 2026).

Eigenspectrum decomposition further partitions the spectrum into contiguous bands (e.g., Dominant, Transition, Mid-range A/B, Lower, Tail), each contributing differently to cluster structure or artifacts. Whitening and clustering within spectral bands tests hypotheses about locality of semantic separability or measurement artifacts (Korun, 8 Mar 2026).

3. Cluster Commitment Metrics and Geometric Taxonomy

After whitening, a clustering (e.g., MiniBatchKMeans with $w_i = W (h_i - \mu)$1) is learned on the calibration set in the whitened space. The principal metric for geometric interpretability is “peak cluster alignment”:

• For each whitened embedding $w_i = W (h_i - \mu)$2, compute $w_i = W (h_i - \mu)$3, where $w_i = W (h_i - \mu)$4 is the $w_i = W (h_i - \mu)$5th cluster centroid.

In the hallucination taxonomy of (Korun, 8 Mar 2026), three failure types are defined by their alignment signatures in cluster space:

Type	Description	max_sim	Cluster Entropy $w_i = W (h_i - \mu)$6	Norm $w_i = W (h_i - \mu)$7
Center-drift (T1)	Weak context; drift to centroid	Intermediate	Low	Low
Wrong-well (T2)	Commit to contextually incorrect cluster	Highest	Initially high	—
Coverage-gap (T3)	Out-of-distribution; no cluster aligns	Lowest	High	—

Theoretical prediction and empirical results confirm that, after whitening, $w_i = W (h_i - \mu)$8 (Korun, 8 Mar 2026).

4. Large-Dimensional Regimes and Corrected Whitening

Classical whitening assumes an abundance of data in low-dimensional space ($w_i = W (h_i - \mu)$9), where empirical covariance is well-conditioned and cluster means become orthogonal after standard whitening. However, in the large-dimensional regime ($\mu$0 with $\mu$1), random matrix theory reveals that sample covariance estimates are spectrally distorted, inducing non-orthogonality among whitened means (Boudjemaa et al., 22 Sep 2025). Specifically, eigenvalue inflation and eigenvector shrinkage (“spiked” model) cause the dot products between whitened cluster means to converge to nonzero limits.

To address this, a correction based on random matrix theory is introduced:

• Estimate “spikes” $\mu$2 from empirical eigenvalues $\mu$3 and the aspect ratio $\mu$4.
• Construct diagonally scaled corrections to reweight the eigenvectors.
• The corrected whitening matrix

$\mu$5

restores asymptotic orthogonality among the cluster means:

$\mu$6

Empirical results demonstrate substantially lower mean-estimation error and consistent tensor recovery compared to uncorrected whitening above the phase transition $\mu$7 (Boudjemaa et al., 22 Sep 2025).

5. Statistical Protocols and Robustness

Cluster-based whitening must be coupled with rigorous validation to mitigate the risk of spurious signals arising from prompt selection, stochasticity, or feature-level pseudoreplication. Key elements of the protocol (Korun, 8 Mar 2026):

• Fix the whitening transform and cluster centroids using a large, representative calibration set.
• Repeat all downstream generation and metric aggregation over multiple ($\mu$8) independent random seeds.
• For each trial, aggregate metrics (e.g., max_sim) at the prompt level and conduct statistical testing (Mann–Whitney U, Holm–Bonferroni correction).
• Double the prompt pool size (e.g., $\mu$9 per class) to expose potential prompt-set artifacts.
• Analyze metric stability, significance rates, and directionality across seeds.

Prompt diversification eliminates cluster entropy artifacts in the dominant spectral band and confirms that only max_sim remains valid as a separator of certain hallucination types after whitening.

6. Applications and Methodological Implications

Cluster-based whitening yields several methodological and interpretive advantages:

• Hallucination type separation: Whitening reveals cluster commitment as the valid geometric separator between coverage-gap and wrong-well failures in LLM representations, with robust statistical support for T2 vs T3 and a directional T1 vs T2 trend.
• Spectral artifact localization: Eigenspectrum decomposition can localize spurious cluster separability (e.g., entropy artifacts) to specific principal component bands, rejecting the spectral mixing hypothesis for entropy signals (Korun, 8 Mar 2026).
• High-dimensional GMM estimation: Corrected whitening recovers asymptotic orthogonality in regimes where conventional whitening fails due to sample sufficiency limitations, improving the accuracy of spectral and tensor-based parameter estimation in GMMs (Boudjemaa et al., 22 Sep 2025).

7. Best-Practice Recommendations

Empirically supported guidelines for cluster-based whitening include (Korun, 8 Mar 2026):

1. Apply full-spectrum PCA-whitening with appropriate dimensionality reduction (e.g., $\mathrm{Cov}(w) \approx I_M$0 components, $\mathrm{Cov}(w) \approx I_M$1).
2. Perform clustering in the whitened space, using robust algorithms such as MiniBatchKMeans with $\mathrm{Cov}(w) \approx I_M$2.
3. Use peak centroid alignment (max_sim) as the primary diagnostic metric for geometric separation of latent types.
4. Ensure validation across both multiple random seeds and diversified prompt sets to avoid micro-signal pseudoreplication.
5. When investigating spectral locality, conduct whitening and clustering within defined variance bands.
6. Control for multiple comparisons (e.g., Holm–Bonferroni) within each metric family.

This protocol isolates cluster commitment as the theoretically grounded geometric feature for distinguishing failure types, allows accurate GMM parameter estimation in high-dimensional settings, and provides a systematic methodology for probing embedding space structure.

---

For in-depth derivations, empirical protocols, and recent extensions, see (Korun, 8 Mar 2026) for geometric measurement in deep contextual spaces and (Boudjemaa et al., 22 Sep 2025) for random-matrix-theoretic corrections in GMM estimation.