Zipfian PCA Whitening

• Zipfian PCA whitening is a technique that replaces uniform-design PCA whitening with frequency-weighted statistics, aligning embeddings with natural language frequency distributions.
• It centers and decorrelates embeddings using eigen-decomposition and SVD, ensuring isotropy under the empirically observed Zipfian measure.
• Empirical evaluations on models like GloVe and word2vec show notable improvements, with performance gains up to 14.7 points on semantic similarity benchmarks.

Zipfian PCA whitening is a post-processing technique for word embedding spaces that replaces standard uniform-statistics whitening with statistics weighted by empirical word frequencies, which follow Zipf's law. This adjustment yields substantial performance improvements in natural language processing tasks and reveals deep connections between the geometry of embedding spaces and the statistical structure of natural language (Yokoi et al., 2024).

1. Zipfian-Weighted Moments and Covariance Structure

Let $\mathcal V = \{w_1, \dots, w_V\}$ denote a vocabulary of size $V$, and for each type $w_i$ let $f_i = p(w_i)$ be its empirical frequency estimated from a corpus. Zipf's law implies that these frequencies are highly non-uniform (Zipfian), with a small number of very frequent words and a long tail of rare terms. The frequencies are usually normalized so that $\sum_{i=1}^V f_i = 1$.

Given an embedding $x_i \in \mathbb R^d$ for each $w_i$, the Zipfian-weighted mean and covariance are defined as: $$\bar x = \sum_{i=1}^V f_i x_i, \qquad C = \sum_{i=1}^V f_i (x_i - \bar x)(x_i - \bar x)^\top$$ This weighted covariance $C \in \mathbb R^{d \times d}$ captures the true statistical structure of the embedding space under the natural frequency distribution, emphasizing rare but informative words.

2. Zipfian PCA and Whitening Transformation

To decorrelate and normalize the embeddings relative to the Zipfian measure, one performs eigendecomposition on $C$: $V$0 The whitening matrix is: $V$1 Applying $V$2 to mean-centered embeddings $V$3, one obtains whitened vectors: $V$4 These satisfy the isotropy condition under the frequency-weighted measure: $V$5 In practice, this moves the embedding space to a position where each principal axis reflects true corpus statistics, reducing the skew introduced by uniform whitening.

3. Algorithmic Implementation

The process can be operationalized as follows:

1. Zipfian centering: Compute the weighted mean $V$6 and center embeddings $V$7.
2. Weighted data matrix: Construct $V$8 with rows $V$9.
3. SVD: Compute the singular value decomposition $w_i$0, where $w_i$1.
4. Whitening: For each $w_i$2, transform $w_i$3 to $w_i$4.

This produces embeddings whitened under the empirical frequency distribution (Yokoi et al., 2024).

4. Information-Theoretic and Probabilistic Foundations

Zipfian PCA whitening is theoretically grounded in the geometry of exponential-family models. Word embeddings define a log-linear probability distribution: $w_i$5 Here, $w_i$6 is a base measure. Using a uniform prior ($w_i$7) recovers uniform whitening. With a Zipfian prior ($w_i$8), as found in skip-gram negative sampling (SGNS), whitening aligns with the true generative model of language.

The Fisher information metric in this model is weighted by $w_i$9, leading to rare, information-rich words acquiring larger vector norms post-whitening. The norm quantifies information gain: $f_i = p(w_i)$0 This geometric structure naturally re-scales the embedding space to prioritize informative, infrequent words (Yokoi et al., 2024).

5. Connections to Existing Embedding and Whitening Methods

Zipfian whitening elucidates and unifies several prominent NLP methodologies:

• SGNS (Skip-gram Negative Sampling): At optimum, its probabilistic model reflects the Zipfian prior, implicitly weighting the softmax by $f_i = p(w_i)$1.
• WhiteningBERT and batch-centering: These methods whiten token embeddings sampled according to frequency, effectively mirroring Zipfian PCA whitening.
• Headless LLMs: Small batch sampling reweights softmax layers by empirical frequency, also embedding a Zipfian base measure.

This perspective clarifies why such methods empirically outperform alternatives that ignore Zipfian statistics (Yokoi et al., 2024).

6. Empirical Results and Practical Impact

Comprehensive evaluations of Zipfian PCA whitening on GloVe, word2vec, fastText, and standard semantic similarity benchmarks (STS12–STS16, STS-B, SICK-R, JSTS) have been conducted using 300-dimensional embeddings. Frequencies $f_i = p(w_i)$2 are obtained from large corpora such as the English Wikipedia or task-specific test sets. Key findings include:

Embedding	Uniform Whitening	Zipfian Whitening	Delta
GloVe (STS-B)	52.2	66.9	+14.7
word2vec	56.0	66.5	+10.5
ABTT	54.3	—	—
SIF+CCR	58.7	—	—

Improvements exceed strong baselines like ABTT and SIF+CCR. These gains generalize across embeddings, tasks, and even languages (e.g., Japanese). Moreover, intrinsic Zipfian symmetry scores (1st/2nd moments) display strong correlation ($f_i = p(w_i)$3) with downstream performance, whereas uniform and average-cosine measures do not (Yokoi et al., 2024).

7. Summary and Significance

Zipfian PCA whitening replaces uniform with frequency-weighted moments and covariance in PCA whitening, aligning embedding post-processing with the statistical laws governing natural language. This minor yet principled modification emphasizes the semantic contribution of rare words and yields marked, reproducible performance improvements across diverse embedding architectures and language tasks (Yokoi et al., 2024). The method provides a unified theoretical foundation for a range of successful NLP algorithms and validates the importance of incorporating the true generative model at the level of representation geometry.