Zipfian Whitening in Word Embeddings

• Zipfian Whitening is a frequency-aware transformation that adjusts traditional PCA whitening by weighting word embeddings based on empirical Zipfian distributions.
• It modifies centering and covariance estimation by incorporating word frequency, thereby correcting geometric asymmetries in embedding spaces.
• Empirical results demonstrate that this method boosts semantic task performance, outperforming traditional whitening techniques on benchmarks like STS.

Searching arXiv for the specified paper and closely related work on Zipfian whitening, whitening in embeddings, and Zipfian distributions. Zipfian whitening is a frequency-aware whitening procedure for word embeddings that replaces the uniform weighting implicit in standard PCA whitening with empirical word frequencies that follow Zipf’s law. In the formulation introduced in "Zipfian Whitening" (Yokoi et al., 2024), the method addresses the observation that embedding spaces in neural models are skewed, while most existing approaches for modeling, correcting, and measuring their symmetry treat words as if they were uniformly distributed. Zipfian whitening therefore redefines centering and covariance normalization so that they respect the highly non-uniform distribution of word types in natural language, with the aim of improving geometric symmetry and downstream task performance.

1. Conceptual basis

Zipf’s law states that the frequency of a word is inversely proportional to its rank in the frequency table, $p(w_{(r)}) \propto \frac{1}{r}$. In natural language, this implies that a small number of words are extremely frequent, while most words are rare. Zipfian whitening starts from the claim that this asymmetry is not a nuisance external to embedding geometry but part of the structure that post-processing should model explicitly (Yokoi et al., 2024).

Traditional PCA whitening, as described for word embedding postprocessing, centers the embedding matrix and transforms the vectors so that their covariance matrix is the identity, thereby decorrelating dimensions and scaling them to unit variance. In that standard form, each word vector is equally weighted. Zipfian whitening changes this assumption at every stage where an expectation is required. The mean, covariance, and related statistics are computed with weights given by empirical word frequency rather than with a uniform measure. The method is therefore a correction to the type-level view that treats all vocabulary entries as equally important, even though token usage in corpora is strongly Zipfian (Yokoi et al., 2024).

This distinction is significant because the embedding table contains word types, whereas language data are generated as word tokens with sharply unequal frequencies. The method’s central claim is that measuring isotropy or enforcing symmetry without frequency weighting can mischaracterize the geometry of the space and can over-represent rare words in the estimation of global statistics (Yokoi et al., 2024).

2. Mathematical construction

Let the vocabulary be $\mathcal{V} = \{w_1,\dots,w_n\}$, let each word embedding be $\mathbf{w}_i \in \mathbb{R}^d$, and let $p(w_i)$ denote the empirical frequency of word $w_i$. Zipfian whitening first computes the frequency-weighted mean

$\hat{\mu} = \sum_i p(w_i)\,\mathbf{w}_i .$

Each word vector is then centered as

$$\overline{\mathbf{w}}_i = \mathbf{w}_i - \hat{\mu} .$$

The weighted covariance structure is formed through the matrix

$$\mathbf{W}_p = \begin{bmatrix} \sqrt{p(w_1)}\,\overline{\mathbf{w}}_1^\top \ \vdots \ \sqrt{p(w_n)}\,\overline{\mathbf{w}}_n^\top \end{bmatrix},$$

followed by an SVD

$$\mathbf{W}_p = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^\top .$$

The whitened representation of each word is then

$$\widetilde{\mathbf{w}}_i = \overline{\mathbf{w}}_i\,\mathbf{V}\mathbf{\Sigma}^{-1}.$$

This yields a representation whose weighted mean is zero and whose weighted covariance is the identity: $\mathcal{V} = \{w_1,\dots,w_n\}$0 The construction is thus formally parallel to ordinary whitening, but the underlying expectation is taken with respect to empirical word frequency rather than a uniform distribution over vocabulary entries (Yokoi et al., 2024).

The contrast with uniform whitening can be summarized succinctly.

Aspect	Uniform whitening	Zipfian whitening
Centroid	Arithmetic mean	Frequency-weighted mean
Covariance	Uniform-weighted	Frequency-weighted
Assumption	All words equally used	Words used per Zipf’s law

The practical consequence of this reformulation is that the transformed space is normalized with respect to actual language use rather than with respect to a hypothetical balanced vocabulary.

3. Probabilistic and information-geometric interpretation

A central theoretical claim of Zipfian whitening is that embedding methods can be organized through an exponential-family view. In that formulation,

$\mathcal{V} = \{w_1,\dots,w_n\}$1

where $\mathcal{V} = \{w_1,\dots,w_n\}$2 is a base measure over words. Existing whitening-based views are associated with a uniform base measure, $\mathcal{V} = \{w_1,\dots,w_n\}$3, whereas Zipfian whitening adopts the empirical measure, $\mathcal{V} = \{w_1,\dots,w_n\}$4. The paper states that both new and existing approaches can therefore be categorized as using exponential-family models with either uniform or Zipfian base measures (Yokoi et al., 2024).

Within this perspective, whitening is connected to making the partition function $\mathcal{V} = \{w_1,\dots,w_n\}$5 nearly constant. The difference is not merely notational. Under a Zipfian base measure, the geometry of the space emphasizes low-frequency words rather than high-frequency ones. From the information-geometric perspective, the norm of a word vector is tied to information content through

$\mathcal{V} = \{w_1,\dots,w_n\}$6

The interpretation given in the paper is that rare, content-bearing words become more informative in the geometry of the space, while uniform whitening can assign greater norm to frequent, lower-information words (Yokoi et al., 2024).

The same section links the method to imbalanced classification. The claim is that the geometric bias induced by Zipfian weighting parallels the effect of frequency-aware loss functions for imbalanced classes, including logit-adjusted cross-entropy. In this view, emphasizing rare words in the normalization step is not an arbitrary heuristic but part of a broader correspondence between embedding geometry, token-frequency priors, and losses designed for long-tail data (Yokoi et al., 2024).

A plausible implication is that this line of reasoning is consonant with broader information-theoretic results on Zipfian sources. Work on universal compression of power-law distributions shows that Zipf structure can substantially reduce expected redundancy relative to unrestricted distributions, and that the number of distinct observed elements becomes the key statistic controlling redundancy (Falahatgar et al., 2015). That result concerns compression rather than embedding postprocessing, but it supports the general proposition that Zipfian structure alters what constitutes an efficient representation.

4. Empirical behavior

The reported empirical result is that simply performing PCA whitening weighted by empirical word frequency significantly improves task performance and surpasses established baselines. The paper further states that Zipfian whitening strongly outperforms traditional whitening and specialized sentence embedding methods such as SIF and ABTT on STS and other benchmarks (Yokoi et al., 2024).

One concrete comparison is given for STS-B:

Model	Uniform Whitening	Zipfian Whitening
GloVe	52.21	66.92
word2vec	56.03	66.50

The same study reports that the correlation between intrinsic symmetry, when measured with Zipfian weighting, and downstream task performance is extremely high, with an example Spearman correlation of $\mathcal{V} = \{w_1,\dots,w_n\}$7, whereas unweighted metrics show almost no correlation (Yokoi et al., 2024).

The explanation supplied for this empirical behavior is that frequency-aware centering removes the influence of very frequent words and standardizes the space against the actual distribution of word use. Under this interpretation, Zipfian whitening gives rare and informative words greater geometric prominence, which is desirable for semantic tasks in which function words would otherwise dominate global statistics (Yokoi et al., 2024).

These claims are specific to the word-embedding setting studied in the paper. They do not amount to a universal statement that any whitening procedure improves any embedding space.

5. Relation to other methods and to adjacent Zipfian phenomena

The paper explicitly relates Zipfian whitening to three established NLP lines: skip-gram negative sampling, WhiteningBERT, and headless LLMs. For skip-gram negative sampling, the claim is that the geometry learned by the model already reflects empirical word frequency through the negative-sampling distribution, and that Zipfian whitening aligns the embedding space with that underlying probabilistic structure. For WhiteningBERT, the argument is that whitening over all tokens rather than over types is equivalent to Zipfian whitening over types, because token counts are proportional to word frequency. For headless LLMs, the paper argues that the effective base measure in the softmax tracks empirical usage statistics over sampled data. The broader conclusion is that these methods work well because their embeddings encode empirical word frequency into the underlying probabilistic model (Yokoi et al., 2024).

A separate but related literature concerns the anatomy of rank-frequency distributions themselves. Work on text mixing argues that the widely observed two-regime structure in large corpora arises from aggregating texts, which induces a decay in the word-introduction rate and shifts the location and severity of scaling breaks. That discussion notes a possible connection to “Zipfian Whitening” in the looser sense that aggregation can smooth or “whiten” local rank-frequency structure, although the connection is not elaborated in depth and does not define the embedding method introduced in 2024 (Williams et al., 2014).

There is also a cross-domain connection to whitening in neural populations. A study of adaptive whitening in sensory systems cites Zipfian or power-law input statistics as common in natural signals and argues that a non-negative gain constraint implements a soft whitening adapted to heavy-tailed, ill-conditioned covariance spectra. This is a distinct framework, but it suggests that whitening procedures tailored to Zipfian statistics are not unique to NLP and can arise whenever the underlying signal distribution is strongly non-uniform (Duong et al., 2023).

6. Misconceptions, limitations, and scope

A common misconception is that whitening is beneficial whenever it increases isotropy. This is not supported uniformly across embedding settings. In sentence embeddings derived from LLMs, experiments on PCA, ZCA, PCA-Cor, ZCA-Cor, and Cholesky whitening show that whitening degenerates embeddings for classification tasks, and the authors conclude that whitening is not recommended for classification. The same work reports that whitening can sometimes help semantic textual similarity for untuned models, but that it harms classification and clustering across the evaluated models and datasets (Forooghi et al., 2024).

This matters for understanding Zipfian whitening. The 2024 proposal is not a blanket endorsement of isotropy as an objective in itself. Rather, its claim is narrower: if whitening is used for word embeddings, the relevant geometry should be defined with respect to empirical frequency rather than a uniform vocabulary measure. The sentence-embedding results indicate that greater isotropy does not by itself guarantee better task accuracy and that post-processing can erase useful anisotropy introduced by fine-tuning (Forooghi et al., 2024).

The principal scope condition is therefore methodological. Zipfian whitening is formulated for word embeddings and for the symmetry of spaces whose statistics are governed by Zipf’s law. It is best understood as a correction to the assumptions behind ordinary whitening, not as a universal prescription for all embedding models or all downstream tasks. Within that scope, the method provides a frequency-aware account of symmetry, a probabilistic interpretation via exponential families with Zipfian base measures, and an empirical argument that respecting token-frequency imbalance can improve semantic representations (Yokoi et al., 2024).