Popularity Bias Memorization Theorem

• Popularity Bias Memorization Theorem is a mathematical framework that explains how recommendation models absorb, retain, and amplify item popularity through spectral analysis.
• It establishes spectral bounds that align the popularity vector with dominant singular directions, clarifying how collaborative filtering embeds bias in predictions.
• The theory informs debiasing strategies and spectral regularization techniques, highlighting trade-offs between fairness and utility in recommendation systems.

The Popularity Bias Memorization Theorem provides a rigorous mathematical foundation for understanding how collaborative filtering and embedding-based recommendation systems absorb, retain, and amplify item popularity distributions present in user-item interaction data. The theorem and its generalizations establish precise spectral bounds governing the alignment between the item-popularity vector and the top singular directions (or subspace) of the predicted score matrix, elucidating the mechanism by which popularity bias is both inherited and exacerbated by learned models. These results are foundational for both the theoretical paper and algorithmic mitigation of popularity bias in recommendation.

1. Spectral Formulation of Popularity Bias

Let $\mathcal U$ and $\mathcal I$ denote the sets of users and items ($n = |\mathcal U|$, $m = |\mathcal I|$). The binary interaction matrix $Y \in \{0,1\}^{n \times m}$ encodes observed interactions, with $y_{ui} = 1$ indicating that user $u$ has interacted with item $i$. The popularity vector $\vec r = (r_1, \dots, r_m)^\top$ is given by $r_i = \sum_u y_{ui}$, capturing the empirical degree of each item.

A recommendation model with sufficient embedding capacity produces a score matrix $\widehat Y$ (typically via $$\widehat Y_{ui} = \mu(\mathbf u_u^\top \mathbf v_i)$$), whose spectral decomposition is $\widehat Y = P\Sigma Q^\top$ with singular values $\sigma_1 \ge \sigma_2 \ge \cdots$ and right singular vectors $q_1, \ldots, q_m$. Crucially, the primary mechanism of bias arises from the alignment between $\vec r$ and $q_1$.

2. The Popularity Bias Memorization Theorem

Original Version under Power Law Assumptions

Under the hypothesis that sorted item popularities follow a power law, $r_g \propto g^{-\alpha}$, and for embedding models trained on such $Y$, Lin et al. (Lin et al., 18 Apr 2024) give the following lower bound: $$\cos(\vec r, q_1) \ge \frac{\sigma_1^2}{r_{\max} \sqrt{\zeta(2\alpha)}} \sqrt{1-\frac{r_{\max}(\zeta(\alpha)-1)}{\sigma_1^2}}$$ where $r_{\max}$ is the popularity of the most popular item and $\zeta(s)$ the Riemann zeta function. When $\alpha > 2$ ($\zeta(\alpha) \leq 2$), a simpler bound holds: $$\cos(\vec r, q_1) \ge \sqrt{\frac{2-\zeta(\alpha)}{\zeta(2\alpha)}}$$ These results formalize that, under strong tail-heaviness and sufficiently large spectral gap, the score matrix's dominant direction is nearly collinear with empirical popularity.

Generalized Version for Arbitrary Degree Distributions

Lyubinin (Lyubinin, 25 Nov 2025) extends the theorem to arbitrary (not necessarily power-law) item degree sequences: $$\cos(\vec r, q_1) \ge \frac{\sigma_1^2}{\sqrt{\mathrm{vol}_2(\mathcal I)} \sqrt{1 - \frac{1}{\sigma_1^2(\mathrm{vol}(\mathcal I) - r_{\max})}}}$$ with $\mathrm{vol}(\mathcal I)=\sum_i r_i$ and $\mathrm{vol}_2(\mathcal I)=\sum_i r_i^2$. No parametric tail assumptions are needed; the bound is controlled solely by spectral and degree statistics.

3. Top-$k$ Singular Hyperspace and Alignment Metrics

The alignment between $\vec r$ and the rank-$k$ dominant singular directions, $Q_k = [q_1, \ldots, q_k]$, is quantified via the projector $\Pi_k = Q_k Q_k^\top$: $$\cos\theta_k = \frac{\|\Pi_k \vec r\|_2}{\|\vec r\|_2} = \sqrt{\sum_{i=1}^k \cos^2(\vec r, q_i)} = \sqrt{\kappa_k}$$ where $\kappa_k$ is the fraction of the squared norm of $\vec r$ explained by the top-$k$ singular directions.

Explicit lower and upper bounds for this alignment fall under three categories:

• Combinatorial: Using item subsets $S\subseteq \mathcal I$, tail mass $\Delta_S$, and index-matched sums to lower-bound $\cos\theta_k$.
• Spectral/Ky Fan: Leveraging eigenvalues of Gram matrices ($B_S^\top B_S$ for complements $S^c$).
• Linear-Programming: Expressing $\kappa_k$ in terms of mixture weights $\alpha_i$, relating alignment to mean squared degree $\mu$ and the spectrum $\{\sigma_i^2\}$, with closed-form LP solutions.

Summary of core bounds:

Type	Scope	Representative Bound
Power Law	$k=1$	$\cos(\vec r, q_1)$ lower bounded by zeta functions and largest singular value/power law exponent
Arbitrary Deg.	$k=1$	$\cos(\vec r, q_1) \ge$ spectral/volume expression, no parametric assumption
General $k$	$k \ge 1$, all	Combinatorial (A1/A2), Ky Fan (B2/B3), LP bounds in terms of $\sigma_k$, tail mass, and $\|\vec r_S\|$

4. Mechanisms: Dimension Collapse and Amplification

Two mechanisms underlie the popularity bias amplification:

• Spectral Alignment: Training models with sufficient capacity results in predicted $\widehat Y$ whose principal singular vector ($q_1$) aligns to the empirical popularity distribution, as this direction captures maximal variance in long-tailed $Y$.
• Dimension Collapse: As the top singular value $\sigma_1$ dominates the spectrum, the model's predictions become almost rank-one, and the resulting prediction matrix is governed almost entirely by popularity. The fraction $\eta$ of users for whom the most popular item is ranked highest satisfies

$$\eta \ge \frac{1}{n} \phi\left(\frac{\sqrt{2\zeta(2\alpha)}}{1 - 2^{-\alpha}(\sum_k \sigma_k / \sigma_1 - 1)}\right)$$

where $\phi(a)$ counts users with large enough components in the first left singular vector. Thus, dimension collapse intensifies the dominance of popular items in recommendation lists.

5. Generalization to Arbitrary Degree Distributions

Lyubinin's generalization (Lyubinin, 25 Nov 2025) dispenses with power-law assumptions and demonstrates that the lower bound on principal direction alignment applies for any degree sequence, provided $\sigma_1^2 \ge r_{\max}$. The theory extends to arbitrary splits of the item set for top-$k$ subspace alignment, using combinatorial parameters (tail degree mass, block minor eigenvalues), Ky Fan inequalities, and spectral LP relaxations. This broad applicability unifies the analysis of memorization phenomena across a range of potential degree distributions—log-normal, truncated power law, or others—commonly encountered in recommendation data.

6. Algorithmic Implications and Debiasing

The theorem has direct implications for the design and evaluation of recommendation models:

• Explanation of Fairness Failures: Models embedding long-tailed data, especially with high $\sigma_1/\sum_k \sigma_k$, inevitably over-represent head items and suppress tail item visibility, degrading fairness and novelty even as recall/NDCG metrics remain high.
• Spectral Regularization: Lin et al. propose a spectral-norm regularizer (ReSN), adding $\beta \|\widehat Y\|_2^2$ to the loss to "flatten" the dominant singular direction and mitigate memorization of popularity:

$$\widetilde L_{\rm ReSN} = L_R(Y, \widehat Y) + \beta \|\mathbf U [\mathbf V^\top \vec r]\|^2$$

This approach is computationally efficient ($O((n+m)d)$ per epoch) and model-agnostic. However, since popularity may correlate with genuine item quality, over-regularization can reduce accuracy; the trade-off parameter $\beta$ must balance debiasing versus utility.

7. Broader Significance and Extensions

The spectral theory underlying the Popularity Bias Memorization Theorem elucidates why collaborative filtering models universally "memorize" and amplify popularity, and how top-$k$ singular space alignment increases with concentration and spectral gap in item degrees. Generalized bounds allow modelers to predict alignment strength from empirical spectra and to select regularization strategies accordingly.

Future work is suggested in extending this framework to multi-step feedback loops (Matthew Effects) and to algorithmic classes beyond inner-product embeddings, such as graph-based models and higher-order interactions (Lin et al., 18 Apr 2024). The key limitation is that if popularity serves as a valid proxy for utility, aggressive regularization may be undesirable; nuanced adaptation to context is needed.

• Lin, S. et al., "How Do Recommendation Models Amplify Popularity Bias? An Analysis from the Spectral Perspective" (Lin et al., 18 Apr 2024).
• Lyubinin, A., "Popularity Bias Alignment Estimates" (Lyubinin, 25 Nov 2025).