The Pitfall of Scaling Up: Uncovering and Mitigating Popularity Bias Amplification in Scaling Transformer-based Recommenders

Abstract: We identify a critical pitfall in scaling transformer-based sequential recommenders: while increasing model size improves recommendation accuracy, it simultaneously amplifies popularity bias. This bias drives systems to over-recommend popular items at the expense of niche ones, which not only undermines fairness but also degrades the broader ecosystem by reinforcing the Matthew effect and filter bubbles. Consequently, this bias amplification emerges as a fundamental obstacle to sustainable model scaling. Through comprehensive theoretical and empirical analyses, we uncover the root cause of this amplification. Our findings reveal that as model depth increases, the two core components of the transformer architecture, i.e., attention aggregation and feed-forward projections, synergistically induce severe spectral collapse in model predictions, which directly translates to the amplification of popularity bias. To address this challenge, we propose SPRINT (Scalable Popularity Regularization IN Transformers), which mitigates spectral collapse during scaling by constraining (i) the maximum column-sums of the attention score matrices and (ii) the spectral norms of the feed-forward parameters. Extensive experiments demonstrate that SPRINT significantly improves both accuracy and long-tail fairness. Crucially, it yields more favorable scaling behaviors when expanding model sizes from 0.05M to 0.34B parameters. The code is available at https://github.com/Tiny-Snow/GenRec.

• The paper identifies spectral collapse in deep transformer recommenders as a primary driver of popularity bias amplification.
• It demonstrates that increased model depth boosts accuracy initially but degrades fairness by favoring head items over long-tail items.
• The proposed SPRINT regularization method effectively constrains attention and feed-forward spectral norms, achieving improved accuracy and fairness.

Uncovering and Mitigating Popularity Bias Amplification in Scaled Transformer-based Sequential Recommenders

Introduction

This paper provides an in-depth analysis of the scaling behavior of transformer-based sequential recommenders, focusing on the emergent and compounding issue of popularity bias amplification. The analysis addresses the empirical observation that while increasing model capacity—especially through deeper transformer stacks—initially enhances accuracy according to prevailing scaling laws, this process also systematically intensifies the tendency to over-recommend popular items at the expense of tail items. Through rigorous theoretical and empirical investigation, the paper identifies spectral collapse—driven by stacked attention and feed-forward projections—as the core architectural mechanism underlying this bias amplification. To counteract these effects, the proposed SPRINT regularization method concurrently bounds the concentration of attention weights and the spectral norms of feed-forward layers, yielding marked improvements in both accuracy and long-tail fairness across diverse sequential recommendation benchmarks.

Figure 1: Scaling laws of accuracy (NDCG@5) and fairness (Fair-0.95) on transformer-based sequential recommenders for various model sizes and transformer depths. SPRINT enables sustained gains in both metrics.

Empirical Analysis of Scaling-Induced Popularity Bias

The experimental results reveal a consistent pattern: modest increases in model size or depth yield accuracy and long-tail fairness improvements; however, scaling to larger sizes, especially via deeper models, leads to a sharp degradation in fairness metrics due to the amplified recommendation of head items. This fairness-accuracy tradeoff is intrinsic to standard transformers, with fairness (as measured by long-tail item exposure) deteriorating well before any accuracy saturation on large-scale datasets such as MovieLens-20M.

The study further demonstrates that in deep models, the power spectrum of the item prediction matrix undergoes collapse—meaning the principal singular value dominates, aligning with the popularity vector. This spectral collapse is empirically coupled with reduced long-tail exposure and higher Gini indices, substantiating the causal link between model scaling, spectrum concentration, and popularity bias.

Figure 2: Increased spectral collapse and worsened fairness as transformer depth increases. The largest singular value dominates, amplifying popularity bias.

Figure 3: Attention distributions: popular items receive over 75% of the attention weights, driving bias in aggregation.

Spectral Collapse as a Mechanism for Bias Amplification

The analysis roots the bias amplification in two architectural phenomena:

1. Popularity-prioritized attention aggregation: Empirically, attention matrices concentrate most of their score mass on popular items. When attention patterns are aligned across layers, their repeated application exponentially increases the spectral norm of the score aggregation, as formalized in the presented lower bound theorem.
2. Low-rank feed-forward projections: Deep cascades of feed-forward networks, especially under gradient-based optimization, rapidly increase the dominant singular value relative to others, pushing the representation geometry toward degeneration.

Both attention and feed-forward modules, therefore, synergistically exacerbate the spectral collapse of the score matrix as depth increases. Ablation studies confirm that omitting either component significantly alleviates this collapse, and thus, regularization must target both pathways.

Figure 4: Ablations: isolating attention or feed-forward layers confirms both drive spectral collapse, necessitating joint regularization.

The SPRINT Regularization Framework

SPRINT introduces two coordinated regularization terms:

• Attention regularization: This term minimizes the surrogate upper bound on the attention matrix spectral norm, instantiated through column-sum constraints and implemented efficiently via log-sum-exp relaxation over column sums.
• Feed-forward spectral regularization: Spectral norms of feed-forward projection matrices in each layer are directly controlled via power iteration estimates, bounding their product across the stack.

With these, SPRINT provably upper-bounds the prediction score matrix's spectral norm, thus constraining the popularity-aligned principal component. This delivers sustained accuracy and fairness improvements as model size and depth increase.

Empirical Results

SPRINT consistently outperforms a diverse spectrum of debiasing baselines spanning spectral regularization, propensity reweighting, DRO, and disentanglement methods, across all tested datasets and two strong transformer-based sequential backbones (SASRec++ and HSTU). Notably, SPRINT yields:

• +15.70% improvement in NDCG/HR and +7.12% in fairness on average.
• Strict Pareto-dominance in accuracy-fairness across all model scales, with degradation avoided even for the largest models.
• Negligible runtime overhead (less than 3%) due to computationally efficient regularizer designs.

Layer-scaling experiments show that alternative methods plateau or regress in fairness at high depth, while SPRINT maintains monotonic improvements in both axes.

Figure 5: Layer scaling—SPRINT’s gains persist as transformer depth grows, addressing the fundamental scaling bottleneck.

Figure 6: Accuracy-fairness Pareto frontier—SPRINT strictly dominates baselines, enabling better tradeoffs.

Figure 7: SPRINT regularization introduces minimal training cost, supporting practical deployment.

Ablations confirm that both attention and feed-forward regularization are necessary for full mitigation. Hyperparameter analyses show that SPRINT is robust, and its regularization weights can be readily tuned for optimal tradeoffs.

Figure 8: SPRINT is robust to attention regularization weight $\lambda_{\text{attn}}$; the tradeoff curve has a clear optimum.

Implications and Theoretical Guarantees

The findings have several implications and yield formal guarantees:

• Unconstrained model scaling is unsustainable for fairness-sensitive applications, as deeper stacks will almost inevitably focus on head items, exacerbating systemic filter bubbles and the Matthew effect.
• Layer-wise, targeted spectral regularization is essential. Global spectral penalties or factorization-based regularizers from collaborative filtering settings are ineffective, unstable, or inapplicable under the dynamic, autoregressive user dynamics of sequential recommenders.
• SPRINT’s theoretical bound guarantees that the overall spectral norm, and thus bias, is globally controlled, even in the presence of layer norm, multi-head attention, residual connections, and non-linearities (details in appendix).

Broader Impacts and Future Directions

On the theoretical side, this work generalizes prior spectral analyses of bias [lin2025recommendation] to the setting of deep, dynamic user modeling, introducing layer-wise regularization as a principled mechanism for controlling emergent pathologies. Practically, SPRINT’s negligible training overhead and compatibility with large architectures open its deployment for web-scale recommenders.

Figure 9: SPRINT ensures stable accuracy and fairness across diverse datasets as layers scale.

Possible extensions include integrating SPRINT with efficient attention architectures (e.g., FlashAttention), and analyzing the joint impacts of other scaling dimensions, such as width, attention heads, and training data regimes, on fairness. More generally, the method’s spectral analytic framework invites application to other domains suffering from scaling-induced collapse.

Conclusion

Scaling up transformer-based sequential recommenders introduces a critical obstacle: popularity bias amplification linked to spectral collapse in model outputs, fundamentally driven by both attention and feed-forward architectural design. The SPRINT method provides a theoretically grounded, architecture-aware regularization strategy that simultaneously enables the accuracy and fairness benefits of scale. Extensive experimental evidence confirms that SPRINT unlocks sustainable scaling, achieving improved accuracy while maintaining balanced recommendation distributions, with trivial computational cost. This establishes new best practices and analytic tools for large-scale recommender development.

---

References

• "The Pitfall of Scaling Up: Uncovering and Mitigating Popularity Bias Amplification in Scaling Transformer-based Recommenders" (2606.21911)
• Lin, Siyi et al., "How do recommendation models amplify popularity bias? An analysis from the spectral perspective" [lin2025recommendation]
• Zhang, Gaowei et al., "Scaling law of large sequential recommendation models", RecSys 2024
• Zivic, Pablo et al., "Scaling sequential recommendation models with transformers", SIGIR 2024

(See main text for additional references and appendix for theoretical derivations.)