Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation

Abstract: Scaling recommendation models is a central challenge in recommender systems. Recently, RankMixer has emerged as an effective solution, operating on a unified token representation and alternating between token mixing and per-token feedforward networks (P-FFNs) to achieve scalable performance. However, RankMixer suffers from \textit{embedding collapse}, where learned representations have low effective rank, limiting expressivity and underutilizing the expanded representation space. Through empirical analysis and theoretical insights, we identify rigid token mixing and P-FFN modules as the primary causes of this phenomenon, jointly inducing a \textbf{damped oscillatory trajectory} in effective-rank evolution across layers. To address it, we propose RankElastor, a novel architecture that produces spectrum-robust representations with provable collapse mitigation. RankElastor introduces two components: (i) \textbf{parameterized full mixing}, which enables expressive token mixing with improved spectral robustness; and (ii) \textbf{GLU-improved P-FFNs}, which stabilize representation spectra through GLU-style FFN modules. Extensive experiments on large-scale industrial datasets demonstrate that RankElastor consistently improves recommendation performance, mitigates embedding collapse, and exhibits robust scaling behavior. Code is available at this GitHub repository: https://github.com/vasile-paskardlgm/RankElastor

• The paper introduces RankElastor, which uses full mixing and GLU-FFN modules to mitigate embedding collapse in deep recommendation systems.
• It empirically demonstrates improved AUC and lower LogLoss on CTR benchmarks by maintaining robust effective-rank dynamics across layers.
• The study’s theoretical analysis links spectrum-preserving nonlinearities to dense scaling, challenging traditional token-mixing methods.

Effective-Rank Dynamics and Dense Scaling in Deep Recommendation: An Analysis of RankElastor

Introduction

The paper "Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation" (2605.23191) addresses the critical challenge of representation collapse in large-scale deep recommender systems. Focusing on token-transformation-based architectures, notably RankMixer, the study dissects the empirical and theoretical limitations in spectral capacity that manifest during model depth scaling and proposes an improved architecture, RankElastor, which robustly mitigates dimensional collapse while inducing superior scaling behavior.

Theoretical Analysis of Embedding Collapse

The study formalizes embedding collapse in the context of multi-field recommender systems using the notion of effective rank—a spectral measure quantifying the diversity and usable capacity of learned representations. Empirical analysis on industrial CTR benchmarks shows that RankMixer exhibits a damped oscillatory trajectory in effective-rank evolution: token mixing introduces a slight increase, but this expansion is consistently overpowered by significant contraction in subsequent P-FFN layers.

Figure 1: Effective rank comparison for RankMixer and baselines on the Criteo dataset, showcasing the oscillatory trajectory versus monotonic decay.

The theoretical examination identifies two causes for this bounded expressivity and incomplete collapse mitigation:

1. Token Mixing: The block-transpose operator in RankMixer introduces only a bounded increase in effective rank, as formalized using operator analytic bounds.
2. P-FFN Contraction: Standard per-token FFNs (with GELU activations) contract the spectrum, in some cases deterministically collapsing rank to two directions for rank-one input.

These findings establish that, despite a slight improvement over classical feature interaction models such as DCNv2 and xDeepFM, RankMixer's spectral limitations fundamentally block the model from attaining robust, high-dimensional representations as depth increases.

RankElastor Architecture

To address these limitations, the paper introduces the RankElastor architecture, composed of two key innovations:

• Parameterized Full Mixing: Generalizes the block-permutation in token mixing to a fully parameterized dense linear operator over token-feature coordinates, significantly expanding the reach of spectral transformations expressible by the model.
• GLU-Improved P-FFNs: Replaces vanilla FFN blocks with GLU-style modules. The gated activation structure enables effective rank recovery, ensuring multiplicative quadratic interactions statistically inject higher-rank directions, and damps rank collapse over depth.

  Figure 2: Structural comparison of RankMixer (a) and RankElastor (b), with color coding for parametric (purple) and fixed (gray) components. Major differences lie in token mixing and P-FFN modules.

These architectural adjustments are theoretically justified by new analysis demonstrating that (i) the fine-grained parameterized mixer strictly subsumes all block-mixing operators in representational capacity, and (ii) for sufficient hidden width, the GLU-improved FFN ensures a strictly positive increase in effective rank layer-wise with high probability.

Empirical Results

Performance Gains

Extensive experimentation on Criteo and Avazu validates the claims. RankElastor consistently outperforms all strong baselines, including RankMixer, on both AUC and LogLoss metrics, with consistent AUC improvements on the order of 0.001—an important margin in large-scale industrial recommendation.

Figure 3: CTR prediction results on Criteo. RankElastor achieves higher AUC and lower LogLoss compared to RankMixer and other baselines.

Spectral Dynamics

Layerwise effective-rank analysis shows RankElastor does not merely moderate collapse but maintains strictly increasing or stably high spectral diversity throughout depth, in stark contrast to the rapid spectral contraction of both classical and RankMixer architectures.

Figure 4: Layerwise shift in effective rank induced by RankElastor’s token mixing layer, illustrating robust spectrum expansion.

Scaling Laws

Under both width and depth scaling regimes, RankElastor demonstrates robust monotonic improvements, with empirical scaling curves exhibiting notably better slopes and higher asymptotes compared to RankMixer.

Figure 5: Width-wise dense parameter scaling on Criteo. RankElastor exhibits superior scaling behavior against RankMixer.

Figure 6: Joint width and depth dense parameter scaling on Criteo, confirming multi-parameter scaling advantages in RankElastor.

Ablation and Efficiency

Ablation studies confirm that both architectural modifications (Full Mixing and GLU-FFN) contribute synergistically to the observed gains. Despite higher computational complexity in the mixing layer ($O(T^2D^2)$), overall GPU and wall-clock cost is only modestly increased (10%–15%), with no penalty in parameter efficiency relative to overall model size.

Practical and Theoretical Implications

The findings contribute several advances to the literature on deep CTR and recommendation modeling:

• By offering a mathematically principled and empirically validated method for spectrum-preserving scaling, the architecture supports larger and deeper recommenders without spectral degeneration.
• The result calls into question the sufficiency of non-parametric mixing techniques for robust scaling, suggesting a shift towards fully parametric and spectrum-aware design.
• The formalization of GLU as a spectrum-stabilizing operator connects feature interaction literature with recent advances in gated architectures from LLMs and Transformers.

These insights have broader implications for high-capacity models in ranking and retrieval, MTL, and encoding tasks where expressive, non-collapsing intermediate representations are critical.

Future Directions

The oscillatory but robust rank behavior induced by stacked parametric mixers and GLU-FFNs indicates architectural degrees-of-freedom remain for further improvement. Future research directions include:

• Exploring spectrum-preserving nonlinearities beyond GLU
• Integrating efficient approximations of full mixing for resource-constrained deployment
• Extending the analysis to self-supervised, cross-domain, and multimodal recommendation scenarios

Conclusion

The presented study delivers a rigorous spectral analysis of deep recommender architectures, exposes critical rank deficiencies in prevailing scaling approaches, and substantiates its claims via theoretical and empirical axes. The proposed RankElastor architecture establishes both the feasibility and necessity of spectrum-aware dense scaling for high-capacity recommendation systems, with direct implications for future foundation model design in information retrieval and automated decision-making.

(2605.23191)