RankUp: Towards High-rank Representations for Large Scale Advertising Recommender Systems

Abstract: The scaling laws for recommender systems have been increasingly validated, where MetaFormer-based architectures consistently benefit from increased model depth, hidden dimensionality, and user behavior sequence length. However, whether representation capacity scales proportionally with parameter growth remains largely unexplored. Prior studies on RankMixer reveal that the effective rank of token representations exhibits a damped oscillatory trajectory across layers, failing to increase consistently with depth and even degrading in deeper layers. Motivated by this observation, we propose \textbf{RankUp}, an architecture designed to mitigate representation collapse and enhance expressive capacity through randomized permutation splitting over sparse features, a multi-embedding paradigm, global token integration, crossed pretrained embedding tokens and task-specific token decoupling. RankUp has been fully deployed in large-scale production across Weixin Video Accounts, Official Accounts and Moments, yielding GMV improvements of 3.41\%, 4.81\% and 2.21\%, respectively.

• The paper introduces RankUp, which enhances effective rank using randomized token splitting, multi-embedding, and task-specific tokens.
• It mitigates representational collapse in deep recommender systems, achieving up to 0.41% AUC and 4.81% GMV lift in real-world tests.
• Empirical evaluations using SVD and mutual information metrics confirm that explicit rank augmentation drives both performance and business impact.

High-Rank Representations in Large-Scale Industrial Recommender Systems: An Analysis of RankUp

Motivation and Theoretical Foundations

The work introduces RankUp, addressing effective-rank collapse—a core limitation in contemporary MetaFormer-based recommender architectures that persists despite observed scaling laws in model size. Empirical evidence shows that increasing depth, width, and sequence length in deep recommenders (e.g., RankMixer, Hiformer, Mixformer) does not monotonically expand the effective rank of intermediate representations; instead, representation collapse and oscillatory rank trajectories are manifested in deeper layers, fundamentally constraining expressivity in high-dimensional industrial feature spaces. Effective-rank analysis, via singular value decomposition and Shannon entropy measures, quantifies this collapse and highlights that naive parameter growth fails to realize the representational benefits theoretically anticipated (2604.17878).

RankUp Architecture and Component Analysis

RankUp’s framework explicitly targets the augmentation of the effective rank, decomposing the bottlenecks in standard token mixing and per-token channelwise FFN design. Key architectural mechanisms include:

• Randomized Permutation Splitting: Sparse features are stochastically partitioned, distributing highly correlated features across token groups and thus decorrelating input representations. This contrasts with legacy semantic splitting, which induces low-rank bottlenecks by aggregating collinear, long-tail feature sets into single tokens. Empirical MI difference matrices validate reduced inter-token redundancy under randomized permutation.

Figure 1: MI difference matrices ($$\mathbf{M}_{\text{Randomized}} - \mathbf{M}_{\text{Semantic}}$$) reveal lower redundancy and improved token independence via randomization.

• Multi-Embedding Paradigm: The input manifold is expanded via $K$ independent embedding tables, allowing each sparse feature to be projected into several orthogonal subspaces. This increases the degrees of freedom in the initial token matrix and mitigates early-stage representational collapse.
• Global Token Integration: By appending an aggregated global contextual token—computed via aggregation (MLP/FM/DCN) over all features—RankUp enables effective cross-token interactions, especially in later layers.
• Cross Integration of Pretrained Embeddings: User-item interaction priors, pretrained in massive retrieval settings, are injected explicitly as tokens by elementwise multiplication, facilitating better utilization of global semantic priors in the downstream ranking tasks.
• Task-Specific Token Decoupling: For multi-objective settings, task-specific tokens allow gradient flow separation, reducing adverse interference among heterogeneous prediction targets, thus preserving high-rank latent space throughout depth.

  Figure 2: Overall framework illustrating token input splitting, multi-embedding, global/contextual, cross-domain, and task-specific token flows in RankUp.

Empirical Evaluation: Rank Dynamics and Mutual Information

Comprehensive experiments on Tencent’s large-scale ad recommendation data sets demonstrate that each architectural intervention addresses distinct representational deficits:

• Effective Rank Analysis: Randomized permutation yields higher and more uniform effective rank across sparse tokens, counteracting the rank collapse of semantic grouping when subjected to low-cardinality feature concentration.

  Figure 3: Token-level effective rank. Randomized splitting uniformly improves rank across 32 sparse tokens.

• Layerwise Dynamics: The introduction of global token and multi-embedding slows effective-rank degeneration in deeper layers across both TokenMixer and FFN modules, as shown by ablation studies. Task-specific tokens demonstrate the largest impact on rank preservation for multi-task optimization.

  Figure 4: Ablation study showing the individual and collective effect of architectural components on the effective-rank trajectory across layers.

• Mutual Information Alignment: The use of task-specific tokens significantly increases the MI between learned representations and downstream task labels, especially as the latent space is discretized into finer clusters. This demonstrates better preservation of predictive signals relevant to each target.

  Figure 5: Task-specific tokens produce larger MI with task labels as cluster granularity increases, indicating improved representational alignment.

Business Impact and Large-Scale Deployment

RankUp was deployed at scale across WeChat’s advertising scenarios, including Video Accounts and Moments. Notable empirical results:

• Realtime AUC Gains: Up to 0.41% AUC increase on high-traffic tasks with incremental improvements from each core mechanism.
• GMV Lift: Online A/B testing showed 3.41%–4.81% absolute GMV lift across product lines, extrapolating to substantial annual revenue impact.
• Cold-Start Robustness: Gains amplified in new-ad scenarios (up to 9.67% GMV) and critical revenue-driving downstream targets (7.18% GMV on “Order” task).

Importantly, improvements in effective-rank metrics translated directly to operational KPIs, validating that explicit optimization for representational diversity can bridge the gap between scaling laws and monetizable model performance.

Theoretical and Practical Implications

This work refines the understanding of scaling in industrial recommenders, demonstrating parameter growth alone is insufficient for maximizing representation capacity. By architecturally modulating token correlation, embedding diversity, and cross-task interference, RankUp offers a model for next-generation MetaFormer variants that target effective space utilization. The introduction of explicit rank-augmentation mechanisms provides new inductive biases and regularization strategies applicable at scale.

The findings challenge the sufficiency of standard scaling and motivate further research on architectural interventions (e.g., decorrelating initializations, explicit MI regularization, learnable task-token routing) to ensure monotonic growth of representation rank and downstream utility. Future developments may focus on dynamic adaptation strategies, cross-domain meta-learned representation splitting, and more granular alignment objectives for contrastive and multi-modal extensions.

Conclusion

RankUp establishes that high-rank representation design, not mere parameter count, governs the expressive threshold in large-scale recommender systems. The methods introduced—randomized token splitting, multi-embedding, global and cross-domain tokenization, and task-aware separation—are validated both in terms of effective-rank analysis and online business impact. These results emphasize that future industrial model development must prioritize not only scaling, but also explicit interventions that foster representational diversity and robustness (2604.17878).