UniFormer: Efficient and Unified Model-Centric Scaling for Industrial Recommendation

Abstract: Recently, substantial progress has been made in industrial recommendation through component-centric model scaling, where individual components such as behavior modeling, feature interaction, or task modeling are independently scaled to improve model capacity. Although recent methods such as HyFormer and OneTrans further explore cross-module co-scaling by jointly modeling behavior and interaction, their designs are still confined to the feature space and lack a unified model-centric scaling framework over the overall modeling space. In this paper, we propose UniFormer, an efficient and unified model-centric scaling framework for industrial recommender systems. To improve efficiency, UniFormer decomposes the overall modeling space into feature and task spaces, which are modeled by stacked Feature-space Interaction Modules and Task-space Interaction Modules, respectively. Moreover, UniFormer introduces semantic-based tokenization scheme to enable user-item decoupling, thereby achieving request-level inference acceleration. To prevent preference collapse, UniFormer employs multi-sequence cross-attention to separately capture heterogeneous behavior patterns, followed by the self-attention to enhance interaction modeling. Besides, dedicated multi-view FFNs are introduced to support flexible and scalable parameter scaling across different modeling components. Extensive online A/B testing in two production scenarios, Kuaishou and Kuaishou Lite, shows that UniFormer consistently improves user engagement and interaction metrics, achieving gains of +0.101%/+0.260% in App Stay Time and +0.729%/+1.113% in Watch Time, respectively.

• The paper demonstrates how a model-centric scaling approach unifies feature and task interactions to outperform traditional component-centric paradigms.
• It introduces key innovations such as semantic tokenization, multi-sequence cross-attention, and multi-view feed-forward networks to achieve balanced parameter allocation.
• Empirical results on Kuaishou’s system reveal significant GAUC improvements and a 48% inference boost, validating the model's industrial efficiency.

UniFormer: A Unified Model-Centric Scaling Approach for Industrial Recommendation

Motivation and Paradigm Shift

The paper "UniFormer: Efficient and Unified Model-Centric Scaling for Industrial Recommendation" (2606.27058) presents a substantive shift in the architectural strategies for industrial recommender systems. Historical approaches predominantly followed a component-centric scaling paradigm, independently scaling behavior modeling, feature interaction, or task modeling. While recent works (e.g., HyFormer, OneTrans) have attempted cross-module co-scaling, such designs are largely confined to the feature space and lack a truly unified scaling strategy. The UniFormer framework advocates model-centric scaling by integrating feature and task spaces as first-class objects, targeting efficiency, parameter allocation, and holistic modeling.

Figure 1: A comparative illustration of scaling paradigms, highlighting UniFormer’s unified co-scaling approach across sequential behaviors, non-sequence features, and tasks.

UniFormer Architecture: Modular Co-Scaling

The UniFormer architecture consists of two principal modules: the Feature-space Interaction Module (FIM) and the Task-space Interaction Module (TIM). Tokenization is central, utilizing semantic grouping to decouple user-item dependencies, thereby supporting request-level inference acceleration. The FIM captures both sequential and non-sequential feature interactions through multi-sequence cross-attention and adaptive FFNs, intentionally preventing preference collapse by modeling heterogeneous behavior sequences separately, followed by self-attention for dense feature interaction enhancement. The TIM aggregates high-order representations from FIM and task features, deploying standardized attention and FFNs for flexible parameter scaling.

Figure 2: The overall architecture of UniFormer, demonstrating tokenization and unified interaction via stacked FIMs and TIMs.

Technical Innovations

Semantic-Based Tokenization

Semantic tokenization distinguishes item-independent and item-dependent features, with further grouping based on explicit semantics (e.g., user ID, item statistics). Item-dependent behavioral sequences are aggregated, enabling user-item decoupling and computational reuse.

Multi-Sequence Cross-Attention

UniFormer’s FIM draws from multi-sequence cross-attention, avoiding a shared attention layer over concatenated sequences to reduce preference collapse. This mechanism allows extraction of multi-view user interests from heterogeneous sequences (short-term, long-term, cross-domain).

Multi-View Feed-Forward Networks

For scalable parameter allocation, multi-view FFNs are instantiated across sequential, non-sequential, and task features, providing balanced scaling and preventing parameter concentration in any single module. Adaptive fusion (global and user-personalized) accommodates user heterogeneity.

Optimizations

• User-level Compression: Compresses redundant user-side features for efficient batch processing.
• Variable-Length FlashAttention: Eliminates padding, reducing attention computation complexity.
• BF16 Mixed Precision: Enhances throughput for large industrial deployments.
• User-Item Decoupling: Achieves up to 48% serving acceleration without sacrificing GAUC quality.

Empirical Validation and Scaling Laws

Offline evaluation on Kuaishou’s short-video recommendation system demonstrates that UniFormer consistently outperforms preceding baselines across all GAUC metrics, with improvements of up to +0.53% GAUC for main interaction tasks and +0.89% for follow tasks. UniFormer-Large further extends these gains, exploiting scalable parameter allocation. Ablation studies confirm the centrality of semantic tokenization, multi-sequence cross-attention, and multi-view FFNs.

Figure 3: Ablation study results for UniFormer, validating contributions of key architectural elements.

UniFormer obeys scaling laws, with a clear power-law relationship between parameter count and GAUC improvement across tasks, efficiently avoiding the diminishing returns seen in component-centric scaling.

Figure 4: Scaling law analysis, showing monotonic GAUC gain with increased model parameters.

Attention Mechanism Visualization

Semantic tokenization induces heterogeneous, layer-adaptive attention distributions enabling competitive specialized feature extraction and mitigating homogenization. Tasks focus adaptively on distinct feature tokens—item statistics, user profiles, or search behaviors—aligned with their predictive objectives.

Figure 5: Visualization of attention patterns, comparing semantic tokenization and global tokenization.

Figure 6: Attention distributions in TIM, elucidating feature-task relationships and cross-task correlation.

Online Deployment and Efficiency

Real-world A/B testing in Kuaishou and Kuaishou Lite validates UniFormer’s industrial feasibility. Gains of +0.101%/+0.260% in App Stay Time and +0.729%/+1.113% in Watch Time are statistically significant, with further improvements in Like, Comment, Collect, and Forward metrics. The serving architecture allows request-level decoupling, providing a 48% inference QPS boost.

Theoretical and Practical Implications

UniFormer unlocks a model-centric scaling paradigm, substantially advancing parameter efficiency, multi-task personalization, and holistic behavioral modeling. The approach suggests robust scaling beyond feature space, explicitly incorporating task-space interactions and supporting flexible deployment optimizations. Its modularity and adaptive parameter allocation pave the way for future generative recommenders, large-scale personalized ranking models, and integration with multi-modal and multi-objective industrial AI systems.

Conclusion

UniFormer establishes a unified, efficient, and model-centric scaling architecture for industrial recommendation, integrating cross-feature, behavior, and task modeling through standardized attention and multi-view FFNs. Empirical results and production deployment substantiate its superiority over feature-centric scaling, providing actionable insights for scaling recommendation models in high-throughput environments. This paradigm likely informs the next wave of industrial recommender systems, with theoretical relevance for emergent scaling laws and practical relevance for personalized large-scale ranking deployments.