Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations

Abstract: Large-scale recommendation systems are characterized by their reliance on high cardinality, heterogeneous features and the need to handle tens of billions of user actions on a daily basis. Despite being trained on huge volume of data with thousands of features, most Deep Learning Recommendation Models (DLRMs) in industry fail to scale with compute. Inspired by success achieved by Transformers in language and vision domains, we revisit fundamental design choices in recommendation systems. We reformulate recommendation problems as sequential transduction tasks within a generative modeling framework ("Generative Recommenders"), and propose a new architecture, HSTU, designed for high cardinality, non-stationary streaming recommendation data. HSTU outperforms baselines over synthetic and public datasets by up to 65.8% in NDCG, and is 5.3x to 15.2x faster than FlashAttention2-based Transformers on 8192 length sequences. HSTU-based Generative Recommenders, with 1.5 trillion parameters, improve metrics in online A/B tests by 12.4% and have been deployed on multiple surfaces of a large internet platform with billions of users. More importantly, the model quality of Generative Recommenders empirically scales as a power-law of training compute across three orders of magnitude, up to GPT-3/LLaMa-2 scale, which reduces carbon footprint needed for future model developments, and further paves the way for the first foundational models in recommendations.

• The paper presents a novel generative approach that reformulates recommendation tasks as sequential transduction problems using Hierarchical Sequential Transduction Units (HSTU).
• It employs a pointwise aggregated attention mechanism that accelerates training and inference, efficiently handling sequences up to 8192 in length.
• The design leverages power-law scaling and achieves a 12.4% improvement in online metrics, paving the way for scalable, resource-efficient recommendation systems.

Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations

In this essay, we explore the implementation and practical implications of the research presented in "Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations" (2402.17152). This paper introduces Generative Recommenders (GRs) as an alternative paradigm for building large-scale recommendation systems using a holistic generative framework. The authors propose a novel architecture, Hierarchical Sequential Transduction Units (HSTU), to efficiently handle high cardinality, non-stationary streaming recommendation data.

Reimagining Recommendation Systems as Generative Tasks

The core idea of the paper is the reformulation of recommendation tasks into sequential transduction tasks under a generative modeling framework. Traditional Deep Learning Recommendation Models (DLRMs) often struggle to scale with compute despite being trained on extensive datasets. Inspired by the success of Transformers in NLP and computer vision, the authors propose a shift towards a generative approach, allowing systemic leverage of redundancies in feature representation, training, and inference.

GRs leverage sequential task formulation where both ranking and retrieval are cast as autoregressive tasks. This approach allows for the transformation and unification of heterogeneous feature spaces prevalent in traditional recommendation systems. By utilizing a consolidated time-series data model, GRs can significantly enhance their ability to make predictions about user behavior.

Figure 1: Comparison of features and training procedures: DLRMs vs GRs.

Hierarchical Sequential Transduction Units (HSTU)

The paper proposes HSTU as an efficient self-attention encoder architecture tailored for generative recommendation tasks. HSTU replaces the diverse and specialized components of traditional DLRMs with a modular attention-based block, effectively performing feature transformation and interaction. It is designed to optimize compute by employing a pointwise aggregated attention mechanism, which contrasts with the traditional softmax approach.

HSTU achieves speedups compared to FlashAttention2-based Transformers due to its tailored attention mechanism, enabling it to handle sequences with lengths up to 8192 efficiently. This efficiency positions GRs as a scalable alternative for large-scale industrial applications, seamlessly processing extensive user interaction data streams in a generative sequence-to-sequence manner.

Figure 2: Encoder-level efficiency: HSTU vs FlashAttention2-based Transformers for Training (a) and Inference (b, c).

Implementation Considerations and Performance Metrics

The paper highlights strong numerical results with GRs achieving improvements by 12.4% in online metrics during A/B testing over baseline models. The generative approach supported by HSTU architecture allows GRs to scale effectively, with empirical evidence pointing to a power-law scaling with the training compute, akin to scaling laws observed in LLMs like GPT-3 and LLaMA-2.

In terms of computational efficiency, the use of HSTU and advanced techniques like Stochastic Length and pointwise attention not only optimize training costs but also reduce the memory required during inference, making it feasible to deploy trillion-parameter GR models in production environments with minimal resource overhead.

Figure 3: Scalability: DLRMs vs GRs in large-scale industrial settings across retrieval (top, middle) and ranking (bottom).

Conclusion

The research presented in this paper establishes a compelling framework for transforming recommendation systems through generative modeling and scalable attention architectures. By converging on a generative transduction task model and leveraging innovations in efficient attention mechanisms, GRs represent a significant shift towards more scalable, computation-efficient recommendation systems. This work not only opens pathways for deploying more robust models in recommendation scenarios but also hints at a potential unified architecture for future generative AI applications across different domains.