Scaling Sequential Recommendation Models with Transformers (2412.07585v1)

Abstract: Modeling user preferences has been mainly addressed by looking at users' interaction history with the different elements available in the system. Tailoring content to individual preferences based on historical data is the main goal of sequential recommendation. The nature of the problem, as well as the good performance observed across various domains, has motivated the use of the transformer architecture, which has proven effective in leveraging increasingly larger amounts of training data when accompanied by an increase in the number of model parameters. This scaling behavior has brought a great deal of attention, as it provides valuable guidance in the design and training of even larger models. Taking inspiration from the scaling laws observed in training LLMs, we explore similar principles for sequential recommendation. We use the full Amazon Product Data dataset, which has only been partially explored in other studies, and reveal scaling behaviors similar to those found in LLMs. Compute-optimal training is possible but requires a careful analysis of the compute-performance trade-offs specific to the application. We also show that performance scaling translates to downstream tasks by fine-tuning larger pre-trained models on smaller task-specific domains. Our approach and findings provide a strategic roadmap for model training and deployment in real high-dimensional preference spaces, facilitating better training and inference efficiency. We hope this paper bridges the gap between the potential of transformers and the intrinsic complexities of high-dimensional sequential recommendation in real-world recommender systems. Code and models can be found at https://github.com/mercadolibre/srt

• The paper introduces a trainable feature extractor that decouples parameter count from catalog size to enhance scalability.
• The paper leverages contrastive learning to capture catalog diversity and improve prediction accuracy in dynamic environments.
• The paper validates scaling laws on the full Amazon Product Data, identifying compute-optimal trajectories for balanced performance.

Scaling Sequential Recommendation Models with Transformers: A Technical Overview

The contemporary landscape of sequential recommendation models has been significantly influenced by inspiration drawn from scaling behaviors of transformers in LLMs. This paper, authored by Zivic et al., explores the scaling principles for transformer-based sequential recommendation systems with meticulous attention to aspects that distinguish recommendation tasks from LLMing.

Key Contributions

In this paper, the authors present pivotal adaptations to transformer-based frameworks aimed at optimizing sequential recommendation tasks in expansive catalogs. Their approach includes two primary modifications to traditional models:

1. Trainable Feature Extractor: They pivot away from fixed-item embeddings and propose using a trainable feature extractor. This methodological shift ensures that the parameter count remains invariant to the size of the item catalog, addressing challenges posed by vast and dynamic item spaces typical of e-commerce platforms.
2. Contrastive Learning Formulation: Incorporating a contrastive learning setup allows for enhanced capturing of catalog diversity, thereby refining representations and improving the model's adaptability and predictive capabilities. This formulation is designed to manage the variance and heterogeneity inherent in user interaction patterns within large-scale recommender systems.

Numerical Results and Scaling Behavior

A cornerstone of the research is the empirical validation of the proposed model's scaling capabilities using the full Amazon Product Data (APD) dataset. The authors unfold scaling laws akin to those observed in LLMs, directing how increasing data access and parameter tuning correlate with performance improvements. Notable is their discovery of a compute-optimal training trajectory that judiciously balances computational overhead against enhancements in accuracy, providing a roadmap for system designers in effectively scaling recommender models.

Implications and Future Directions

The implications of these findings extend across both theoretical and practical realms. Theoretically, the insights reinforce the hypothesis that scaling behaviors in one domain can inform strategies in seemingly disparate domains like recommendation systems. Practically, these insights lay the groundwork for more robust, scalable, and efficient models suitable for deployment in high-dimensional preference spaces, such as large-scale e-commerce platforms.

Looking forward, one possible trajectory for further research is an exploration of transforming these sequential recommendation models into zero-shot or few-shot learners, leveraging the pre-training and fine-tuning paradigm that's indicative of success across other fields of AI. Moreover, given the dynamic nature of real-world item catalogs and user preferences, continuous learning frameworks that can adapt to new data in real time while maintaining computational efficiency might become indispensable.

Conclusion

In conclusion, Zivic et al.'s work navigates the complexities of scaling sequential recommendation models with precision, crafting a technically sound blueprint for leveraging transformer architectures in domains enriched with temporal user interaction data. The models they propose represent a step forward in bridging the conceptual gaps between LLM scaling and recommendation system design, demonstrating that with thoughtful adaptations, transformer-based models can be effectively tailored to meet the demands of large-scale, real-world recommendation tasks.