Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction (2006.05639v2)

Abstract: Rich user behavior data has been proven to be of great value for click-through rate prediction tasks, especially in industrial applications such as recommender systems and online advertising. Both industry and academy have paid much attention to this topic and propose different approaches to modeling with long sequential user behavior data. Among them, memory network based model MIMN proposed by Alibaba, achieves SOTA with the co-design of both learning algorithm and serving system. MIMN is the first industrial solution that can model sequential user behavior data with length scaling up to 1000. However, MIMN fails to precisely capture user interests given a specific candidate item when the length of user behavior sequence increases further, say, by 10 times or more. This challenge exists widely in previously proposed approaches. In this paper, we tackle this problem by designing a new modeling paradigm, which we name as Search-based Interest Model (SIM). SIM extracts user interests with two cascaded search units: (i) General Search Unit acts as a general search from the raw and arbitrary long sequential behavior data, with query information from candidate item, and gets a Sub user Behavior Sequence which is relevant to candidate item; (ii) Exact Search Unit models the precise relationship between candidate item and SBS. This cascaded search paradigm enables SIM with a better ability to model lifelong sequential behavior data in both scalability and accuracy. Apart from the learning algorithm, we also introduce our hands-on experience on how to implement SIM in large scale industrial systems. Since 2019, SIM has been deployed in the display advertising system in Alibaba, bringing 7.1\% CTR and 4.4\% RPM lift, which is significant to the business. Serving the main traffic in our real system now, SIM models user behavior data with maximum length reaching up to 54000, pushing SOTA to 54x.

• The paper introduces the SIM framework, a two-stage method combining general and exact search units to effectively model lifelong user behavior for CTR prediction.
• SIM scales user behavior analysis to sequences of up to 54,000 events, far surpassing previous models limited to 1,000 events.
• Deployed in Alibaba’s advertising system, the model achieves a 7.1% uplift in CTR and a 4.4% increase in RPM, showcasing its industrial impact.

Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction

The paper "Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction" authored by researchers from Alibaba Group presents an advanced approach to harnessing extensive user behavior data for improving Click-Through Rate (CTR) prediction models. Traditional models often face limitations when applied to long sequences of user data due to computational and storage constraints. The authors introduce a novel method named the Search-based Interest Model (SIM), designed to overcome these limitations and efficiently model user interests using extensive user behavior sequences.

Key Contributions

The paper makes several significant contributions:

1. Introduction of SIM: The authors propose a two-stage user interest modeling framework called SIM that integrates a general search unit with an exact search unit. This approach enables the effective filtering and precise modeling of user behavior data.
2. Scalability: SIM demonstrates the ability to scale the modeling of user sequential behavior data up to lengths of 54,000, surpassing previous state-of-the-art models like MIMN, which had a maximum length capability of 1,000.
3. Industrial Application: The model has been successfully deployed in Alibaba's display advertising system, achieving a substantial uplift of 7.1% in CTR and 4.4% in Revenue Per Mille (RPM). The deployment details showcase its practical benefits and scalability.

Methodology

The SIM framework utilizes a two-stage approach for processing long-term user behavior sequences:

• General Search Unit (GSU): This initial stage employs either a hard-search or soft-search strategy to identify and filter relevant user behavior sequences concerning a candidate item. The hard-search leverages category information for filtering, while the soft-search uses metric-driven similarity to select top-K relevant behaviors. The search process significantly reduces the sequence length, making subsequent processing feasible within practical time constraints.
• Exact Search Unit (ESU): The reduced sequence is passed through an exact search unit, which utilizes multi-head attention mechanisms to capture precise user interests. This component augments the model's capacity to discern nuanced interests aligned with specific candidate items.

Results and Implications

In extensive experiments on public datasets, SIM outperforms existing state-of-the-art models like MIMN, highlighting its effectiveness and robustness. The integration of lifelong user data poses considerable advantages, allowing models to capture rich datasets of user behavior that span extensive temporal frames. Furthermore, the deployment of SIM in Alibaba's massive-scale industrial system indicates its practical applicability and substantial impact on business metrics.

Future Developments

Looking forward, the paper suggests potential future work in developing user-specific models that adaptively structure lifelong behavior data according to personalized schemas, thereby enhancing customization and accuracy in understanding user interest dynamics. This could potentially lead to even more sophisticated personalization approaches in both recommendation and advertising systems.

The advancements presented in this paper reflect an essential evolution in handling large-scale user behavior data, offering insights and methodologies that could be transformative for various applications within artificial intelligence, particularly in domains where understanding user interaction patterns is paramount.