Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation (2409.07416v1)

Abstract: Modern listwise recommendation systems need to consider both long-term user perceptions and short-term interest shifts. Reinforcement learning can be applied on recommendation to study such a problem but is also subject to large search space, sparse user feedback and long interactive latency. Motivated by recent progress in hierarchical reinforcement learning, we propose a novel framework called mccHRL to provide different levels of temporal abstraction on listwise recommendation. Within the hierarchical framework, the high-level agent studies the evolution of user perception, while the low-level agent produces the item selection policy by modeling the process as a sequential decision-making problem. We argue that such framework has a well-defined decomposition of the outra-session context and the intra-session context, which are encoded by the high-level and low-level agents, respectively. To verify this argument, we implement both a simulator-based environment and an industrial dataset-based experiment. Results observe significant performance improvement by our method, compared with several well-known baselines. Data and codes have been made public.

• The paper presents a novel mccHRL framework that decouples long-term user modeling and in-session item selection using hierarchical reinforcement learning.
• The methodology leverages a high-level agent for outer-session context and a low-level agent for real-time item selection on edge devices.
• Experimental results on MovieLens-100k and Alibaba datasets show improved average ratings and AUC metrics compared to traditional recommendation approaches.

Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation

In the paper "Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation," the authors present a novel framework called Mobile-Cloud Collaborative Hierarchical Reinforcement Learning (mccHRL) to address the complex problem of listwise recommendation systems, especially in contexts where both long-term and short-term user interest must be considered. The research aims to leverage hierarchical reinforcement learning (HRL) to decouple the temporal aspect of user interest evolution and item selection policies, incorporating both inter-session and intra-session contexts.

Introduction

The complexity of listwise recommendation systems, often employed in various domains such as product recommendations, advertisements, and content feeds, necessitates sophisticated methodologies to enhance interaction with users. Traditional approaches in RL-based recommendations are limited either by coarse-grained time intervals or by computational complexities when dealing with large-scale, session-based recommendations. To overcome these bottlenecks, the paper proposes the mccHRL framework. This framework employs two levels of agents: a high-level agent focused on long-term user perception and a low-level agent optimizing item selection within a session.

Methodology

The mccHRL framework introduces an innovative structure of hierarchical RL where:

• High-Level Agent (HRA): This agent models user perception and outer-session context. The state comprises the user profile, historical interactions, and temporal context. The action is an embedding vector representing the user's long-term preference for a session, and the reward structure is based on user interaction metrics like click-through rates (CTR) over a session.
• Low-Level Agent (LRA): This agent deals with item selection within a session, considering on-device, intra-session context. The state includes previous item selections and immediate user interactions. The action involves scoring and selecting the next item in the ranked list, leveraging on-device features to refine the recommendation in real-time.

The HRA updates the user's long-term preference, while the LRA optimizes item selection to achieve the high-level goals set by the HRA. This hierarchical setup not only tackles the sparse reward issues commonly faced in RL but also improves computational efficiency through mobile-cloud collaboration, as the low-level agent operates on edge devices.

Implementation

The authors have presented detailed experimental setups to validate the mccHRL framework. Two primary evaluation strategies were used:

1. Simulator-based Experiment: A recommendation simulator was created using the MovieLens-100k dataset. Specific user features were designated as edge features, and on-device data was intentionally delayed to simulate real-world latency.
2. Dataset-based Experiment: A large industrial dataset from Alibaba's Taobao front-page recommendation system was utilized. The dataset included both on-device and cloud-accessible features, adding realism to the experiment.

In both cases, the performance metrics focused on average user ratings and AUC (Area Under the ROC Curve) for classification tasks, respectively.

Results

The mccHRL framework demonstrated improved performance compared to several baselines, including traditional methods like GRU4rec, DIN, and RL-based methods like LIRD. The empirical results highlighted:

• Simulator Experiment: The mccHRL framework achieved an average rating of 3.824 surpassing other methods like DIN and LIRD. This validated its efficacy in dealing with simulated real-world recommendation tasks.
• Dataset-based Experiment: The D-AUC for mccHRL reached 0.853, higher than that of the baseline models, including a hypothetical DIN model with zero latency, underscoring the practical utility of mccHRL in handling edge-cloud collaborations effectively.

Discussion

The hierarchical approach effectively addresses the limitations of single-layer RL models by segregating the user modeling and item selection tasks across temporal scales. The consideration of on-device features and real-time orchestration between mobile and cloud components provides a robust framework that can adapt to changing user behaviors and contexts.

Future Work

The current work could be further explored in various dimensions, including:

• Enhancing the low-level agent's responsiveness to real-time user interactions within sessions.
• Extending the framework to incorporate multi-goal optimization, balancing user satisfaction with business metrics.
• Investigating the scalability of mccHRL in larger and more diverse industrial setups.

Conclusion

The mccHRL framework offers a structured and efficient approach to handling listwise recommendations by leveraging hierarchical reinforcement learning and mobile-cloud infrastructure. This dual-agent system allows for more nuanced and effective user interaction modeling, demonstrating significant improvements over traditional methodologies. The paper provides a promising direction for future research in hierarchical and collaborative machine learning frameworks for industrial applications.