Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback (2401.08959v1)

Published 17 Jan 2024 in cs.LG and cs.AI

Abstract: Probabilistic learning to rank (LTR) has been the dominating approach for optimizing the ranking metric, but cannot maximize long-term rewards. Reinforcement learning models have been proposed to maximize user long-term rewards by formulating the recommendation as a sequential decision-making problem, but could only achieve inferior accuracy compared to LTR counterparts, primarily due to the lack of online interactions and the characteristics of ranking. In this paper, we propose a new off-policy value ranking (VR) algorithm that can simultaneously maximize user long-term rewards and optimize the ranking metric offline for improved sample efficiency in a unified Expectation-Maximization (EM) framework. We theoretically and empirically show that the EM process guides the leaned policy to enjoy the benefit of integration of the future reward and ranking metric, and learn without any online interactions. Extensive offline and online experiments demonstrate the effectiveness of our methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Variational inference: A review for statisticians. JASA.
  2. Learning to rank using gradient descent. In ICML, 89–96.
  3. Learning to rank: from pairwise approach to listwise approach. In ICML.
  4. Top-K Off-Policy Correction for a REINFORCE Recommender System. In WSDM, 456–464.
  5. Doubly Robust Policy Evaluation and Learning. In ICML.
  6. Off-policy deep reinforcement learning without exploration. In ICML.
  7. Posterior regularization for structured latent variable models. JMLR.
  8. Reinforcement learning with deep energy-based policies. In ICML.
  9. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML.
  10. Expectation propagation for approximate inference in. In UAI.
  11. Session-based recommendations with recurrent neural networks. In ICLR.
  12. Distilling the Knowledge in a Neural Network. stat, 1050: 9.
  13. Collaborative filtering for implicit feedback datasets. In ICDM.
  14. Recsim: A configurable simulation platform for recommender systems. arXiv.
  15. Self-attentive sequential recommendation. In ICDM, 197–206. IEEE.
  16. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779.
  17. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643.
  18. Variational autoencoders for collaborative filtering. In WWW, 689–698.
  19. Continuous control with deep reinforcement learning. arXiv.
  20. Safe and efficient off-policy reinforcement learning. In NIPS, 1054–1062.
  21. A view of the EM algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, 355–368. Springer.
  22. BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618.
  23. Rényi, A.; et al. 1961. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability.
  24. Robertson, S. E. 1977. The probability ranking principle in IR. Journal of documentation.
  25. Prioritized Experience Replay. In ICLR.
  26. Personalized top-n sequential recommendation via convolutional sequence embedding. In WSDM, 565–573.
  27. Q-learning. Machine learning.
  28. Listwise approach to learning to rank: theory and algorithm. In ICML.
  29. Learning How to Propagate Messages in Graph Neural Networks. In KDD, 1894–1903.
  30. Hierarchical neural variational model for personalized sequential recommendation. In WWW, 3377–3383.
  31. A General Offline Reinforcement Learning Framework for Interactive Recommendation. In AAAI, 4512–4520.
  32. Self-supervised reinforcement learning for recommender systems. In SIGIR, 931–940.
  33. Deep reinforcement learning for page-wise recommendations. In RecSys, 95–103.
  34. Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning. In KDD, 1040–1048.
  35. DRN: A Deep Reinforcement Learning Framework for News Recommendation. In WWW, 167–176.
  36. Interactive Recommender System via Knowledge Graph-enhanced Reinforcement Learning. In SIGIR, 179–188.
  37. Pseudo Dyna-Q: A Reinforcement Learning Framework for Interactive Recommendation. In WSDM, 816–824.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets