Papers
Topics
Authors
Recent
Search
2000 character limit reached

Factual and Personalized Recommendations using Language Models and Reinforcement Learning

Published 9 Oct 2023 in cs.AI | (2310.06176v1)

Abstract: Recommender systems (RSs) play a central role in connecting users to content, products, and services, matching candidate items to users based on their preferences. While traditional RSs rely on implicit user feedback signals, conversational RSs interact with users in natural language. In this work, we develop a comPelling, Precise, Personalized, Preference-relevant LLM (P4LM) that recommends items to users while putting emphasis on explaining item characteristics and their relevance. P4LM uses the embedding space representation of a user's preferences to generate compelling responses that are factually-grounded and relevant w.r.t. the user's preferences. Moreover, we develop a joint reward function that measures precision, appeal, and personalization, which we use as AI-based feedback in a reinforcement learning-based LLM framework. Using the MovieLens 25M dataset, we demonstrate that P4LM delivers compelling, personalized movie narratives to users.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp.  1, 2004.
  2. The impact of popularity bias on fairness and calibration in recommendation. arXiv preprint arXiv:1910.05755, 2019.
  3. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71:89–129, 2008.
  4. Sample-efficient deep reinforcement learning for dialog control. arXiv preprint arXiv:1612.06000, 2016.
  5. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
  6. Latent cross: Making use of context in recurrent recommender systems. In Proceedings of the eleventh ACM international conference on web search and data mining, pp.  46–54, 2018.
  7. Fairness in recommendation ranking through pairwise comparisons. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp.  2212–2220, 2019.
  8. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015.
  9. On the independence of association bias and empirical fairness in language models. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp.  370–378, 2023.
  10. Multi-dqn: An ensemble of deep q-learning agents for stock market forecasting. Expert systems with applications, 164:113820, 2021.
  11. Bias and debias in recommender system: A survey and future directions. ACM Transactions on Information Systems, 41(3):1–39, 2023.
  12. Towards knowledge-based recommender dialog system. arXiv preprint arXiv:1908.05391, 2019.
  13. Markov recurrent neural network language model. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp.  807–813. IEEE, 2019.
  14. Towards conversational recommender systems. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp.  815–824, 2016.
  15. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
  16. Distilling policy distillation. In The 22nd international conference on artificial intelligence and statistics, pp.  1331–1340. PMLR, 2019.
  17. The expertise problem: Learning from specialized feedback. arXiv preprint arXiv:2211.06519, 2022.
  18. Persuasive recommendation: serial position effects in knowledge-based recommender systems. In Persuasive Technology: Second International Conference on Persuasive Technology, PERSUASIVE 2007, Palo Alto, CA, USA, April 26-27, 2007, Revised Selected Papers 2, pp.  283–294. Springer, 2007.
  19. Leveraging large language models in conversational recommender systems. arXiv preprint arXiv:2305.07961, 2023.
  20. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.  1587–1596. PMLR, 2018.
  21. Prince: Provider-side interpretability with counterfactual explanations in recommender systems. In Proceedings of the 13th International Conference on Web Search and Data Mining, pp.  196–204, 2020.
  22. PaLM 2 technical report, 2023.
  23. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.  1861–1870. PMLR, 2018.
  24. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4):1–19, 2015.
  25. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, pp.  173–182. ACM, 2017.
  26. Q2: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering. arXiv preprint arXiv:2104.08202, 2021.
  27. TRUE: Re-evaluating factual consistency evaluation. pp.  3905–3920, July 2022.
  28. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
  29. Generalized bradley-terry models and multi-class probability estimates. Journal of Machine Learning Research, 7(1), 2006.
  30. Personalized language model for query auto-completion. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pp.  700–705. Association for Computational Linguistics, 2018.
  31. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456, 2019.
  32. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023.
  33. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
  34. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020.
  35. Faithful or extractive? on mitigating the faithfulness-abstractiveness trade-off in abstractive summarization. arXiv preprint arXiv:2108.13684, 2021.
  36. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023.
  37. Conversational recommendation: Formulation, methods, and evaluation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.  2425–2428, 2020.
  38. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.01541, 2016.
  39. Towards deep conversational recommendations. Advances in neural information processing systems, 31, 2018.
  40. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.  74–81, 2004.
  41. Brio: Bringing order to abstractive summarization. arXiv preprint arXiv:2203.16804, 2022.
  42. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp.  193–200, 2007.
  43. Probabilistic matrix factorization. Advances in neural information processing systems, 20, 2007.
  44. Sentence-T5: Scalable sentence encoders from pre-trained Text-to-Text models. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pp.  1864–1874. Association for Computational Linguistics, 2022a.
  45. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp.  9844–9855, 2022b.
  46. OpenAI. Gpt-4 technical report, 2023.
  47. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  48. Moral: Aligning ai with human norms through multi-objective reinforced active learning. arXiv preprint arXiv:2201.00012, 2021.
  49. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779, 2020.
  50. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, 2016.
  51. Neural collaborative filtering vs. matrix factorization revisited. In RecSys 2020: Fourteenth ACM Conference on Recommender Systems, Virtual Event, Brazil, September 22-26, 2020, pp.  240–248. ACM, 2020.
  52. Recommender systems. Communications of the ACM, 40(3):56–58, 1997.
  53. Factually consistent summarization via reinforcement learning with textual entailment feedback. arXiv preprint arXiv:2306.00186, 2023.
  54. Behavioral cloning from noisy demonstrations. In International Conference on Learning Representations, 2020.
  55. Collaborative filtering recommender systems. In The adaptive web: methods and strategies of web personalization, pp.  291–324. Springer, 2007.
  56. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  57. Towards understanding and mitigating unintended biases in language model-driven conversational recommendation. Information Processing & Management, 60(1):103139, 2023.
  58. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460–9471, 2022.
  59. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
  60. A survey of collaborative filtering techniques. Advances in artificial intelligence, 2009, 2009.
  61. Hao Sun. Offline prompt evaluation and optimization with inverse reinforcement learning. arXiv preprint arXiv:2309.06553, 2023.
  62. Yueming Sun and Yi Zhang. Conversational recommender system. In The 41st international acm sigir conference on research & development in information retrieval, pp.  235–244, 2018.
  63. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  64. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992.
  65. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
  66. Transact: Transformer-based realtime user action model for recommendation at pinterest. arXiv preprint arXiv:2306.00248, 2023.
  67. The in-sample softmax for offline reinforcement learning. arXiv preprint arXiv:2302.14372, 2023.
  68. Mixed negative sampling for learning two-tower neural networks in recommendations. In Proceedings of the Web Conference (WWW-20), pp.  441–447, Taipei, 2020.
  69. Sampling-bias-corrected neural modeling for large corpus item recommendations. In Proceedings of the Thirteenth ACM Conference on Recommender Systems (RecSys19), pp.  269–277, Copenhagen, 2019.
  70. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pp.  11328–11339. PMLR, 2020a.
  71. Explainable recommendation: A survey and new perspectives. Foundations and Trends® in Information Retrieval, 14(1):1–101, 2020b.
  72. Towards topic-guided conversational recommender system. arXiv preprint arXiv:2010.04125, 2020.
  73. Principled reinforcement learning with human feedback from pairwise or k𝑘kitalic_k-wise comparisons. arXiv preprint arXiv:2301.11270, 2023.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up