ChatGPT for Conversational Recommendation: Refining Recommendations by Reprompting with Feedback (2401.03605v1)
Abstract: Recommendation algorithms have been pivotal in handling the overwhelming volume of online content. However, these algorithms seldom consider direct user input, resulting in superficial interaction between them. Efforts have been made to include the user directly in the recommendation process through conversation, but these systems too have had limited interactivity. Recently, LLMs like ChatGPT have gained popularity due to their ease of use and their ability to adapt dynamically to various tasks while responding to feedback. In this paper, we investigate the effectiveness of ChatGPT as a top-n conversational recommendation system. We build a rigorous pipeline around ChatGPT to simulate how a user might realistically probe the model for recommendations: by first instructing and then reprompting with feedback to refine a set of recommendations. We further explore the effect of popularity bias in ChatGPT's recommendations, and compare its performance to baseline models. We find that reprompting ChatGPT with feedback is an effective strategy to improve recommendation relevancy, and that popularity bias can be mitigated through prompt engineering.
- Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
- Second workshop on information heterogeneity and fusion in recommender systems (HetRec2011). In Proceedings of the fifth ACM conference on Recommender systems. 387–388.
- Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
- Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720 (2022).
- Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System. arXiv preprint arXiv:2303.14524 (03 2023).
- Chat-rec: Towards interactive and explainable llms-augmented recommender system. arXiv preprint arXiv:2303.14524 (2023).
- F Maxwell Harper and Joseph A Konstan. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) 5, 4 (2015), 1–19.
- An algorithmic framework for performing collaborative filtering. In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval. 230–237.
- Introducing serendipity in a content-based recommender system. In 2008 eighth international conference on hybrid intelligent systems. IEEE, 168–173.
- IMDb. 2023. IMDb Top 250. https://www.imdb.com/chart/top/ Accessed: 2023-07.
- Intra-list similarity and human diversity perceptions of recommendations: the details matter. User Modeling and User-Adapted Interaction (2022), 1–34.
- Marius Kaminskas and Derek Bridge. 2016. Diversity, serendipity, novelty, and coverage: a survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Transactions on Interactive Intelligent Systems (TiiS) 7, 1 (2016), 1–42.
- Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction. arXiv preprint arXiv:2305.06474 (2023).
- A Survey of Algorithmic Recourse: Contrastive Explanations and Consequential Recommendations. ACM Comput. Surv. 55, 5, Article 95 (dec 2022), 29 pages. https://doi.org/10.1145/3527848
- Bing Liu et al. 2011. Web data mining: exploring hyperlinks, contents, and usage data. Vol. 1. Springer.
- Is chatgpt a good recommender? a preliminary study. arXiv preprint arXiv:2304.10149 (2023).
- Training language models to follow instructions with human feedback. arXiv:2203.02155 [cs.CL]
- Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
- Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferences. arXiv:2307.14225 [cs.IR]
- Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web. 285–295.
- BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. arXiv:1904.06690 [cs.IR]
- A personalized system for conversational recommendations. Journal of Artificial Intelligence Research 21 (2004), 393–428.
- Actionable Recourse in Linear Classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM. https://doi.org/10.1145/3287560.3287566
- Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837.
- Xi Ye and Greg Durrett. 2022. The unreliability of explanations in few-shot in-context learning. arXiv preprint arXiv:2205.03401 (2022).
- Li Yujian and Liu Bo. 2007. A normalized Levenshtein distance metric. IEEE transactions on pattern analysis and machine intelligence 29, 6 (2007), 1091–1095.
- Language models as recommender systems: Evaluations and limitations. (2021).
- Improving recommendation lists through topic diversification. In Proceedings of the 14th international conference on World Wide Web. 22–32.
- Kyle Dylan Spurlock (1 paper)
- Cagla Acun (1 paper)
- Esin Saka (1 paper)
- Olfa Nasraoui (10 papers)