A Preliminary Study of ChatGPT on News Recommendation: Personalization, Provider Fairness, Fake News (2306.10702v1)
Abstract: Online news platforms commonly employ personalized news recommendation methods to assist users in discovering interesting articles, and many previous works have utilized LLM techniques to capture user interests and understand news content. With the emergence of LLMs like GPT-3 and T-5, a new recommendation paradigm has emerged, leveraging pre-trained LLMs for making recommendations. ChatGPT, with its user-friendly interface and growing popularity, has become a prominent choice for text-based tasks. Considering the growing reliance on ChatGPT for language tasks, the importance of news recommendation in addressing social issues, and the trend of using LLMs in recommendations, this study conducts an initial investigation of ChatGPT's performance in news recommendations, focusing on three perspectives: personalized news recommendation, news provider fairness, and fake news detection. ChatGPT has the limitation that its output is sensitive to the input phrasing. We therefore aim to explore the constraints present in the generated responses of ChatGPT for each perspective. Additionally, we investigate whether specific prompt formats can alleviate these constraints or if these limitations require further attention from researchers in the future. We also surpass fixed evaluations by developing a webpage to monitor ChatGPT's performance on weekly basis on the tasks and prompts we investigated. Our aim is to contribute to and encourage more researchers to engage in the study of enhancing news recommendation performance through the utilization of LLMs such as ChatGPT.
- User-centered evaluation of popularity bias in recommender systems. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization. 119–129.
- PM Abernathy. 2018. The Expanding News Desert, Center for Innovation and Sustainability in Local Media.
- Neural news recommendation with long-and short-term user representations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 336–345.
- A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023).
- Balanced neighborhoods for multi-sided fairness in recommendation. In Conference on fairness, accountability and transparency. PMLR, 202–214.
- Kathleen Chaykowski. 2018. Facebook’s Latest Algorithm Change: Here Are The News Sites That Stand To Lose The Most. (2018). https://www.forbes.com/sites/kathleenchaykowski/2018/03/06/facebooks-latest-algorithm-change-here-are-the-news-sites-that-stand-to-lose-the-most/?sh=74eef85134ec
- Yahui Chen. 2015. Convolutional neural network for sentence classification. Master’s thesis. University of Waterloo.
- Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).
- The echo chamber effect on social media. Proceedings of the National Academy of Sciences 118, 9 (2021), e2023301118.
- M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems. arXiv preprint arXiv:2205.08084 (2022).
- Terence Day. 2023. A Preliminary Investigation of Fake Peer-Reviewed Citations and References Generated by ChatGPT. The Professional Geographer (2023), 1–4.
- Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
- Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5). arXiv preprint arXiv:2203.13366 (2022).
- Learning to fake it: limited responses and fabricated references provided by ChatGPT for medical questions. medRxiv (2023), 2023–03.
- David Hamilton. [n. d.]. ChatGPT user in China detained for creating and spreading fake news, police say. https://apnews.com/article/chatgpt-china-deepfakes-criminal-detention-7985cf38ffa33b09d3ad4f8ea5299967
- A good prompt is worth millions of parameters? low-resource prompt-based learning for vision-language models. arXiv preprint arXiv:2110.08484 (2021).
- Personalized prompt learning for explainable recommendation. arXiv preprint arXiv:2202.07371 (2022).
- Towards personalized fairness based on causal notion. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1054–1063.
- Towards Better Representation Learning for Personalized News Recommendation: a Multi-Channel Deep Fusion Approach.. In IJCAI. 3805–3811.
- Is ChatGPT a Good Recommender? A Preliminary Study. arXiv preprint arXiv:2304.10149 (2023).
- Ramishah Maruf. 2023. Lawyer apologizes for fake court citations from ChatGPT. https://apnews.com/article/chatgpt-china-deepfakes-criminal-detention-7985cf38ffa33b09d3ad4f8ea5299967
- Exploring the filter bubble: the effect of using recommender systems on content diversity. In Proceedings of the 23rd international conference on World wide web. 677–686.
- Embedding-based news recommendation for millions of users. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 1933–1942.
- ProFairRec: Provider fairness-aware news recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1164–1173.
- Is ChatGPT a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476 (2023).
- Improving language understanding by generative pre-training. (2018).
- Partha Pratim Ray. 2023. ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems (2023).
- Opportunistic multi-aspect fairness through personalized re-ranking. In Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization. 239–247.
- Ralf C Staudemeyer and Eric Rothstein Morris. 2019. Understanding LSTM–a tutorial into long short-term memory recurrent neural networks. arXiv preprint arXiv:1909.09586 (2019).
- Attention is all you need. Advances in neural information processing systems 30 (2017).
- The spread of true and false news online. science 359, 6380 (2018), 1146–1151.
- Neural news recommendation with attentive multi-view learning. arXiv preprint arXiv:1907.05576 (2019).
- Neural news recommendation with topic-aware news representation. In Proceedings of the 57th Annual meeting of the association for computational linguistics. 1154–1159.
- Neural news recommendation with multi-head self-attention. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 6389–6394.
- Is News Recommendation a Sequential Recommendation Task?. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2382–2386.
- Fairness-aware news recommendation with decomposed adversarial learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 4462–4469.
- Mind: A large-scale dataset for news recommendation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 3597–3606.
- TFROM: A two-sided fairness-aware recommendation model for both customers and providers. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1013–1022.
- Language Models as Recommender Systems: Evaluations and Limitations. In I (Still) Can’t Believe It’s Not Better! NeurIPS 2021 Workshop.
- Exploring AI ethics of chatgpt: A diagnostic analysis. arXiv preprint arXiv:2301.12867 (2023).
- Xinyi Li (97 papers)
- Yongfeng Zhang (163 papers)
- Edward C. Malthouse (5 papers)