Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Navigating User Experience of ChatGPT-based Conversational Recommender Systems: The Effects of Prompt Guidance and Recommendation Domain (2405.13560v1)

Published 22 May 2024 in cs.HC and cs.AI

Abstract: Conversational recommender systems (CRS) enable users to articulate their preferences and provide feedback through natural language. With the advent of LLMs, the potential to enhance user engagement with CRS and augment the recommendation process with LLM-generated content has received increasing attention. However, the efficacy of LLM-powered CRS is contingent upon the use of prompts, and the subjective perception of recommendation quality can differ across various recommendation domains. Therefore, we have developed a ChatGPT-based CRS to investigate the impact of these two factors, prompt guidance (PG) and recommendation domain (RD), on the overall user experience of the system. We conducted an online empirical study (N = 100) by employing a mixed-method approach that utilized a between-subjects design for the variable of PG (with vs. without) and a within-subjects design for RD (book recommendations vs. job recommendations). The findings reveal that PG can substantially enhance the system's explainability, adaptability, perceived ease of use, and transparency. Moreover, users are inclined to perceive a greater sense of novelty and demonstrate a higher propensity to engage with and try recommended items in the context of book recommendations as opposed to job recommendations. Furthermore, the influence of PG on certain user experience metrics and interactive behaviors appears to be modulated by the recommendation domain, as evidenced by the interaction effects between the two examined factors. This work contributes to the user-centered evaluation of ChatGPT-based CRS by investigating two prominent factors and offers practical design guidance.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems. 1–13.
  2. Abdullah M Baabdullah. 2024. Generative conversational AI agent for managerial practices: The role of IQ dimensions, novelty seeking and ethical concerns. Technological Forecasting and Social Change 198 (2024), 122951.
  3. Bradford S Bell and Steve WJ Kozlowski. 2002. Adaptive guidance: Enhancing self-regulation, knowledge, and performance in technology-based training. Personnel Psychology 55, 2 (2002), 267–306.
  4. ChatGPT Hallucinates Non-existent Citations: Evidence from Economics. The American Economist 69, 1 (2024), 80–87.
  5. Impacts of personal characteristics on user trust in conversational recommender systems. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–14.
  6. Is more always better? The effects of personal characteristics and level of detail on the perception of explanations in a recommender system. In Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization. 254–264.
  7. Human decision making and recommender systems. ACM Transactions on Interactive Intelligent Systems (TiiS) 3, 3 (2013), 1–7.
  8. Avishek Choudhury and Zaira Chaudhry. 2024. Large Language Models and User Trust: Consequence of Self-Referential Learning Loop and the Deskilling of Health Care Professionals. Journal of Medical Internet Research 26 (2024), e56764.
  9. Uncovering chatgpt’s capabilities in recommender systems. In Proceedings of the 17th ACM Conference on Recommender Systems. 1126–1132.
  10. Yashar Deldjoo. 2024. Understanding Biases in ChatGPT-based Recommender Systems: Provider Fairness, Temporal Stability, and Recency. arXiv preprint arXiv:2401.10545 (2024).
  11. Evaluating chatgpt as a recommender system: A rigorous approach. arXiv preprint arXiv:2309.03613 (2023).
  12. Sabit Ekin. 2023. Prompt engineering for ChatGPT: a quick guide to techniques, tips, and best practices. Authorea Preprints (2023).
  13. Developing and evaluating a university recommender system. Frontiers in Artificial Intelligence 4 (2022), 796268.
  14. Stefano Filippi. 2023. Measuring the impact of ChatGPT on fostering concept generation in innovative product design. Electronics 12, 16 (2023), 3535.
  15. Advances and challenges in conversational recommender systems: A survey. AI open 2 (2021), 100–126.
  16. Chat-rec: Towards interactive and explainable llms-augmented recommender system. arXiv preprint arXiv:2303.14524 (2023).
  17. Carlos A Gomez-Uribe and Neil Hunt. 2015. The netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS) 6, 4 (2015), 1–19.
  18. Explaining and exploring job recommendations: a user-driven approach for interacting with knowledge-based job recommender systems. In Proceedings of the 13th ACM Conference on Recommender Systems. 60–68.
  19. Leah Henrickson and Albert Meroño-Peñuela. 2023. Prompting meaning: a hermeneutic approach to optimising prompt engineering with ChatGPT. AI & SOCIETY (2023), 1–16.
  20. Human decision making and recommender systems. Recommender systems handbook (2015), 611–648.
  21. A survey on conversational recommender systems. ACM Computing Surveys (CSUR) 54, 5 (2021), 1–36.
  22. Key qualities of conversational recommender systems: From users’ perspective. In Proceedings of the 9th International Conference on Human-Agent Interaction. 93–102.
  23. CRS-Que: A user-centric evaluation framework for conversational recommender systems. ACM Transactions on Recommender Systems 2, 1 (2024), 1–34.
  24. Effects of personal characteristics on music recommender systems with different levels of controllability. In Proceedings of the 12th ACM Conference on Recommender Systems. 13–21.
  25. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and individual differences 103 (2023), 102274.
  26. Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking. In Proceedings of the 29th International Conference on Intelligent User Interfaces. 288–303.
  27. Decisions with ChatGPT: Reexamining choice overload in ChatGPT recommendations. Journal of Retailing and Consumer Services 75 (2023), 103494.
  28. Each to his own: how different users call for different interaction methods in recommender systems. In Proceedings of the fifth ACM conference on Recommender systems. 141–148.
  29. Explaining the user experience of recommender systems. User modeling and user-adapted interaction 22 (2012), 441–504.
  30. Joseph A Konstan and John Riedl. 2012. Recommender systems: from algorithms to user experience. User modeling and user-adapted interaction 22 (2012), 101–123.
  31. Artificial intelligence prompt engineering as a new digital competence: Analysis of generative AI technologies such as ChatGPT. Entrepreneurial Business and Economics Review 11, 3 (2023), 25–37.
  32. High stakes decision making: Normative, descriptive and prescriptive considerations. Marketing Letters 13 (2002), 259–268.
  33. Few-shot is enough: exploring ChatGPT prompt engineering method for automatic question generation in english education. Education and Information Technologies (2023), 1–33.
  34. A preliminary study of chatgpt on news recommendation: Personalization, provider fairness, fake news. arXiv preprint arXiv:2306.10702 (2023).
  35. BookGPT: A General Framework for Book Recommendation Empowered by Large Language Model. Electronics 12, 22 (2023), 4654.
  36. Is chatgpt a good recommender? a preliminary study. arXiv preprint arXiv:2304.10149 (2023).
  37. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Comput. Surveys 55, 9 (2023), 1–35.
  38. To explain or not to explain: the effects of personal characteristics when explaining music recommendations. In Proceedings of the 24th international conference on intelligent user interfaces. 397–407.
  39. Patterns for how users overcome obstacles in voice user interfaces. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–7.
  40. Anam Nazir and Ze Wang. 2023. A comprehensive survey of ChatGPT: Advancements, applications, prospects, and challenges. Meta-radiology (2023), 100022.
  41. Pearl Pu and Li Chen. 2007. Trust-inspiring explanation interfaces for recommender systems. Knowledge-Based Systems 20, 6 (2007), 542–556.
  42. A user-centric evaluation framework for recommender systems. In Proceedings of the fifth ACM conference on Recommender systems. 157–164.
  43. Francesco Ricci. 2010. Mobile recommender systems. Information Technology & Tourism 12, 3 (2010), 205–231.
  44. Ben Shneiderman. 2022. Human-centered AI. Oxford University Press.
  45. Leveraging ChatGPT for Automated Human-centered Explanations in Recommender Systems. In Proceedings of the 29th International Conference on Intelligent User Interfaces. 597–608.
  46. ChatGPT for Conversational Recommendation: Refining Recommendations by Reprompting with Feedback. arXiv preprint arXiv:2401.03605 (2024).
  47. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI conference on human factors in computing systems. 143–146.
  48. Prompting Large Language Models for Recommender Systems: A Comprehensive Framework and Empirical Analysis. arXiv preprint arXiv:2401.04997 (2024).
  49. Dongning Yan and Li Chen. 2023. The influence of personality traits on user interaction with recommendation interfaces. ACM Transactions on Interactive Intelligent Systems 13, 1 (2023), 1–39.
  50. Toward a user-oriented recommendation system for real estate websites. Information systems 38, 2 (2013), 231–243.
  51. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21.
  52. Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems. 993–999.
  53. Auralist: introducing serendipity into music recommendation. In Proceedings of the fifth ACM international conference on Web search and data mining. 13–22.
  54. Recommender Systems in the Era of Large Language Models (LLMs). IEEE Transactions on Knowledge and Data Engineering (2024).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Yizhe Zhang (127 papers)
  2. Yucheng Jin (11 papers)
  3. Li Chen (590 papers)
  4. Ting Yang (40 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets