Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Using LLMs to Investigate Correlations of Conversational Follow-up Queries with User Satisfaction (2407.13166v1)

Published 18 Jul 2024 in cs.HC and cs.IR

Abstract: With LLMs, conversational search engines shift how users retrieve information from the web by enabling natural conversations to express their search intents over multiple turns. Users' natural conversation embodies rich but implicit signals of users' search intents and evaluation of search results to understand user experience with the system. However, it is underexplored how and why users ask follow-up queries to continue conversations with conversational search engines and how the follow-up queries signal users' satisfaction. From qualitative analysis of 250 conversational turns from an in-lab user evaluation of Naver Cue:, a commercial conversational search engine, we propose a taxonomy of 18 users' follow-up query patterns from conversational search, comprising two major axes: (1) users' motivations behind continuing conversations (N = 7) and (2) actions of follow-up queries (N = 11). Compared to the existing literature on query reformulations, we uncovered a new set of motivations and actions behind follow-up queries, including asking for subjective opinions or providing natural language feedback on the engine's responses. To analyze conversational search logs with our taxonomy in a scalable and efficient manner, we built an LLM-powered classifier (73% accuracy). With our classifier, we analyzed 2,061 conversational tuples collected from real-world usage logs of Cue: and examined how the conversation patterns from our taxonomy correlates with satisfaction. Our initial findings suggest some signals of dissatisfactions, such as Clarifying Queries, Excluding Condition, and Substituting Condition with follow-up queries. We envision our approach could contribute to automated evaluation of conversation search experience by providing satisfaction signals and grounds for realistic user simulations.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Conversational Interfaces for Information Search, in: W. T. Fu, H. van Oostendorp (Eds.), Understanding and Improving Information Search: A Cognitive Approach, Human–Computer Interaction Series, Springer International Publishing, Cham, 2020, pp. 267–287. URL: https://doi.org/10.1007/978-3-030-38825-6_13. doi:10.1007/978-3-030-38825-6_13.
  2. Large-scale validation and analysis of interleaved search evaluation, ACM Transactions on Information Systems 30 (2012) 6:1–6:41. URL: https://doi.org/10.1145/2094072.2094078. doi:10.1145/2094072.2094078.
  3. Beyond clicks: query reformulation as a predictor of search satisfaction, in: Proceedings of the 22nd ACM international conference on Information & Knowledge Management, CIKM ’13, Association for Computing Machinery, New York, NY, USA, 2013, pp. 2019–2028. URL: https://doi.org/10.1145/2505515.2505682. doi:10.1145/2505515.2505682.
  4. Patterns and transitions of query reformulation during web searching, International Journal of Web Information Systems 3 (2007) 328–340. URL: https://doi.org/10.1108/17440080710848116. doi:10.1108/17440080710848116, publisher: Emerald Group Publishing Limited.
  5. J. Huang, E. N. Efthimiadis, Analyzing and evaluating query reformulation strategies in web search logs, in: Proceedings of the 18th ACM conference on Information and knowledge management, CIKM ’09, Association for Computing Machinery, New York, NY, USA, 2009, pp. 77–86. URL: https://doi.org/10.1145/1645953.1645966. doi:10.1145/1645953.1645966.
  6. Leaving so soon? understanding and predicting web search abandonment rationales, in: Proceedings of the 21st ACM international conference on Information and knowledge management, CIKM ’12, Association for Computing Machinery, New York, NY, USA, 2012, pp. 1025–1034. URL: https://doi.org/10.1145/2396761.2398399. doi:10.1145/2396761.2398399.
  7. Online Evaluation for Information Retrieval, Foundations and Trends® in Information Retrieval 10 (2016) 1–117. URL: http://www.nowpublishers.com/article/Details/INR-051. doi:10.1561/1500000051.
  8. Patterns for How Users Overcome Obstacles in Voice User Interfaces, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, Association for Computing Machinery, New York, NY, USA, 2018, pp. 1–7. URL: https://dl.acm.org/doi/10.1145/3173574.3173580. doi:10.1145/3173574.3173580.
  9. A. See, C. Manning, Understanding and predicting user dissatisfaction in a neural generative chatbot, in: H. Li, G.-A. Levow, Z. Yu, C. Gupta, B. Sisman, S. Cai, D. Vandyke, N. Dethlefs, Y. Wu, J. J. Li (Eds.), Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics, Singapore and Online, 2021, pp. 1–12. URL: https://aclanthology.org/2021.sigdial-1.1. doi:10.18653/v1/2021.sigdial-1.1.
  10. Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems, in: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM ’19, Association for Computing Machinery, New York, NY, USA, 2019, pp. 1281–1290. URL: https://dl.acm.org/doi/10.1145/3357384.3358047. doi:10.1145/3357384.3358047.
  11. Automatic online evaluation of intelligent assistants, in: Proceedings of the 24th International Conference on World Wide Web, WWW ’15, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 2015, p. 506–516. URL: https://doi.org/10.1145/2736277.2741669. doi:10.1145/2736277.2741669.
  12. Qrfa: A data-driven model of information seeking dialogues, in: Advances in Information Retrieval, Springer International Publishing, 2019, pp. 541–557.
  13. Analyzing and characterizing user intent in information-seeking conversations, in: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 989–992. URL: https://doi.org/10.1145/3209978.3210124. doi:10.1145/3209978.3210124.
  14. Conceptualizing agent-human interactions during the conversational search process, 2018. The Second International Workshop on Conversational Approaches to Information Retrieval, CAIR ; Conference date: 12-07-2018 Through 12-07-2018.
  15. F. Radlinski, N. Craswell, A Theoretical Framework for Conversational Search, in: Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, ACM, Oslo Norway, 2017, pp. 117–126. URL: https://dl.acm.org/doi/10.1145/3020165.3020183. doi:10.1145/3020165.3020183.
  16. An exploration of reasons for query reformulations, Proceedings of the Association for Information Science and Technology 54 (2017) 337–346. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/pra2.2017.14505401037. doi:10.1002/pra2.2017.14505401037.
  17. A comparative analysis of the information-seeking behavior of visually impaired and sighted searchers, Journal of the American Society for Information Science and Technology 63 (2012) 377–391.
  18. Is exploratory search different? A comparison of information search behavior for exploratory and lookup tasks, Journal of the Association for Information Science and Technology 67 (2016) 2635–2651. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/asi.23617. doi:10.1002/asi.23617.
  19. Patterns of query reformulation during web searching, J. Am. Soc. Inf. Sci. Technol. 60 (2009) 1358–1371.
  20. Exploring the relationships between search intentions and query reformulations, Proceedings of the Association for Information Science and Technology 53 (2016) 1–9. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/pra2.2016.14505301048. doi:10.1002/pra2.2016.14505301048.
  21. S. Y. Rieh, H. I. Xie, Analysis of multiple query reformulations on the web: The interactive information retrieval context, Information Processing & Management 42 (2006) 751–768. URL: https://www.sciencedirect.com/science/article/pii/S030645730500066X. doi:10.1016/j.ipm.2005.05.005.
  22. How do users respond to voice input errors? lexical and phonetic query reformulation in voice search, in: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, 2013, pp. 143–152.
  23. Mining broad latent query aspects from search sessions, 2009, pp. 867–876. doi:10.1145/1557019.1557114.
  24. Resilient chatbots: Repair strategy preferences for conversational breakdowns, in: Proceedings of the 2019 CHI conference on human factors in computing systems, 2019, pp. 1–12.
  25. An intent taxonomy for questions asked in web search, in: Proceedings of the 2021 Conference on Human Information Interaction and Retrieval, 2021, pp. 85–94.
  26. Query reformulation mining: models, patterns, and applications, Information retrieval 14 (2011) 257–289.
  27. Generating clarifying questions for information retrieval, in: Proceedings of The Web Conference 2020, WWW ’20, Association for Computing Machinery, New York, NY, USA, 2020, p. 418–428. URL: https://doi.org/10.1145/3366423.3380126. doi:10.1145/3366423.3380126.
  28. Informing the Design of Spoken Conversational Search: Perspective Paper, in: Proceedings of the 2018 Conference on Human Information Interaction & Retrieval, CHIIR ’18, Association for Computing Machinery, New York, NY, USA, 2018, pp. 32–41. URL: https://dl.acm.org/doi/10.1145/3176349.3176387. doi:10.1145/3176349.3176387.
  29. Llm-assisted content analysis: Using large language models to support deductive coding, arXiv preprint arXiv:2306.14924 (2023).
  30. Supporting qualitative analysis with large language models: Combining codebook with gpt-3 for deductive coding, in: Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, 2023, pp. 75–78.
  31. Query recommendation using query logs in search engines, in: International conference on extending database technology, Springer, 2004, pp. 588–596.
  32. Context-driven interactive query simulations based on generative large language models, in: N. Goharian, N. Tonellotto, Y. He, A. Lipani, G. McDonald, C. Macdonald, I. Ounis (Eds.), Advances in Information Retrieval, Springer Nature Switzerland, Cham, 2024, pp. 173–188.
  33. Exploiting simulated user feedback for conversational search: Ranking, rewriting, and beyond, in: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, Association for Computing Machinery, New York, NY, USA, 2023, p. 632–642. URL: https://doi.org/10.1145/3539618.3591683. doi:10.1145/3539618.3591683.
  34. Evaluating mixed-initiative conversational search systems via user simulation, in: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM ’22, Association for Computing Machinery, New York, NY, USA, 2022, p. 888–896. URL: https://doi.org/10.1145/3488560.3498440. doi:10.1145/3488560.3498440.
  35. E. Svikhnushina, P. Pu, Approximating online human evaluation of social chatbots with prompting, in: S. Stoyanchev, S. Joty, D. Schlangen, O. Dusek, C. Kennington, M. Alikhani (Eds.), Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Association for Computational Linguistics, Prague, Czechia, 2023, pp. 268–281. URL: https://aclanthology.org/2023.sigdial-1.25. doi:10.18653/v1/2023.sigdial-1.25.
  36. Usersimcrs: A user simulation toolkit for evaluating conversational recommender systems, in: Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM ’23, Association for Computing Machinery, New York, NY, USA, 2023, p. 1160–1163. URL: https://doi.org/10.1145/3539597.3573029. doi:10.1145/3539597.3573029.
  37. Using large language models for qualitative analysis can introduce serious bias, arXiv preprint arXiv:2309.17147 (2023).
  38. Collabcoder: A gpt-powered workflow for collaborative qualitative analysis, in: Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing, CSCW ’23 Companion, Association for Computing Machinery, New York, NY, USA, 2023, p. 354–357. URL: https://doi.org/10.1145/3584931.3607500. doi:10.1145/3584931.3607500.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Hyunwoo Kim (52 papers)
  2. Yoonseo Choi (2 papers)
  3. Taehyun Yang (4 papers)
  4. Honggu Lee (3 papers)
  5. Chaneon Park (1 paper)
  6. Yongju Lee (3 papers)
  7. Jin Young Kim (20 papers)
  8. Juho Kim (56 papers)
Citations (3)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets