Analyzing Coherency in Facet-based Clarification Prompt Generation for Search (2401.04524v1)
Abstract: Clarifying user's information needs is an essential component of modern search systems. While most of the approaches for constructing clarifying prompts rely on query facets, the impact of the quality of the facets is relatively unexplored. In this work, we concentrate on facet quality through the notion of facet coherency and assess its importance for overall usefulness for clarification in search. We find that existing evaluation procedures do not account for facet coherency, as evident by the poor correlation of coherency with automated metrics. Moreover, we propose a coherency classifier and assess the prevalence of incoherent facets in a well-established dataset on clarification. Our findings can serve as motivation for future work on the topic.
- Asking clarifying questions in open-domain information-seeking conversations. In Proceedings of the 42nd international acm sigir conference on research and development in information retrieval, pages 475–484, 2019.
- A trinomial test for paired data when there are many ties. Mathematics and Computers in Simulation, 81(6):1153–1160, 2011.
- Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
- Learning multiple intent representations for search queries. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 669–679, 2021.
- Extracting query facets from search results. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval, pages 93–102, 2013.
- Precision-oriented query facet extraction. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pages 1433–1442, 2016.
- Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
- Towards facet-driven generation of clarifying questions for conversational search. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval, pages 167–175, 2021.
- Mimics-duo: Offline & online evaluation of search clarification. In SIGIR, pages 3198–3208. ACM, 2022.
- Huggingface’s transformers: State-of-the-art natural language processing, 2019.
- Generating clarifying questions for information retrieval. In Proceedings of the web conference 2020, pages 418–428, 2020a.
- Mimics: A large-scale data collection for search clarification. In Proceedings of the 29th ACM international conference on information & knowledge management, pages 3189–3196, 2020b.