Asking Clarifying Questions in Open-Domain Information-Seeking Conversations (1907.06554v1)

Abstract: Users often fail to formulate their complex information needs in a single query. As a consequence, they may need to scan multiple result pages or reformulate their queries, which may be a frustrating experience. Alternatively, systems can improve user satisfaction by proactively asking questions of the users to clarify their information needs. Asking clarifying questions is especially important in conversational systems since they can only return a limited number of (often only one) result(s). In this paper, we formulate the task of asking clarifying questions in open-domain information-seeking conversational systems. To this end, we propose an offline evaluation methodology for the task and collect a dataset, called Qulac, through crowdsourcing. Our dataset is built on top of the TREC Web Track 2009-2012 data and consists of over 10K question-answer pairs for 198 TREC topics with 762 facets. Our experiments on an oracle model demonstrate that asking only one good question leads to over 170% retrieval performance improvement in terms of P@1, which clearly demonstrates the potential impact of the task. We further propose a retrieval framework consisting of three components: question retrieval, question selection, and document retrieval. In particular, our question selection model takes into account the original query and previous question-answer interactions while selecting the next question. Our model significantly outperforms competitive baselines. To foster research in this area, we have made Qulac publicly available.

Overview of Clarifying Questions in Open-Domain Information-Seeking Conversational Systems

The research presented in "Asking Clarifying Questions in Open-Domain Information-Seeking Conversations" addresses the complexities involved in conversational systems when users are unable to express their information needs effectively in a single query. This work is pivotal in refining systems that assist users in navigating ambiguous or multi-faceted queries by incorporating clarifying questions to enhance retrieval performance significantly.

Core Contributions and Results

The paper introduces a methodology to evaluate systems that ask clarifying questions in open-domain settings. It is supported by a novel dataset, Qulac, derived from the TREC Web track 2009-2012 data, consisting of over 10,000 question-answer pairs. By implementing an oracle model, the authors illustrate a potential retrieval performance improvement of over 170% in P@1 when introducing clarifying questions. This marked enhancement underlines the considerable value of incorporating clarifying questions in retrieval frameworks.

Specifically, the retrieval framework devised consists of three crucial modules: question retrieval, question selection, and document retrieval. Within this architecture, a neural model is employed for question selection, utilizing the initial query and conversation history to effectively pinpoint subsequent clarifications. The model notably outperforms competitive baselines, substantiating its efficacy.

Implications and Future Directions

The research underscores the dual benefits of enhanced retrieval performance and improved user satisfaction through clarifying questions. Practically, it provides conversational systems with a structured approach to refining responses based on nuanced user interactions. Theoretical implications suggest a reconsideration of traditional query-response models, advocating for more dynamic and context-sensitive interaction handling in information retrieval.

Future directions in AI could include advancements in personalizing clarification processes, thus achieving a balance between probing questions and predictive relevance assessments. Extending the methodology to multi-turn scenarios and further refining question generation in real-time conversational settings pose intriguing challenges and opportunities.

In conclusion, this work lays a foundational approach to an often-overlooked aspect of conversational AI, inviting further exploration and expansion to accommodate the evolving complexity of user interactions in information retrieval contexts.