Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information (1805.04655v2)

Abstract: Inquiry is fundamental to communication, and machines cannot effectively collaborate with humans unless they can ask questions. In this work, we build a neural network model for the task of ranking clarification questions. Our model is inspired by the idea of expected value of perfect information: a good question is one whose expected answer will be useful. We study this problem using data from StackExchange, a plentiful online resource in which people routinely ask clarifying questions to posts so that they can better offer assistance to the original poster. We create a dataset of clarification questions consisting of ~77K posts paired with a clarification question (and answer) from three domains of StackExchange: askubuntu, unix and superuser. We evaluate our model on 500 samples of this dataset against expert human judgments and demonstrate significant improvements over controlled baselines.

Overview of Ranking Clarification Questions using Neural EVPI

In "Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information," the authors introduce a novel approach to the problem of ranking clarification questions using a neural network model inspired by the Expected Value of Perfect Information (EVPI). The research is motivated by the need for machines to inquire effectively, filling information gaps essential for collaboration with humans.

Methodology

The paper proposes a decision-theoretic framework for question utility, drawing on EVPI to rank candidate questions based on the expected utility of their answers. This is implemented in a neural network architecture that calculates both the likelihood of potential answers and their utility in enhancing the information content of a given post. The authors utilize StackExchange as a data resource, gathering $\sim$77K posts with associated clarification questions and answers across three domains: askubuntu, unix, and superuser. The model's performance is evaluated using expert human judgments and is shown to outperform various baselines.

Key Contributions

This work offers two major contributions to the field:

1. Neural Network Model: The presentation of a neural network model uniquely structured to utilize EVPI principles for ranking clarification questions.
2. Dataset Creation: The construction and release of a specialized dataset derived from StackExchange, designed to facilitate the learning of clarification question-asking patterns.

These contributions significantly advance the goal of automatically identifying effective clarification questions within user-generated content, a task critical to enhancing dialogue systems and AI communication platforms.

Results

Significant findings indicate that the EVPI-based model improves upon neural baselines that do not leverage this framework. The model showed marked superiority in precision metrics, achieving a higher Mean Average Precision and outperforming previous work such as the Community QA approach. The approach revealed that incorporating the potential utility of answers, alongside their likelihood, is beneficial in selecting appropriate questions.

Implications and Future Directions

The implications of this research extend to AI systems that interact with human users, as the ability to autonomously seek missing information enhances dialogues' coherence and efficacy. Future work should explore integrating EVPI into reinforcement learning frameworks to handle dialogues spanning multiple turns. Additionally, a transition towards question generation may require sequence-to-sequence models for improved template-driven inquiries, ultimately aiming to develop systems capable of both ranking and generating questions autonomously.

This paper's contributions set a substantial precedent for harnessing decision-theoretic frameworks in neural model architectures, emphasizing the importance of pragmatic inquiry in AI applications. Researchers are encouraged to further explore and refine these models, potentially fostering systems that more accurately emulate human question-asking behavior.