BERT with History Answer Embedding for Conversational Question Answering (1905.05412v2)

Abstract: Conversational search is an emerging topic in the information retrieval community. One of the major challenges to multi-turn conversational search is to model the conversation history to answer the current question. Existing methods either prepend history turns to the current question or use complicated attention mechanisms to model the history. We propose a conceptually simple yet highly effective approach referred to as history answer embedding. It enables seamless integration of conversation history into a conversational question answering (ConvQA) model built on BERT (Bidirectional Encoder Representations from Transformers). We first explain our view that ConvQA is a simplified but concrete setting of conversational search, and then we provide a general framework to solve ConvQA. We further demonstrate the effectiveness of our approach under this framework. Finally, we analyze the impact of different numbers of history turns under different settings to provide new insights into conversation history modeling in ConvQA.

BERT with History Answer Embedding for Conversational Question Answering

The paper "BERT with History Answer Embedding for Conversational Question Answering" presents a novel approach to address the challenges inherent in conversational question answering (ConvQA). The primary focus of this work is on effectively modeling conversation history, a crucial aspect in achieving coherent and context-aware responses in multi-turn dialogues.

Overview

Conversational search, increasingly pivotal in the information retrieval domain, involves a system's ability to conduct multi-turn interactions to resolve user queries. Traditional methods either overly simplify this by concatenating conversation history with the new query or employ complex attention mechanisms that hinder computational efficiency. This paper introduces history answer embedding (HAE), a method integrated within a BERT-based architecture to incorporate conversation history more seamlessly and effectively.

Methodology

The authors propose a framework where ConvQA is treated as a simplified instance of conversational search. This involves handling conversation context akin to the machine comprehension problem but with the added complexity of sequential historical dialogue. The approach involves:

1. History Answer Embedding: Instead of prepending previous conversational turns to the current question—an approach that risks losing efficiency and context—the authors introduce embeddings indicating whether tokens are part of historical answers. This leverages BERT's capacity to encode complex contextual information without unnecessarily extending the input sequence.
2. BERT Integration: The model harnesses BERT’s transformer-based architecture, pre-trained on large text corpora, improving the understanding of bidirectional contexts and enabling robust handling of conversation turns.
3. Framework Application: The implementation includes a modular framework that separates history selection and history modeling, allowing flexibility and future enhancement possibilities.

Experimental Results

The paper evaluates its history modeling technique against competitive baselines on the QuAC dataset, a benchmark for information-seeking dialogues. Results assert that the proposed BERT + HAE model achieves an F1 score of 63.1 on the validation set, surpassing traditional attention models like BiDAF++ and even outperforming BERT variants that merely prepend history turns. Additionally, the efficiency of training is notably enhanced, requiring substantially less time compared to models like FlowQA.

Significance and Implications

The high performance of BERT + HAE in modeling multi-turn conversations signifies a leap forward in ConvQA, aligning system responses closer to human-like intelligence. This method not only outperforms existing conversational models but also offers significant improvements in computational efficiency. The innovative use of history embeddings within BERT's architecture provides a template for future expansions in dialogue systems and conversational AI applications.

Implications for Future Research:

• Advanced History Selection: Further advancements could include integrating AI-driven history selection mechanisms to determine the most relevant dialogue turns, reducing noise from irrelevant context.
• Complex Queries and Diverse Domains: While the current method shows strength in well-structured data like QuAC, adapting HAE to more diverse datasets or freeform conversational data poses an intriguing challenge.
• Cross-Task Adaptability: There is potential in exploring the adaptability of HAE in other NLP domains where understanding sequential dependencies or context over time is crucial.

In conclusion, this paper makes a significant contribution to the ConvQA landscape, presenting a methodology that is both elegant in its simplicity and potent in its application. The use of history answer embeddings within BERT stands out as an effective means to handle the intricacies of conversation history, paving the way for more nuanced and efficient conversational systems.