Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph (1801.10314v2)

Abstract: While conversing with chatbots, humans typically tend to ask many questions, a significant portion of which can be answered by referring to large-scale knowledge graphs (KG). While Question Answering (QA) and dialog systems have been studied independently, there is a need to study them closely to evaluate such real-world scenarios faced by bots involving both these tasks. Towards this end, we introduce the task of Complex Sequential QA which combines the two tasks of (i) answering factual questions through complex inferencing over a realistic-sized KG of millions of entities, and (ii) learning to converse through a series of coherently linked QA pairs. Through a labor intensive semi-automatic process, involving in-house and crowdsourced workers, we created a dataset containing around 200K dialogs with a total of 1.6M turns. Further, unlike existing large scale QA datasets which contain simple questions that can be answered from a single tuple, the questions in our dialogs require a larger subgraph of the KG. Specifically, our dataset has questions which require logical, quantitative, and comparative reasoning as well as their combinations. This calls for models which can: (i) parse complex natural language questions, (ii) use conversation context to resolve coreferences and ellipsis in utterances, (iii) ask for clarifications for ambiguous queries, and finally (iv) retrieve relevant subgraphs of the KG to answer such questions. However, our experiments with a combination of state of the art dialog and QA models show that they clearly do not achieve the above objectives and are inadequate for dealing with such complex real world settings. We believe that this new dataset coupled with the limitations of existing models as reported in this paper should encourage further research in Complex Sequential QA.

An Expert Analysis of Complex Sequential Question Answering over Knowledge Graphs

The paper "Complex Sequential Question Answering: Towards Learning to Converse Over Linked Question Answer Pairs with a Knowledge Graph" tackles a significant challenge in the field of artificial intelligence involving the integration of question answering (QA) systems with dialog systems over knowledge graphs. This research explores complex conversational scenarios where AI systems need to manage and infer information from linked question-answer pairs, leveraging large-scale knowledge graphs (KG) for contextually rich and coherent interactions.

Research Motivation and Dataset Creation

The paper originates from the intersection of two prominent AI tasks: QA and dialog systems. The researchers argue that real-world applications require systems capable of dealing with both tasks simultaneously, using a vast KG to answer sequential and context-dependent inquiries. To achieve this, they introduce the task of Complex Sequential Question Answering (CSQA), where systems must process factual questions, engage in dialog by maintaining conversation continuity, resolve ambiguities, and handle complex question structures.

A primary contribution of this work is the construction of a novel dataset designed for CSQA. This dataset comprises approximately 200,000 dialogs amounting to 1.6 million conversational turns, achieved through a semi-automated process that combined manual annotation with crowdsourced data creation. Unlike simpler existing QA datasets focused on individual tuples, the CSQA dataset demands logical, quantitative, and comparative reasoning over subgraphs of a KG. The creation process emphasizes generating both simple and complex questions, ensuring the inclusion of coreferencing, ellipses, and dialog specific intricacies, thereby reflecting real conversation dynamics.

Model Architecture and Challenges

The authors propose a hybrid model, leveraging contemporary neural architectures from dialog systems and QA methodologies. Specifically, they integrate a hierarchical recurrent encoder-decoder (HRED) with a key-value memory network to parse and answer complex, sequential questions. The model architecture is designed to capture the dialog context, handle large vocabularies, and effectively manage the candidate generation process for relevant KG tuples.

Despite these advancements, the research acknowledges that current state-of-the-art models exhibit insufficient capabilities in parsing complex questions and performing necessary logical and quantitative operations. The experimental findings illustrate marked discrepancies in model performance when comparing simpler direct questions to more complex and contextually linked questions, indicating a need for more sophisticated parsing and reasoning mechanisms in future models.

Implications and Future Directions

The findings and dataset provided by this paper hold substantial implications for the development of more advanced QA systems capable of engaging in genuine conversational interactions. The inadequacies of existing models underline several technical challenges that require further research. These include the development of explicit aggregation mechanisms for reasoning, enhanced candidate tuple generation for large KGs, and more structured memory networks to handle the complexity of interactions.

The authors suggest that future research should explore innovative solutions to improve model efficiency and effectiveness in real-time dialog settings. Such advancements could bridge existing gaps, facilitating the creation of AI systems with robust understanding and context management capabilities.

In conclusion, this paper signifies an important step toward sophisticated AI systems capable of complex multi-turn interactions involving large-scale knowledge graphs. The introduction of the CSQA task and corresponding dataset establishes a foundation for future explorations, potentially driving significant progress in the domain of conversational AI and its applications. Though substantial challenges remain, the directions set forth by this research are likely to catalyze further development and refinement of conversational models in AI.