Key-Value Memory Networks for Directly Reading Documents (1606.03126v2)

Abstract: Directly reading documents and being able to answer questions from them is an unsolved challenge. To avoid its inherent difficulty, question answering (QA) has been directed towards using Knowledge Bases (KBs) instead, which has proven effective. Unfortunately KBs often suffer from being too restrictive, as the schema cannot support certain types of answers, and too sparse, e.g. Wikipedia contains much more information than Freebase. In this work we introduce a new method, Key-Value Memory Networks, that makes reading documents more viable by utilizing different encodings in the addressing and output stages of the memory read operation. To compare using KBs, information extraction or Wikipedia documents directly in a single framework we construct an analysis tool, WikiMovies, a QA dataset that contains raw text alongside a preprocessed KB, in the domain of movies. Our method reduces the gap between all three settings. It also achieves state-of-the-art results on the existing WikiQA benchmark.

• The paper introduces KV-MemNNs, a novel architecture that directly reads documents using key-value memory pairs to enhance QA performance.
• KV-MemNNs outperform traditional models by achieving 93.9% hits@1 on structured KBs and competitive results on raw documents.
• The approach narrows the gap between structured KBs and unstructured text, paving the way for improved document-based QA systems.

Key-Value Memory Networks for Directly Reading Documents

The paper "Key-Value Memory Networks for Directly Reading Documents" by Miller et al. examines the challenge of question answering (QA) through document reading. To bypass the intrinsic complexities associated with this task, the prevailing trend has been to rely on structured Knowledge Bases (KBs). However, KBs are inherently limited due to their fixed schemas and incompleteness, which raises the prospect of exploring document-based QA.

Methodology

The authors introduce Key-Value Memory Networks (KV-MemNNs) as a novel neural network architecture designed to bridge the gap between QA from structured KBs and unstructured text documents. KV-MemNNs leverage encoding strategies that differ between the key and value stages of the memory read process, facilitating more efficient retrieval and answer prediction.

The essence of KV-MemNNs lies in structuring memory as key-value pairs. During the addressing phase, the query interacts with the keys, while in the subsequent output phase, the corresponding values are processed. This dual encoding allows for leveraging complex transforms between keys and values, significantly improving the model's interpretative prowess.

Dataset: WikiMovies

To evaluate KV-MemNNs, the authors curated a new dataset, named WikiMovies. This dataset comprises around 100,000 QA pairs related to the movie domain and includes raw text from Wikipedia, a preprocessed KB from the Open Movie Database (OMDb), and an Information Extraction (IE) generated KB. This allows for a direct comparison of model performance across different knowledge representations. The dataset offers a substantial corpus for training machine learning models and allows for a thorough analysis of varying QA techniques.

Experimental Setup

The models were benchmarked across three settings: human-annotated KB (KB), information extraction-based KB (IE), and raw Wikipedia documents (Doc). The performance metric utilized was hits@1, representing the accuracy of the top answer.

Results

The experimental results presented in the paper highlight the superiority of KV-MemNNs across all three knowledge sources:

• KV-MemNNs vs. Baselines: KV-MemNNs achieve state-of-the-art results with a hits@1 of 93.9% on the KB, significantly outperforming other models like Supervised Embeddings and traditional Memory Networks (MemNNs).
• IE and Doc Performance: Despite the inherent complexities in parsing unstructured text, KV-MemNNs managed to perform competitively, attaining a hits@1 of 76.2% on the raw Wikipedia documents setting, which is notably higher than the IE-based KB performance of 68.3%.

Analysis and Insights

A detailed analysis indicates that structured KBs consistently yield better performance due to their organized nature, making information retrieval straightforward. Conversely, the documents setting, although less structured, offers a broader knowledge base. The inclusion of sophisticated encoding strategies such as incorporating window-level context and document titles substantially improved document comprehensibility, enhancing QA performance.

Moreover, the authors created synthetic documents from the KB using template sentences to simulate the complexity of documents. This experimental setup demonstrated that coreference resolution and conjunctions significantly impact performance, underlining the challenges of natural language understanding in QA systems.

Comparison with WikiQA

To further validate the versatility and efficacy of KV-MemNNs, the paper also presents experiments on the WikiQA dataset. KV-MemNNs attained state-of-the-art performance in this broader context as well, surpassing other attention-based models and achieving remarkable scores in both MAP and MRR metrics.

Implications and Future Work

The development of KV-MemNNs marks a significant step towards robust document-based QA systems. The flexible architecture of KV-MemNNs, enabling separate key and value encodings, enhances their applicability across varied QA tasks and domains. While the gap between document-based QA and KB-based QA narrows, future research must address remaining disparities. Enhancing coreference resolution and understanding the intricate dependencies within text remain critical areas of exploration.

Additionally, the potential of KV-MemNNs extends beyond QA, encompassing broader applications such as dialog systems and other contexts where memory-based reasoning is pivotal. Their adaptability to encode prior task-specific knowledge provides a strong foundation for evolving AI applications.

In summary, the paper provides a comprehensive framework for advancing document-based QA through KV-MemNNs, offering a robust methodology and thorough empirical evaluation that paves the way for future advancements in the field.