SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine (1704.05179v3)

Abstract: We publicly release a new large-scale dataset, called SearchQA, for machine comprehension, or question-answering. Unlike recently released datasets, such as DeepMind CNN/DailyMail and SQuAD, the proposed SearchQA was constructed to reflect a full pipeline of general question-answering. That is, we start not from an existing article and generate a question-answer pair, but start from an existing question-answer pair, crawled from J! Archive, and augment it with text snippets retrieved by Google. Following this approach, we built SearchQA, which consists of more than 140k question-answer pairs with each pair having 49.6 snippets on average. Each question-answer-context tuple of the SearchQA comes with additional meta-data such as the snippet's URL, which we believe will be valuable resources for future research. We conduct human evaluation as well as test two baseline methods, one simple word selection and the other deep learning based, on the SearchQA. We show that there is a meaningful gap between the human and machine performances. This suggests that the proposed dataset could well serve as a benchmark for question-answering.

An Overview of SearchQA: A Comprehensive Dataset for Machine Comprehension in Question-Answering

The paper introduces SearchQA, a large-scale dataset explicitly designed for machine comprehension in the context of open-domain question-answering (QA). Distinguished from other existing datasets such as DeepMind's CNN/DailyMail and Stanford's SQuAD, SearchQA offers a full pipeline representation for QA. This dataset utilizes real-world questions sourced from the J! Archive and augments them with supplementary textual snippets obtained through queries to Google search, a feature that uniquely infuses realistic noise and variability into the dataset.

Dataset Construction and Characteristics

SearchQA begins with existing question-answer pairs extracted from the J! Archive. These questions are then used to query Google, obtaining additional text snippets to provide context for answering the questions. This approach results in a dataset comprising over 140,000 question-answer pairs, each associated with approximately 50 text snippets on average. Notably, the dataset includes supplementary metadata, such as snippet URLs, potentially beneficial for extended research applications.

The SearchQA dataset design aims to bridge the gap between traditional closed-world QA datasets and the demands of open-domain QA systems. Unlike prior datasets that guarantee well-curated contexts, SearchQA incorporates noise through the search-generated snippets, mimicking the challenges a generalized automatic QA system would face when deriving answers from a web search's less structured and potentially irrelevant snippets.

Evaluation and Benchmarking

To validate the dataset's efficacy, the authors conducted both human evaluations and baseline machine learning tests. Two fundamental approaches are tested: a simplistic word-selection algorithm and a more sophisticated deep learning model, the Attention Sum Reader (ASR). The human evaluation was conducted with participants providing answers in a limited time frame, revealing a significant performance gap between human subjects and machine models. The ASR outperforms baseline TF-IDF methods, offering a structured reference benchmark for further development.

The ASR's results demonstrate that while existing methodologies can process the SearchQA dataset effectively, significant room remains for advancement. The performance discrepancies highlight the complexity and challenge level SearchQA introduces for more robust machine comprehension systems.

Implications and Future Work

The implications of SearchQA stretch both into the practical and theoretical realms of AI and NLP research. By creating a testbed that simulates realistic information retrieval conditions, SearchQA is a pivotal resource for constructing more adept QA systems capable of handling noise and ambiguity inherent in open-domain sources. Contingent on this development, future research can concentrate on refining algorithms that improve contextual understanding and retrieval accuracy, thereby enhancing automated systems' competencies.

As researchers continue to build upon SearchQA, there is an opportunity to explore new relationships between search engine algorithms and QA performance, along with the potential to innovate in areas such as snippet extraction, context understanding, and multi-sentence reasoning within machine comprehension systems. This work opens a path for forthcoming studies to compare SearchQA with other datasets like MS MARCO, fostering a more profound understanding of search engine dynamics and its influence on QA models.

By releasing SearchQA to the public domain, the authors anticipate spurring progress in the QA field. This dataset provides a foundational toolset for evaluating machine comprehension systems within an environment that mirrors practical scenarios, encouraging the evolution of more intelligent and adaptive QA technologies.