Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (1803.05457v1)

Abstract: We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.

Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge," is a well-constructed paper presenting a significant benchmark dataset aimed at advancing question-answering (QA) research. Authored by Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord from the Allen Institute for Artificial Intelligence, the paper introduces the AI2 Reasoning Challenge (ARC) dataset to push the boundaries of current QA systems beyond simple retrieval tasks and co-occurrence methods.

The ARC dataset comprises 7,787 grade-school science questions divided into two subsets: the Challenge Set and the Easy Set. The Challenge Set consists of 2,590 questions that were incorrectly answered by both a retrieval-based algorithm and a word co-occurrence algorithm. The Easy Set contains the remaining 5,197 questions. Importantly, the ARC dataset exclusively features natural, grade-school science questions, making it the largest public-domain collection of its kind.

Datasets and Methodology

The primary aim of ARC is to shift focus from surface-level cue answering to questions requiring deeper reasoning and knowledge-type inference. The authors underscore the limitations of existing datasets, such as SQuAD and SNLI, which primarily cater to retrieval-style QA systems. These systems often succeed by leveraging surface-level word matching and do not adequately test advanced reasoning capabilities.

Notably, the ARC dataset includes questions similar to standardized tests, a format that engages a broad spectrum of linguistic and inferential challenges, varying significantly across difficulty levels. Examples of such questions include:

• Challenge Example: "A student riding a bicycle observes that it moves faster on a smooth road than on a rough road. This happens because the smooth road has (A) less gravity (B) more gravity (C) less friction correct more friction."
• Easy Example: "Which property of a mineral can be determined just by looking at it? (A) luster correct mass (C) weight (D) hardness."

Baseline Performance

The paper evaluates the ARC Challenge Set using several baseline methods, including prominent neural models like Decomposable Attention (DecompAttn), Bidirectional Attention Flow (BiDAF), and the Decomposed Graph Entailment Model (DGEM). Surprisingly, even the most advanced models failed to significantly outperform the random baseline on the Challenge Set, while they performed notably better on the Easy Set, with scores reaching up to 61%.

Robust Documentation and Supporting Resources

To facilitate further research, the authors have also released the ARC Corpus, a complementary dataset containing 14 million science-related sentences. This corpus is instrumental for tasks requiring large knowledge bases and advanced text comprehension.

Additionally, the authors have provided implementations for three baseline neural models—DecompAttn, BiDAF, and DGEM—aiming to support the community in developing more sophisticated approaches to tackling the ARC Challenge. They emphasize that current models perform well on the Easy Set but struggle significantly on the Challenge Set, highlighting the latter's difficulty.

Implications and Future Directions

The ARC dataset serves as a salient benchmark for future QA research, offering considerable challenges that demand novel methodologies in natural language understanding, reasoning, and knowledge inference. By segmenting questions into Easy and Challenge Sets, the ARC dataset encourages the development of more nuanced and capable QA systems. This structure exposes the limitations of straightforward retrieval-based methods and pushes for advancements involving commonsense knowledge, multi-step reasoning, and more sophisticated comprehension techniques.

The implications for AI development are both practical and theoretical. Practically, advancements inspired by the ARC dataset could lead to more robust and intelligent QA systems, applicable in education, support systems, and various analytical tasks. Theoretically, successful approaches to ARC could unveil new paradigms in machine reasoning, potentially influencing future AI research directions.

Conclusion

The ARC dataset poses a significant challenge to the AI community, aiming to drive research beyond the capabilities of current state-of-the-art QA models. By addressing both commonsense reasoning and advanced text comprehension, ARC serves as a pivotal resource for pushing the boundaries of QA research. Researchers are encouraged to engage with this comprehensive dataset and explore innovative approaches to overcome the presented challenges.