Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2101.02235v1)

Abstract: A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce StrategyQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, StrategyQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in StrategyQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well (87%) on this task, while our best baseline reaches an accuracy of $\sim$66%.

• The paper presents StrategyQA, a benchmark of 2,780 Boolean questions that require multi-step implicit reasoning without explicit guidance.
• It uses an innovative crowdsourcing pipeline to decompose questions into reasoning steps, linking each to corresponding Wikipedia evidence.
• Baseline models, achieving around 63.6% accuracy compared to 87% for humans, underscore the need for advanced strategies in retrieval and reasoning.

StrategyQA: A Benchmark for Implicit Multi-Hop Reasoning

The paper presents "StrategyQA," a meticulously curated question answering (QA) benchmark designed to address the limitations intrinsic to existing multi-hop reasoning datasets. Unlike traditional QA datasets where questions explicitly outline the reasoning steps needed for resolution, StrategyQA introduces questions where reasoning must be implicitly inferred. This novel approach necessitates the development of more sophisticated reasoning models capable of strategy inference.

Key Contributions

The authors have proposed a new way of generating strategy questions that involve multi-step implicit reasoning. Specifically, StrategyQA questions require models to infer a strategy from the question itself without explicitly defined reasoning steps. This setup adds a layer of complexity that closely aligns with challenges faced in real-world scenarios.

The dataset comprises 2,780 Boolean questions, each annotated with a strategic decomposition into reasoning steps and corresponding evidence paragraphs sourced from Wikipedia. This structured annotation facilitates both the understanding of the reasoning process and the verification of retrieved contextual information.

Methodology

The StrategyQA dataset was crafted through an innovative crowdsourcing pipeline, designed to ensure diversity and creativity in questions:

1. Creative Question Writing: Annotators were primed with minimal context from Wikipedia while controlling answer distribution and implementing adversarial filtering with model-in-the-loop techniques. This method ensured that only questions necessitating genuine multi-step implicit reasoning were included.
2. Question Decomposition: Annotators decomposed questions into a sequence of steps, each representing a necessary reasoning component. Steps were annotated with expected Wikipedia titles containing the necessary information, guiding further data collection.
3. Evidence Matching: For each decomposition step, corresponding evidence paragraphs from Wikipedia were identified, ensuring the feasibility of answering the question within a realistic corpus.

Dataset Analysis

The paper outlines impressive diversity and complexity within StrategyQA. Questions span a wide array of topics, necessitating reasoning skills across various domains such as physical, biological, and temporal. Further, the dataset's questions predominantly require multi-step reasoning, with strong emphasis on implicit strategies, which are generally absent in previous datasets.

Performance of Existing Models

Baseline evaluations revealed that pre-trained LLMs, such as RoBERTa, achieve limited performance improvements suggesting strategy questions pose significant challenges. Specifically, models fine-tuned solely on related datasets reach an accuracy of around 63.6%, considerably lower than human performance at 87%.

Implications and Future Work

StrategyQA poses substantial challenges, both in retrieval of relevant context and in executing complex reasoning processes. For future research, it suggests two primary directions:

• Enhancing retrieval strategies that leverage decomposition to improve evidence gathering from large corpora.
• Developing advanced models capable of inferring implicit reasoning strategies from minimal guidance.

By fostering advancements in these areas, this work potentially sets the stage for more adept AI systems, better aligned with human-like reasoning. The integration of implicit reasoning strategies is an essential step toward more robust AI applications, including natural language processing and decision-making systems.