How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks (1808.04926v2)

Abstract: Many papers address reading comprehension, where examples consist of (question, passage, answer) tuples. Presumably, a model must combine information from both questions and passages to predict corresponding answers. However, despite intense interest in the topic, with hundreds of published papers vying for leaderboard dominance, basic questions about the difficulty of many popular benchmarks remain unanswered. In this paper, we establish sensible baselines for the bAbI, SQuAD, CBT, CNN, and Who-did-What datasets, finding that question- and passage-only models often perform surprisingly well. On $14$ out of $20$ bAbI tasks, passage-only models achieve greater than $50\%$ accuracy, sometimes matching the full model. Interestingly, while CBT provides $20$-sentence stories only the last is needed for comparably accurate prediction. By comparison, SQuAD and CNN appear better-constructed.

• The paper reveals that simplified models focusing solely on passages or questions achieve competitive accuracy, challenging the assumed complexity of RC benchmarks.
• It critically evaluates datasets like bAbI and CBT, demonstrating that design flaws allow models to succeed without genuine passage-question integration.
• The study advocates for rigorous baseline evaluations and improved benchmark designs to drive authentic advances in reading comprehension research.

An Analysis of Reading Comprehension Benchmarks

The paper "How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks," authored by Divyansh Kaushik and Zachary C. Lipton, presents a compelling critique of the current state of reading comprehension (RC) datasets and the methods used to assess the efficacy of models trained on these datasets. The authors specifically examine prominent RC datasets, including bAbI, SQuAD, CBT, CNN, and Who-did-What, in order to evaluate whether the structure of these datasets adequately reflects the complexity of reading comprehension tasks.

Critical Examination of RC Benchmarks

Despite the ongoing development of advanced deep learning algorithms aimed at solving RC tasks, this paper challenges some fundamental assumptions underlying popular benchmarks. The central concern addressed is whether these datasets genuinely require a model to integrate information from both the given passages and the related questions to predict answers. The researchers demonstrate that for several prominent datasets, models that focus exclusively on either the passage or the question can achieve surprisingly competitive results. For example, passage-only models demonstrated over 50% accuracy on 14 out of 20 bAbI tasks, sometimes even matching models with complete access to both passage and question information.

Key Findings and Implications

1. Model Baselines: The paper establishes that using question-only (Q-only) and passage-only (P-only) models as baselines reveal that many datasets are not as challenging as traditionally perceived. On the bAbI tasks, passage-only models surpassed 50% accuracy, while some tasks showed question-only models achieving better-than-expected results.
2. Dataset Design Flaws: In the Children's Books Test (CBT) dataset, the authors find that using only the last sentence of the passage leads to comparable performance with models using the full passage. This suggests that the necessity to distill information from multiple sentences is not as critical in some datasets as might be assumed.
3. Architectural Implications: The critique extends to architectural designs that typically claim to leverage passage-question interactions. The research suggests that observed model improvements might not always arise from better passage-question integration, but from other confounding factors.
4. Practical Recommendations: The authors recommend benchmark creators provide rigorous baselines to better characterize task difficulties. This means not only focusing on the full-task performance but also assessing how critical each dataset component (e.g., question and passage) is to achieving high accuracy.
5. Effect on Future AI Developments: By highlighting these issues, the authors advocate for a more nuanced understanding of what reading comprehension tasks entail and how they are currently evaluated. This calls for more thoughtful design and comprehensive reporting in future RC datasets, potentially spurring the development of more robust AI that better understands human language.

Speculation on Future Developments

This investigation raises important questions about the future trajectory of AI in NLP. The insights presented underscore the necessity for more complex and contextually rich datasets that can truly challenge RC models to mimic human reading comprehension. The paper suggests improvements in dataset design could lead to more meaningful advances in AI, as models will need to develop deeper semantic reasoning capabilities rather than merely exploiting dataset-specific shortcuts. Future research should thus prioritize empirical rigor, ensuring that model improvements genuinely reflect advances in understanding and not just dataset exploitation.

In conclusion, the paper reveals significant shortcomings in the current evaluation paradigms of reading comprehension tasks, positing that these benchmarks may inadequately reflect the true complexity of human language understanding. By critically analyzing these prominent datasets, the authors provide a roadmap for refining these benchmarks and advancing the state of AI in reading comprehension.