A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning (1908.05514v2)

Abstract: Rapid progress has been made in the field of reading comprehension and question answering, where several systems have achieved human parity in some simplified settings. However, the performance of these models degrades significantly when they are applied to more realistic scenarios, such as answers involve various types, multiple text strings are correct answers, or discrete reasoning abilities are required. In this paper, we introduce the Multi-Type Multi-Span Network (MTMSN), a neural reading comprehension model that combines a multi-type answer predictor designed to support various answer types (e.g., span, count, negation, and arithmetic expression) with a multi-span extraction method for dynamically producing one or multiple text spans. In addition, an arithmetic expression reranking mechanism is proposed to rank expression candidates for further confirming the prediction. Experiments show that our model achieves 79.9 F1 on the DROP hidden test set, creating new state-of-the-art results. Source code\footnote{\url{https://github.com/huminghao16/MTMSN}} is released to facilitate future work.

• The paper introduces MTMSN, a novel network that predicts a variety of answer types, including arithmetic and logical negation.
• It employs multi-span extraction to capture multiple non-overlapping answers, significantly boosting performance on complex datasets.
• The model integrates an arithmetic reranking mechanism for context-aware reasoning, setting a new benchmark on the DROP dataset.

Analyzing the Multi-Type Multi-Span Network for Reading Comprehension Requiring Discrete Reasoning

The paper presents a focused investigation into the complexities of reading comprehension, specifically targeting scenarios that demand discrete reasoning abilities. Termed the Multi-Type Multi-Span Network (MTMSN), the proposed model introduces innovative techniques to enhance reading comprehension systems, particularly when addressing a diverse set of answer types, from straightforward text spans to more intricate numerical and logical reasoning.

Key Contributions

The principal contribution of this research lies in the development of a neural architecture adept at handling a range of answer types, expanding the scope beyond traditional text span extraction. MTMSN is notable for several specific advancements:

1. Multi-Type Answer Predictor: This mechanism supports a variety of answer types, including spans, count numbers, arithmetic expressions, and introduces support for logical negation, broadening the model’s flexibility in terms of answer prediction.
2. Multi-Span Extraction: Designed to dynamically produce multiple non-overlapping spans, this innovation allows the model to function effectively in scenarios where multiple answers are warranted, a situation prevalent in more complex reading comprehension datasets like DROP.
3. Arithmetic Expression Reranking: To reinforce the correctness of arithmetic reasoning, the model incorporates a reranking mechanism that evaluates expression candidates based on contextual information. This ensures that selected arithmetic expressions are contextually plausible, addressing potential prediction inaccuracies.

The MTMSN achieves state-of-the-art results on the DROP dataset, marking a significant advancement with an F1 score of 79.9, which is illustrative of its efficacy in handling complex comprehension tasks.

Experimental Insights

The comprehensive evaluation includes baseline comparisons, component ablations, and performance breakdowns by answer type. Significant insights include the critical role of arithmetic expression prediction in the model’s overall performance, underscoring the prevalence of tasks involving numerical reasoning. The introduction of logical negation notably contributes to performance gains, addressing an oversight in prior work.

Furthermore, analysis of the multi-answer scenario reveals its impact on F1 over EM scores, highlighting the model's capability in partial answer extraction. The model’s reliance on the reranking component for arithmetic expressions affirms the necessity of context-aware reasoning in comprehension tasks requiring numerical calculations.

Implications and Future Directions

The research underscores the centrality of diverse answer type support in reading comprehension systems, particularly for datasets predicated on discrete reasoning requirements. The successful implementation of logical operations, such as negation and multi-span extraction, broadens the potential applications of reading comprehension models in real-world scenarios, including those requiring sophisticated logical processing.

Future directions could involve extending this framework to integrate more complex reasoning tasks, such as sorting and higher-order arithmetic operations like multiplication and division. The ongoing integration of symbolic reasoning frameworks may also enhance the model’s capacity to handle intricate composition tasks, stored within the neural architecture.

In conclusion, MTMSN sets a high benchmark in reading comprehension with discrete reasoning. Its advances resonate with future AI developments, especially those orienting towards seamless integration of symbolic reasoning capabilities within neural architectures.