Improving Machine Reading Comprehension with General Reading Strategies (1810.13441v2)

Abstract: Reading strategies have been shown to improve comprehension levels, especially for readers lacking adequate prior knowledge. Just as the process of knowledge accumulation is time-consuming for human readers, it is resource-demanding to impart rich general domain knowledge into a deep LLM via pre-training. Inspired by reading strategies identified in cognitive science, and given limited computational resources -- just a pre-trained model and a fixed number of training instances -- we propose three general strategies aimed to improve non-extractive machine reading comprehension (MRC): (i) BACK AND FORTH READING that considers both the original and reverse order of an input sequence, (ii) HIGHLIGHTING, which adds a trainable embedding to the text embedding of tokens that are relevant to the question and candidate answers, and (iii) SELF-ASSESSMENT that generates practice questions and candidate answers directly from the text in an unsupervised manner. By fine-tuning a pre-trained LLM (Radford et al., 2018) with our proposed strategies on the largest general domain multiple-choice MRC dataset RACE, we obtain a 5.8% absolute increase in accuracy over the previous best result achieved by the same pre-trained model fine-tuned on RACE without the use of strategies. We further fine-tune the resulting model on a target MRC task, leading to an absolute improvement of 6.2% in average accuracy over previous state-of-the-art approaches on six representative non-extractive MRC datasets from different domains (i.e., ARC, OpenBookQA, MCTest, SemEval-2018 Task 11, ROCStories, and MultiRC). These results demonstrate the effectiveness of our proposed strategies and the versatility and general applicability of our fine-tuned models that incorporate these strategies. Core code is available at https://github.com/nlpdata/strategy/.

Improving Machine Reading Comprehension with General Reading Strategies

The research paper "Improving Machine Reading Comprehension with General Reading Strategies" addresses enhancements in machine reading comprehension (MRC) through the application of strategies inspired by cognitive science. The authors present three main strategies: Back and Forth Reading, Highlighting, and Self-Assessment, which are implemented during the fine-tuning phase of a pre-trained LLM.

The paper is premised on the challenges of non-extractive MRC tasks, where candidate answers are not limited to text spans in the reference document. These tasks require complex reading skills and are indicative of the comprehensive abilities of machine readers. With limited computational resources available, the authors propose innovative strategies to augment the comprehension capacity of machine learning models without an extensive pre-training phase.

Key Strategies

1. Back and Forth Reading (BF): This strategy involves considering both the original and reverse order of the input sequences. This approach is reminiscent of human readers navigating back and forth within a text to understand relationships among ideas, thus allowing the model to gather comprehensive context from different perspectives.
2. Highlighting (HL): Relevant to human practices of remembering crucial information through highlights, this strategy integrates a trainable embedding that highlights tokens pertinent to the question and candidate answers. By doing so, the model's focus is drawn to significant parts related to the task.
3. Self-Assessment (SA): Inspired by self-directed question and answer sessions in human learning, this strategy generates questions and potential answers in an unsupervised manner from the text. This mechanism aids in developing models with deeper insight by encouraging internal practice.

Results and Implications

Through these strategies, the paper reports a 5.8% increase in accuracy on the intricate RACE dataset, a multi-domain multiple-choice MRC benchmark. Furthermore, when fine-tuned on targeted MRC tasks, the model surpasses existing state-of-the-art methods by 6.2% across six different representative non-extractive MRC datasets including ARC, OpenBookQA, MCTest, SemEval-2018 Task 11, ROCStories, and MultiRC.

The success of these strategies demonstrates their efficacy and versatility, suggesting potential enhancements in real-world applications where diverse reading skills are paramount. The results underscore the paper's contribution to improving the performance of pre-trained models on non-extractive MRC tasks with limited computational overhead.

Future Directions

The paper opens avenues for further exploration into integrating cognitive strategies into machine learning frameworks beyond MRC. Future research could explore hybrid strategies combining cognitive insights with traditional machine learning approaches to optimize performance further. Additionally, examining the impact of such strategies on datasets with different characteristics or even in multilingual contexts could broaden the applicability of these approaches.

Overall, the paper provides a compelling case for leveraging human-inspired reading techniques in machine learning models to achieve enhanced reading comprehension, thereby contributing valuable insights into the future development of artificial intelligence.