MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (1905.13453v1)

Abstract: A large number of reading comprehension (RC) datasets has been created recently, but little analysis has been done on whether they generalize to one another, and the extent to which existing datasets can be leveraged for improving performance on new ones. In this paper, we conduct such an investigation over ten RC datasets, training on one or more source RC datasets, and evaluating generalization, as well as transfer to a target RC dataset. We analyze the factors that contribute to generalization, and show that training on a source RC dataset and transferring to a target dataset substantially improves performance, even in the presence of powerful contextual representations from BERT (Devlin et al., 2019). We also find that training on multiple source RC datasets leads to robust generalization and transfer, and can reduce the cost of example collection for a new RC dataset. Following our analysis, we propose MultiQA, a BERT-based model, trained on multiple RC datasets, which leads to state-of-the-art performance on five RC datasets. We share our infrastructure for the benefit of the research community.

MultiQA: An Empirical Investigation of Generalization and Transfer

The paper "MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension" by Alon Talmor and Jonathan Berant offers an extensive empirical analysis on generalization and transfer learning across multiple reading comprehension (RC) datasets. The research aims to discern if training models on one RC dataset allows them to effectively generalize to others, and whether leveraging existing datasets can enhance the performance on new target RC datasets.

Key Findings and Contributions

The authors examine ten RC datasets with differing attributes such as context source, question formulation, and reasoning requirements. By training models on one or several source datasets and evaluating them on target datasets, the authors draw conclusions on performance generalization in zero-shot scenarios — those wherein no target dataset examples are used for training — and transfer performance when fine-tuning on small target dataset batches.

The paper utilizes two models: DocQA, based on BiDAF architecture, and the BERT-based BertQA, with BERT's contextualized representations offering a robust platform for improved generalization. Results indicate that models typically over-fit to specific datasets, revealing significant gaps in zero-shot evaluation scenarios. However, BERT-based models exhibited enhanced generalization, particularly when the context was similar to BERT's training material (Wikipedia, Newswire).

In controlled experiments, pre-training dramatically boosted performance across smaller datasets, even with BERT-based models. Training on mixed datasets (Multi-75K) relieved the burden of individually selecting a dataset and boosted robustness, making it a practical alternative. Notably, generalization improved as training data from multiple datasets increased, indicating that larger data volumes help bridge accuracy gaps across differently-styled benchmarks.

Moreover, the paper investigates context augmentation, specifically in TriviaQA, demonstrating that combining contexts from varied sources can significantly enhance performance. This strategy allows models to capitalize on the nuances of different dataset formats, providing a comprehensive context for question-answering tasks.

Implications for Future Research and Applications

The insights drawn from this investigation have implications for both theoretical exploration and practical deployment in NLP. Multi-task learning via aggregation of datasets like MultiQA could optimize the training process for RC tasks, offering alternate pathways for dataset size limitations. This approach also suggests a pathway for reducing the costs associated with data collection and annotation, by capitalizing on existing sources.

Experiments revealing BERT's context-specific favoritism on Wikipedia and Newswire could drive future research in fine-tuning pretrained models specifically tailored to varying text corpuses, enhancing performance consistency across distinct benchmarks.

Conclusion

In conclusion, the empirical research conducted through MultiQA substantially advances the understanding of model generalization and transferability in reading comprehension tasks. The empirical evidence underscores the efficacy of multi-task learning methods in circumventing dataset-specific challenges and provides a clear trajectory for harnessing large-scale LLMs in RC tasks. This research contributes significant foundational work for developing more generalized, broadly competent NLP systems. The infrastructure shared by the authors promises to facilitate continued exploration and innovation in the community.