emrQA: A Large Corpus for Question Answering on Electronic Medical Records (1809.00732v1)

Abstract: We propose a novel methodology to generate domain-specific large-scale question answering (QA) datasets by re-purposing existing annotations for other NLP tasks. We demonstrate an instance of this methodology in generating a large-scale QA dataset for electronic medical records by leveraging existing expert annotations on clinical notes for various NLP tasks from the community shared i2b2 datasets. The resulting corpus (emrQA) has 1 million question-logical form and 400,000+ question-answer evidence pairs. We characterize the dataset and explore its learning potential by training baseline models for question to logical form and question to answer mapping.

An Expert's Insight into "emrQA: A Large Corpus for Question Answering on Electronic Medical Records"

The paper "emrQA: A Large Corpus for Question Answering on Electronic Medical Records" presents a novel methodological approach for generating domain-specific large-scale question answering (QA) datasets. This work is focused on the repurposing of existing annotations from various NLP tasks to facilitate the creation of datasets within the clinical domain, specifically for electronic medical records (EMRs).

Methodology and Dataset Composition

The authors have developed a methodological framework that systematically generates QA datasets by employing existing annotations, thus providing an efficient alternative to costly manual annotation processes. This framework has been practically applied to create the emrQA dataset, a substantial corpus for QA on EMRs. Utilizing the i2b2 dataset's annotations, the emrQA corpus encompasses 1 million question-logical form pairs and over 400,000 question-answer evidence pairs.

The creation of emrQA involved several critical steps: collecting clinical question templates, associating these templates with logical forms, and finally, using i2b2 annotations to fill these templates and retrieve answers. The authors performed a detailed analysis of the questions' complexity, relations, and reasoning requirements, providing both neural and heuristic baselines for mapping questions to logical forms and answers.

Key Contributions

The paper outlines three principal contributions:

1. Framework for QA Dataset Generation: A framework is proposed that can be utilized in any domain where manual annotation is challenging, leveraging limited annotations available for other NLP tasks.
2. Comprehensive EMR-Specific Dataset: The creation of the emrQA dataset is the first instance of an accessible patient-specific EMR QA corpus, allowing the benchmarking of interpretable models that map questions to logical forms.
3. Introduction of New Reasoning Challenges: The dataset includes new reasoning challenges such as arithmetic and temporal reasoning, which are often absent in open-domain datasets like SQuAD.

Analysis and Complexity

The authors delve into the complexity of the emrQA dataset, highlighting that clinical questions often require multi-sentence reasoning, involve extensive domain-specific terminology, and require temporal and arithmetic reasoning over longitudinal clinical narratives. The paper illustrates that the dataset is representative of real-world scenarios, noting that a significant percentage of questions require deeper medical and world knowledge.

Baseline Models and Experimental Insight

Baseline performance for question-to-logical-form and question-to-answer mapping was established using neural and heuristic models, with an analysis highlighting areas for improvement. The seq2seq model employed shows the need for innovation in model architecture to handle complex paraphrasing and extensive logical form structures. The implementation of DrQA for answer extraction highlighted the challenges posed by long-form clinical texts, reflecting the need for future work in this area to improve state-of-the-art performance.

Implications and Future Directions

The implications of this paper are significant both practically and theoretically. Practically, emrQA provides a foundation for developing clinical QA systems that can assist healthcare professionals in accessing vital patient information. Theoretically, the dataset and framework open avenues for research in interpretable AI and hybrid reasoning models that can handle complex domain-specific queries with increased robustness and accuracy.

The future developments in AI, as suggested by this work, would likely focus on hybrid models that integrate the accuracy of neural networks with the interpretability of logical systems. Additionally, the framework's potential application in other domains could foster the creation of even more diverse and representative datasets outside the medical domain.

In conclusion, this paper contributes substantially to the field of clinical NLP by introducing the emrQA dataset and providing a replicable framework for large-scale dataset generation in challenging domains. As AI continues to evolve, such frameworks and datasets will be crucial in pushing the boundaries of what is possible with machine comprehension and logic-based reasoning models.