Improving Health Question Answering with Reliable and Time-Aware Evidence Retrieval (2404.08359v1)

Abstract: In today's digital world, seeking answers to health questions on the Internet is a common practice. However, existing question answering (QA) systems often rely on using pre-selected and annotated evidence documents, thus making them inadequate for addressing novel questions. Our study focuses on the open-domain QA setting, where the key challenge is to first uncover relevant evidence in large knowledge bases. By utilizing the common retrieve-then-read QA pipeline and PubMed as a trustworthy collection of medical research documents, we answer health questions from three diverse datasets. We modify different retrieval settings to observe their influence on the QA pipeline's performance, including the number of retrieved documents, sentence selection process, the publication year of articles, and their number of citations. Our results reveal that cutting down on the amount of retrieved documents and favoring more recent and highly cited documents can improve the final macro F1 score up to 10%. We discuss the results, highlight interesting examples, and outline challenges for future research, like managing evidence disagreement and crafting user-friendly explanations.

Enhancing Performance of Health Question Answering Systems through Optimal Evidence Retrieval Strategies

Introduction to Health Question Answering Systems

Health Question Answering (QA) Systems leverage vast collections of documented medical research to provide answers to health-related inquiries. Given the abundance of medical literature and the rapid evolution of clinical recommendations, sourcing the most relevant and up-to-date evidence is pivotal. Traditional QA systems, however, often fall short when faced with novel queries, primarily due to their reliance on predefined evidence documents. This paper embarks on refining the open-domain QA system – a more realistic approach that necessitates the retrieval of pertinent evidence from extensive document corpora before formulating an answer. By exploring various retrieval settings, including the volume of documents retrieved and the incorporation of metadata such as publication year and citation count, this research aims to fine-tune the QA system's performance within the domain of health.

The Intricacies of Open-Domain QA Systems

Open-domain QA systems, characterized by their ability to query extensive document collections, primarily consist of two components: the retriever and the reader. The retriever's role is to source documents that potentially contain the answer, while the reader extracts and formulates this answer based on the evidence provided by the retriever. This paper posits the hypothesis that the performance of the QA system predominantly hinges on the effectiveness of the retriever component. The premise being, the quality and relevance of the documents retrieved play a crucial role in the accuracy of the final answer provided.

To validate this, experiments were crafted around PubMed's collection of medical research documents, testing various configurations of the retrieve-then-read pipeline. These configurations included adjustments in the number of documents and sentences retrieved, as well as considerations for the publication year and citation count of these documents. Findings from this research indicate a potential improvement in macro F1 scores by up to 10% through the optimization of retrieval strategies alone.

Methodological Approach

The paper embarked on a series of experiments to evaluate the impact of different evidence retrieval configurations on the health QA system's accuracy. Three health-related question datasets were employed, using PubMed as the source for evidence retrieval. By fixing the reader component and varying the retrieval strategies, the research isolated the effects of retrieval adjustments on system performance.

Key experiments included varying the number of documents retrieved and extracting the top sentences from these documents for QA processing. Additionally, the paper delved into the influence of document quality – assessed by recency and citation count – on QA accuracy. Performance metrics such as precision, recall, and F1 score were used to evaluate the system's effectiveness across different settings.

Insights and Implications

The investigation revealed several key insights pertinent to the optimization of open-domain health QA systems:

• Reducing the volume of documents retrieved tends to enhance QA performance, suggesting a higher signal-to-noise ratio with fewer selected documents.
• Extracting top relevant sentences from selected documents further refines the quality of evidence, although the ideal number of sentences varies between datasets.
• Favoring recent and highly cited documents as sources of evidence generally leads to improvements in QA accuracy. This underscores the value of considering document metadata in the retrieval process.

Future Directions

Building on these findings, future research avenues could explore the integration of evidence strength and conflict resolution mechanisms within the QA pipeline. The adoption of models that account for the varying levels of evidence strength across different types of medical studies may offer a more nuanced approach to evidence retrieval. Furthermore, new strategies for handling evidence disagreement and enhancing the interpretability of answers could significantly improve the utility of health QA systems for end-users.

Concluding Remarks

This paper contributes to the ongoing refinement of health question answering systems by highlighting the critical role of evidence retrieval strategies in optimizing system performance. By systematically analyzing the impact of document selection processes and incorporating document quality metrics, the research offers valuable insights for the development of more accurate and reliable health QA systems. As the domain of medical research continues to evolve, so too will the methodologies for effectively navigating its vast literatures to support health information seeking and decision-making.