MIRIAD: Augmenting LLMs with millions of medical query-response pairs (2506.06091v2)

Abstract: LLMs are bound to transform healthcare with advanced decision support and flexible chat assistants. However, LLMs are prone to generate inaccurate medical content. To ground LLMs in high-quality medical knowledge, LLMs have been equipped with external knowledge via RAG, where unstructured medical knowledge is split into small text chunks that can be selectively retrieved and integrated into the LLMs context. Yet, existing RAG pipelines rely on raw, unstructured medical text, which can be noisy, uncurated and difficult for LLMs to effectively leverage. Systematic approaches to organize medical knowledge to best surface it to LLMs are generally lacking. To address these challenges, we introduce MIRIAD, a large-scale, curated corpus of 5,821,948 medical QA pairs, each rephrased from and grounded in a passage from peer-reviewed medical literature using a semi-automated pipeline combining LLM generation, filtering, grounding, and human annotation. Unlike prior medical corpora, which rely on unstructured text, MIRIAD encapsulates web-scale medical knowledge in an operationalized query-response format, which enables more targeted retrieval. Experiments on challenging medical QA benchmarks show that augmenting LLMs with MIRIAD improves accuracy up to 6.7% compared to unstructured RAG baselines with the same source corpus and with the same amount of retrieved text. Moreover, MIRIAD improved the ability of LLMs to detect medical hallucinations by 22.5 to 37% (increase in F1 score). We further introduce MIRIAD-Atlas, an interactive map of MIRIAD spanning 56 medical disciplines, enabling clinical users to visually explore, search, and refine medical knowledge. MIRIAD promises to unlock a wealth of down-stream applications, including medical information retrievers, enhanced RAG applications, and knowledge-grounded chat interfaces, which ultimately enables more reliable LLM applications in healthcare.

• The paper demonstrates that MIRIAD’s structured dataset significantly boosts LLM performance by integrating over 5.8M curated medical query-response pairs.
• The methodology employs a semi-automated pipeline with LLM supervision, rule-based filtering, and expert review to ensure reliable, precise medical data.
• Empirical results show up to 6.7% increased accuracy and a 22.5%-37% F1 score improvement in detecting hallucinations, advancing clinical decision support.

Augmenting LLMs with Curated Medical Knowledge: Insights from the MIRIAD Approach

The paper "MIRIAD: Augmenting LLMs with Millions of Medical Query-Response Pairs" presents a significant advancement in the integration of LLMs with curated medical data to improve their applicability in the healthcare domain. The research introduces a comprehensive dataset—MIRIAD—that aims to address the inadequacies of existing LLMs in generating accurate medical content by grounding them in high-quality medical literature.

Overview of MIRIAD

MIRIAD is a sizeable curated corpus containing 5,821,948 medical instruction-response pairs. Each pair is synthesized and aligned with peer-reviewed medical literature through a semi-automated pipeline that incorporates LLM generation, filtering, grounding, and human annotation. The emphasis on a structured query-response format distinguishes MIRIAD from previous unstructured medical datasets, aiming for more precise retrieval capabilities. Through rigorous quality control processes, which include rule-based filtering, LLM-based supervision, and human expert validation, MIRIAD stands out in its attempt to offer a reliable resource for training and enhancing medical information retrieval and decision-making systems.

Empirical Evaluation and Results

The efficacy of MIRIAD is validated through experiments on medical question-answering tasks, demonstrating a noticeable improvement in LLM performance. Specifically, augmenting LLMs with MIRIAD increased accuracy by up to 6.7% over unstructured retrieval augmented generation (RAG) baselines, using the same corpus and amount of retrieved text. Furthermore, MIRIAD significantly improved the detection of medical hallucinations in LLM outputs, with an F1 score increase ranging from 22.5% to 37%. These results underscore the dataset's potential to refine the information retrieval processes, thereby enhancing the reliability of LLM-generated medical content.

Implications and Future Directions

The integration of MIRIAD into existing LLM frameworks exemplifies a promising pathway toward mitigating the generation of inaccurate or misleading medical data. In doing so, it contributes not only to practical applications, such as improving the accuracy of clinical decision support systems but also to theoretical advancements in the field of RAG applications. The interactive tool, MIRIAD-Atlas, further extends its utility by enabling users from different medical disciplines to explore and refine medical knowledge base queries visually.

Looking ahead, the research opens avenues for further expansion and refinement of the dataset, particularly in covering more specialized and emerging areas of medicine. Future developments may include the incorporation of additional retrieval strategies, such as hybrid retrieval techniques and enhanced retrieval model training, to further improve structured information retrieval. Moreover, the possibility of using MIRIAD as a training ground for developing new medical retriever models could advance applications ranging from clinical trials matching to medical digital twins.

In conclusion, the MIRIAD corpus represents a critical step forward in aligning LLMs with structured, curated medical knowledge, thereby addressing key challenges in the deployment of AI in healthcare settings. This work serves as a blueprint for future research endeavors aimed at blending sophisticated retrieval mechanisms with nuanced medical datasets to elevate the performance and trustworthiness of AI in medicine.