Development and Testing of Retrieval Augmented Generation in Large Language Models -- A Case Study Report (2402.01733v1)

Abstract: Purpose: LLMs hold significant promise for medical applications. Retrieval Augmented Generation (RAG) emerges as a promising approach for customizing domain knowledge in LLMs. This case study presents the development and evaluation of an LLM-RAG pipeline tailored for healthcare, focusing specifically on preoperative medicine. Methods: We developed an LLM-RAG model using 35 preoperative guidelines and tested it against human-generated responses, with a total of 1260 responses evaluated. The RAG process involved converting clinical documents into text using Python-based frameworks like LangChain and Llamaindex, and processing these texts into chunks for embedding and retrieval. Vector storage techniques and selected embedding models to optimize data retrieval, using Pinecone for vector storage with a dimensionality of 1536 and cosine similarity for loss metrics. Human-generated answers, provided by junior doctors, were used as a comparison. Results: The LLM-RAG model generated answers within an average of 15-20 seconds, significantly faster than the 10 minutes typically required by humans. Among the basic LLMs, GPT4.0 exhibited the best accuracy of 80.1%. This accuracy was further increased to 91.4% when the model was enhanced with RAG. Compared to the human-generated instructions, which had an accuracy of 86.3%, the performance of the GPT4.0 RAG model demonstrated non-inferiority (p=0.610). Conclusions: In this case study, we demonstrated a LLM-RAG model for healthcare implementation. The pipeline shows the advantages of grounded knowledge, upgradability, and scalability as important aspects of healthcare LLM deployment.

Overview of LLM-RAG Model Development

In the exploration of LLMs for healthcare applications, particularly within the domain of preoperative medicine, the Retrieval Augmented Generation (RAG) presents a novel solution, as detailed in a comprehensive case paper report. The paper examined the efficacy of an LLM-RAG model, assessing its performance in generating accurate and practical preoperative instructions, benchmarked against human counterparts.

Methodology

The development process involved embedding 35 preoperative guidelines into an LLM-RAG framework. A total of 1260 responses were analyzed across different modalities, comparing human-generated instructions with those produced by baseline LLMs and their RAG-augmented versions. A sophisticated Python-based text conversion approach was employed to tailor these clinical guidelines for compatibility with the RAG framework. In terms of embeddings, models such as OpenAI's text-embedding-ada-002 were used in conjunction with cloud-based vector storage solutions like Pinecone. The RAG retrieval was managed by a customized Retrieval Agent that harnesses the stored vectors to find the most pertinent chunks of knowledge in relation to user queries.

Efficacy Outcomes

The results demonstrated powerful performance attributes of the LLM-RAG models. The enhanced GPT4.0-RAG model in particular exhibited the highest accuracy rate at 91.4%, whilst requiring a mere 15-20 seconds to generate responses—dramatically outspeeding the traditional 10-minute timeframe associated with human effort. Additionally, this model showcased comparable results to human practitioners, with an accuracy rate non-inferior at a p-value of 0.610. The efficiency and scalability of the LLM-RAG deployment in healthcare environments are further highlighted by these findings.

Conclusion and Implications

The investigations conclude that the integration of domain-specific knowledge through RAG can significantly boost LLM capacities within subspecialty healthcare domains. It offers a speed advantage while maintaining accuracy parity with human professionals. The LLM-RAG model, particularly based on GPT4.0, aligns with the priorities of modern healthcare—rapid, reliable, and scalable solutions for delivering patient care. The paper suggests that when applied judiciously, tailored LLM-RAG systems have the potential to augment human expertise effectively, promoting consistency and reducing the subjective variability in preoperative assessments.

Future Perspectives

Despite the promising outcomes, the authors also recognize certain constraints, emphasizing the need for periodic updates to the model as medical literature evolves. They propose a cautious implementation of such AI systems, complementing human expertise rather than substituting it. This is especially crucial given ethical considerations and potential biases inherent in AI deployment within clinical settings. The need for a benchmarked evaluation framework for RAG-LLM models in clinical applications is also identified as an essential step forward for the field.