MedPromptExtract (Medical Data Extraction Tool): Anonymization and Hi-fidelity Automated data extraction using NLP and prompt engineering (2405.02664v3)

Abstract: Introduction: The labour-intensive nature of data extraction from sources like discharge summaries (DS) poses significant obstacles to the digitisation of medical records particularly for low- and middle-income countries (LMICs). In this paper we present a completely automated method MedPromptExtract to efficiently extract data from DS while maintaining confidentiality. Methods: The source of data was Discharge Summaries (DS) from Kokilaben Dhirubhai Ambani Hospital (KDAH) of patients having Acute Kidney Injury (AKI). A pre-existing tool EIGEN which leverages semi-supervised learning techniques for high-fidelity information extraction was used to anonymize the DS, NLP was used to extract data from regular fields. We used Prompt Engineering and LLM(LLM) to extract custom clinical information from free flowing text describing the patients stay in the hospital. Twelve features associated with occurrence of AKI were extracted. The LLM responses were validated against clinicians annotations. Results: The MedPromptExtracttool first subjected DS to the anonymization pipeline which took three seconds per summary. Successful anonymization was verified by clinicians, thereafter NLP pipeline extracted structured text from the anonymized pdfs at the rate of 0.2 seconds per summary with 100% accuracy.Finally DS were analysed by the LLM pipeline using Gemini Pro for the twelve features. Accuracy metrics were calculated by comparing model responses to clinicians annotations with seven features achieving AUCs above 0.9, indicating high fidelity of the extraction process. Conclusion: MedPromptExtract serves as an automated adaptable tool for efficient data extraction from medical records with a dynamic user interface. Keywords: Digitizing Medical Records, Automated Anonymisation, Information Retrieval, LLMs, Prompt Engineering

An Evaluation of MedPromptExtract: A Medical Data Extraction Tool Using NLP and Prompt Engineering

The paper "MedPromptExtract (Medical Data Extraction Tool): Anonymization and Hi-fidelity Automated Data Extraction using NLP and Prompt Engineering" explores the development and application of MedPromptExtract, a tool designed to automate the extraction and anonymization of medical data from discharge summaries. The tool addresses key challenges associated with digitizing non-interoperable medical records, particularly in low and middle-income countries, by leveraging advances in NLP, LLMs, and prompt engineering.

Methodological Framework

The authors employ a comprehensive methodology involving semi-supervised learning, optical character recognition (OCR), and probabilistic models for data extraction and anonymization. The dataset comprises 914 discharge summaries from patients at Kokilaben Dhirubhai Ambani Hospital, focusing on identifying indicators of Acute Kidney Injury (AKI).

Anonymization and Preprocessing

The anonymization process utilizes EIGEN, a tool that combines LLMs and labeling functions to maintain high-fidelity anonymization of patient data, such as names and locations. This is complemented by LayoutLM, a pre-trained deep neural network model, to enhance information extraction. The data preprocessing stage employs stopwords to exclude irrelevant sections and focuses on pertinent medical information by using regular expressions and stopword filtering.

Data Extraction and Validation

MedPromptExtract uses prompt engineering with the Gemini API to analyze and extract relevant clinical information from text under the "course in hospital" section of the discharge summaries. The extracted data undergoes rigorous validation against human annotations, demonstrating substantial agreement with expert evaluations. This validation is quantitatively captured, with notable metrics such as the AUC exceeding 0.9 for several clinical features, reflecting the tool's effectiveness in extracting high-fidelity data.

Numerical Results and Implications

The paper presents compelling numerical findings, particularly in terms of model accuracy and inter-rater agreement, where an average processing time per summary of seven seconds highlights the system's efficiency. The tool successfully processes 852 out of 914 summaries, with strong validation outcomes that substantiate its clinical utility. The statistical results underscore the robustness of MedPromptExtract in transforming discharge summaries into a structured dataset suitable for further analysis, thereby facilitating research and clinical decision-making.

Discussion and Future Directions

MedPromptExtract offers significant implications for the healthcare sector. Its ability to automate and anonymize medical data efficiently can save substantial time for clinicians and researchers. Additionally, its integration into existing Electronic Health Record (EHR) systems represents a viable pathway for scaling this solution across diverse healthcare settings.

However, the paper acknowledges certain limitations, notably the specificity of the pipeline to the source hospital's summary format. While the methodology can be adapted to similar contexts with minor adjustments, further research could extend its applicability to more varied data sources.

Future developments may enhance the tool's versatility, potentially incorporating machine learning models trained on more extensive and diverse datasets. As AI and NLP continue to evolve, integrating more advanced LLMs could refine data extraction accuracy and expand the range of extractable medical information.

Conclusion

The MedPromptExtract tool exemplifies a practical implementation of AI-driven solutions capable of addressing critical inefficiencies in medical data digitization and extraction. Through sophisticated NLP and prompt engineering techniques, the paper showcases a robust framework for transforming raw medical records into valuable datasets for clinical and research purposes. The documented achievements underscore the tool's potential to significantly impact medical data management and pave the way for further advancements in AI applications within healthcare.