Question-Answering Based Summarization of Electronic Health Records using Retrieval Augmented Generation (2401.01469v1)

Abstract: Summarization of electronic health records (EHRs) can substantially minimize 'screen time' for both patients as well as medical personnel. In recent years summarization of EHRs have employed machine learning pipelines using state of the art neural models. However, these models have produced less than adequate results that are attributed to the difficulty of obtaining sufficient annotated data for training. Moreover, the requirement to consider the entire content of an EHR in summarization has resulted in poor performance due to the fact that attention mechanisms in modern LLMs adds a quadratic complexity in terms of the size of the input. We propose here a method that mitigates these shortcomings by combining semantic search, retrieval augmented generation (RAG) and question-answering using the latest LLMs. In our approach summarization is the extraction of answers to specific questions that are deemed important by subject-matter experts (SMEs). Our approach is quite efficient; requires minimal to no training; does not suffer from the 'hallucination' problem of LLMs; and it ensures diversity, since the summary will not have repeated content but diverse answers to specific questions.

Introduction

The utility of electronic health records (EHRs) in healthcare is indisputable, yet their summarization remains a time-consuming task for medical professionals. The application of LLMs in generating these summaries has shown potential, but there are significant obstacles, including the adequacy of the results, the lack of domain-specific training data, and the unsuitability of LLM attention mechanisms for large inputs. Addressing these challenges, a novel approach utilizing semantic search, retrieval augmented generation (RAG), and question-answering is proposed, which promises efficient, minimal-training-required solutions to EHR summarization.

Methods

Data Source and Preparation

The authors have selected the MIMIC-III dataset for its comprehensive and representative nature, featuring a plethora of clinical notes integral to a patient’s EHR. The emphasis is on "complete" EHR data from long-stay patients (>10 days), with three clinician SMEs involved in the annotation and summary generation processes.

Summarization Approach

The approach stands on segmenting EHRs into paragraphs as semantically complete units, indexing these using vector embeddings for efficient retrieval. It then strategically fetches relevant snippets via questions shaped by SMEs, to form the crux of a summary. This process exploits the latest LLMs to generate responses to the queries with an accompanying confidence score. A weighted combination of scores aids in selecting sentences that collectively form a comprehensive summary.

Evaluation

For validation, the method combines subjective assessments by SMEs with objective measures such as ROUGE and BLEU scores. In addition, semantic methods using embeddings and vector similarity will be leveraged. This mixture of quantitative and qualitative evaluations is designed to ascertain the summaries' accuracy, relevance, and the critical compression ratio, showcasing the method's efficacy beyond traditional metrics.

Concluding Remarks

The proposed approach to EHR summarization via question-answering and RAG offers a paradigm shift from traditional machine-learning methods. It holds the promise of substantial time savings for clinicians, focusing on the extraction of answers to predetermined questions, signaling essential information as identified by SMEs. Initial experiments have yielded promising results, and the authors are scaling this evaluation to more extensive datasets, signaling wider applications across domains where contextualized summaries are required. This undertaking could potentially transform tasks such as HR interview summarization in similar ways, signifying its broader applicability and impact on information handling.