EHRCon: Dataset for Checking Consistency between Unstructured Notes and Structured Tables in Electronic Health Records (2406.16341v1)

Abstract: Electronic Health Records (EHRs) are integral for storing comprehensive patient medical records, combining structured data (e.g., medications) with detailed clinical notes (e.g., physician notes). These elements are essential for straightforward data retrieval and provide deep, contextual insights into patient care. However, they often suffer from discrepancies due to unintuitive EHR system designs and human errors, posing serious risks to patient safety. To address this, we developed EHRCon, a new dataset and task specifically designed to ensure data consistency between structured tables and unstructured notes in EHRs. EHRCon was crafted in collaboration with healthcare professionals using the MIMIC-III EHR dataset, and includes manual annotations of 3,943 entities across 105 clinical notes checked against database entries for consistency. EHRCon has two versions, one using the original MIMIC-III schema, and another using the OMOP CDM schema, in order to increase its applicability and generalizability. Furthermore, leveraging the capabilities of LLMs, we introduce CheckEHR, a novel framework for verifying the consistency between clinical notes and database tables. CheckEHR utilizes an eight-stage process and shows promising results in both few-shot and zero-shot settings. The code is available at https://github.com/dustn1259/EHRCon.

• The paper presents EHRCon, a novel dataset manually annotating 3,943 entities across 105 clinical notes to bridge unstructured notes and structured EHR data.
• The authors introduce CheckEHR, an eight-stage framework leveraging LLMs, including GPT-3.5, which achieved a peak recall of 61.06% in few-shot settings.
• The study underscores the potential to enhance patient safety and reduce manual burden by automating consistency checks in electronic health records.

Overview of "EHRCon: Dataset for Checking Consistency between Unstructured Notes and Structured Tables in Electronic Health Records"

The paper "EHRCon: Dataset for Checking Consistency between Unstructured Notes and Structured Tables in Electronic Health Records" introduces a novel dataset, EHRCon, aimed at addressing the discrepancies between unstructured clinical notes and structured tables within Electronic Health Records (EHRs). Created in collaboration with healthcare professionals and leveraging data from the MIMIC-III dataset, EHRCon encompasses the manual annotation of 3,943 entities from 105 clinical notes cross-checked against corresponding database entries. The dataset includes two versions: one based on the original MIMIC-III schema and another on the OMOP CDM schema, enhancing its applicability across varied EHR schema types.

The paper underscores the necessity of such a dataset given that inconsistencies between structured and unstructured EHR data can pose significant risks to patient safety due to errors introduced either through unintuitive EHR system designs or practitioner overload. To validate the alignment between clinical notes and database tables automatically, the authors introduce CheckEHR, an eight-stage framework utilizing LLMs to verify consistency.

Numerical Results and Experimental Findings

The authors' experimental analysis on EHRCon showcases the framework's performance under both few-shot and zero-shot settings, employing popular LLMs such as GPT-3.5 (0613), Tulu2, Mixtral, and Llama-3. The highest recall achieved was 61.06% under a few-shot setting using GPT-3.5 (0613), underscoring the complexity of the task. The paper also featured detailed experiments across MIMIC-III and OMOP CDM schemas. Specifically, a performance drop in MIMIC-OMOP datasets, particularly with Type 1 entities, hinted at the complexity introduced by the varied representation of the same entity in these schemas.

Methodology and Analysis

The annotation process for creating EHRCon involved meticulous cross-referencing of clinical notes with structured tables, identifying entities, and labeling them as either 'Consistent' or 'Inconsistent'. The authors implemented additional quality control processes to ensure the dataset's reliability. Their approach to entity recognition leveraged an item search tool to match entities from clinical notes with database items, addressing synonym and abbreviation variations.

CheckEHR's eight-stage framework involves:

1. Note Segmentation: Dividing lengthy clinical notes into coherent sub-texts.
2. Named Entity Recognition (NER): Extracting relevant entities.
3. Time Filtering: Classifying time expressions.
4. Table Identification: Identifying the pertinent tables.
5. Pseudo Table Creation: Generating a pseudo table to extract structured data from clinical notes.
6. Self Correction: Correcting hallucinated information from the LLM.
7. Value Reformatting: Aligning data types with the actual database format.
8. Query Generation: Producing SQL queries to verify consistency.

The framework's robust design is reflected in its ability to consolidate complex tasks in verifying EHR consistency, although it continues to face challenges due to the intrinsic unstructured nature of clinical notes and the sophisticated, varied schemas of EHR databases.

Theoretical and Practical Implications

Theoretical implications of this research span the domains of NLP within healthcare contexts. The paper illustrates how task-specific datasets like EHRCon can drive advancements in LLM capabilities to handle domain-specific complexities such as abbreviations, context-based time expressions, and nuanced clinical language.

Practically, the framework if scaled and refined, could immensely aid healthcare organizations by automating consistency checks, thereby reducing the manual burden on healthcare professionals and improving the accuracy and safety of EHRs. The findings suggest potential applications in other domains requiring consistency checks between structured and unstructured data, such as legal or financial records.

Future Directions

Future research could focus on enhancing LLM reasoning capabilities, particularly in comprehending and processing medical terminologies and nuanced clinical narratives. Scalable approaches to extend the dataset by including real hospital data can mitigate some of the limitations of the current preprocessed MIMIC-III data. Additionally, developing methods to dynamically interact with EHR databases and free text can enhance the practical utility of such frameworks.

In conclusion, EHRCon and CheckEHR together represent a significant step towards improving the reliability of EHRs. By addressing discrepancies between clinical notes and structured data, this research paves the way for enhanced patient safety and more efficient healthcare data management. The work lays a solid foundation for future advancements in AI-driven healthcare documentation systems.