Towards Error-Free EHRs: Reasoning-Intensive Consistency Verification Between Clinical Notes and Structured Tables in Electronic Health Records

Abstract: Data consistency between unstructured clinical notes and structured tables in Electronic Health Records (EHRs) is essential for patient safety and clinical decision-making. However, existing work on note-table consistency verification mainly relies on surface-level matching of numeric values or simple events. Such approaches fail to capture the reasoning underlying real-world EHR documentation, including clinical interpretation, event relations, and temporal changes. To address this gap, we introduce EHR-ReasonCon, a reasoning-intensive benchmark for note-table consistency verification. Built on MIMIC-III with expert-guided annotations, it comprises 8,048 entities derived from clinical notes and provides high-quality ground-truth labels. The annotation protocol is supported by specialized table-exploration tools to ensure systematic evidence retrieval and reliable consistency assessment. We also propose EHR-Inspector, an LLM-based framework that segments notes, extracts anchor entities and temporal references, and uses table-exploration tools to verify consistency against structured tables. Evaluated using expert-validated LLM-as-a-judge metrics under harsh and lenient criteria, EHR-Inspector achieves state-of-the-art performance across multiple model backbones. Analyses further demonstrate the effectiveness of its components and highlight differences from human verification.

• The paper introduces EHR-ReasonCon and EHR-Inspector, innovative tools that use detailed reasoning to verify consistency between clinical notes and structured EHR data.
• It presents a multi-stage, expert-informed annotation protocol and LLM-augmented framework that effectively address semantic, temporal, and compositional discrepancies.
• Benchmarking results demonstrate significant improvements over previous methods, showing robust performance in handling complex clinical data and external dataset validations.

Reasoning-Intensive Consistency Verification for EHRs: EHR-ReasonCon and the EHR-Inspector Framework

Introduction

Accurate and reliable documentation within Electronic Health Records (EHRs) is essential for both patient safety and effective clinical decision-making. EHR data is captured in both structured tables (e.g., vital signs, laboratory events, medication administration) and unstructured clinical notes. In practice, there is a high prevalence of semantic, temporal, and factual discrepancies between these modalities—posing risks for downstream analytics, care quality, and medico-legal liability. Although prior work has developed automated verification methods, most are predicated on surface-level alignment (e.g., direct value matching or presence checks) and lack the capacity for nuanced, context-dependent reasoning required to assess consistency in naturalistic clinical records.

This paper addresses this limitation by introducing EHR-ReasonCon, a large-scale, reasoning-intensive benchmark for note–table consistency verification, and proposing EHR-Inspector, an LLM- and tool-augmented framework that mirrors human annotation workflows for comprehensive and scalable consistency assessment.

Figure 1: Reasoning-intensive consistency verification requires evaluating interpretive, relational, and temporal alignment between notes and structured tables, illustrated with clinical-centric examples.

EHR-ReasonCon: Dataset and Annotation Protocol

EHR-ReasonCon comprises over 8,000 expertly annotated anchor entities across 105 clinical notes (spanning discharge summaries, physician notes, and nursing notes) from the MIMIC-III database, linked to 14 structured tables encompassing time series, categorical, event, and diagnostic information. This dataset is unique for its explicit design to reflect the reasoning demands of real-world EHR documentation, such as:

• Interpretive Relations: Clinical notes may contain derived or interpreted statements (e.g., “anemia on admission”) that must be cross-validated using medical criteria and quantitative values in tables.
• Temporal Complexity: Statements often describe longitudinal trends or time-bounded facts (e.g., “no fever from day 3 to 5”), which require reconstructing absolute and relative temporal references to structured records.
• Compositional and Semantic Alignment: Notes and tables may use distinct nomenclature and abstractions, necessitating ontology-guided mapping and semantic expansion.

The annotation process incorporates a multi-stage protocol:

Figure 2: The annotation workflow: protocol/tool development, anchor entity selection, tool-assisted evidence retrieval, entity-level verification, and dataset-wide quality control.

• Stage 0: Protocol and tool design, iteratively refined with clinical experts, resulting in eight specialized table-exploration tools.
• Stage 1: Anchor entity identification, extracting target concepts from notes for table alignment.
• Stage 2: Systematic evidence retrieval via tool-assisted database queries.
• Stage 3: Entity-level verification with temporal and clinical reasoning, including authoritative reference checks.
• Stage 4: Rigorous dual annotation, reconciliation, and adjudication, yielding high inter-annotator agreement (NER: 0.897, labeling: 0.888).

EHR-Inspector: Framework for Automated Reasoning-Intensive Consistency Verification

EHR-Inspector operationalizes the human annotation paradigm within a tool-augmented LLM-based agent. The key components are:

Figure 3: EHR-Inspector decomposes the verification task into note segmentation/temporal resolution, anchor entity extraction, temporal scope filtering, and tool-augmented consistency assessment.

1. Note Segmentation and Temporal Reference Resolution: Long, multi-event notes are partitioned into topic-coherent segments. Global and segment-level temporal expressions (e.g., “Postoperative Day 3”) are resolved using LLMs to enable accurate mapping to table timestamps.
2. Hierarchical Entity Extraction: Efficient extraction is achieved via a two-stage process: (a) patient-specific filtering of structured items and associated value distributions for in-database alignment, and (b) ontology-guided expansion to capture entities absent from current patient data but semantically linked within EHR schemas.
3. Temporal Scope Filtering: Only those entities temporally within the current hospitalization are retained for verification, using LLM-synthesized temporal cues and a classification into past, active, or planned status.
4. Tool-Augmented Consistency Verification:

To avoid the impracticality of context-packing large, high-frequency tables, EHR-Inspector uses specialized retrieval tools for (a) lexical/semantic item alignment, (b) database exploration and value profiling, and (c) temporal/conditional record retrieval. Verification results are cached and reused within local context windows to minimize redundant queries and resource overhead.

Figure 4: Validation cache schema conceptually illustrating efficient reuse of verification results for colocalized entities.

Benchmark Analysis and Ablative Insights

EHR-Inspector is benchmarked against CheckEHR—the prior state-of-the-art LLM pipeline—across multiple LLM backbones (Gemini 2.5 Flash, Qwen3-32B, GPT-OSS 20B, and MedGemma 27B), under both harsh (strict match) and lenient (clinically plausible reasoning) LLM-as-a-judge evaluations (validated against expert review: 99.46% and 95.35% agreement, respectively).

• Numerical Results: EHR-Inspector substantially outperforms CheckEHR across all models, with especially pronounced gains for advanced models (e.g., Gemini 2.5 Flash: Lenient F1 74.75, Recall 77.36, Precision 72.31 under lenient evaluation).
• Ablation Studies: Note segmentation, temporal scope filtering, and hierarchical entity extraction each contribute significantly to F1 performance. Especially, segmentation reduces verification errors in long, complex notes, and scope filtering is critical for balancing recall/precision through suppression of spurious entity extractions.

Figure 5: Error analysis across columns and entities reveals that time errors dominate, most inconsistencies are localized to single columns, and the pattern of discrepancy types varies by note category.

Additionally:

• Human annotators optimize tool usage with compact, efficient traces, while model-based agents demonstrate broader and more exploratory (yet often less focused) tool strategies.
• Robustness is demonstrated on external datasets (EHRCon) and under perturbations (schema changes, dataset shifts), with only modest performance degradation, indicating generalization capacity beyond prior knowledge of specific database schemas.

Implications, Limitations, and Future Developments

EHR-ReasonCon sets a new bar for the empirical evaluation and development of automated agents capable of sophisticated, clinical-grade consistency auditing across structured and unstructured EHR data. The multi-faceted reasoning requirements and annotation standards bridge clinical reality with algorithmic modeling rigor, exposing limitations of prior string or surface-comparison approaches.

The demonstration of tool-augmented LLMs that follow granular, protocolized workflows opens the door for increasingly reliable and scalable auditing/asynchronous review within EHR systems. However, the framework does have some limitations:

• Dataset Scope: The data volume and variety, while larger and more heterogeneous than previous efforts, are still limited; rare edge cases and specialty scenarios may be underrepresented.
• Model-Bias in Judgment: The LLM-as-a-judge paradigm, although highly scalable and consistent with expert consensus, may still introduce subtle biases, especially in ambiguous or rare-case scenarios.
• Human/Agent Tool Usage: LLM-driven agents often follow less optimized tool paths than expert annotators, warranting future work on adaptive, data-driven toolset orchestration or learning optimal verification strategies.

Anticipated advances include richer expert-in-the-loop evaluation pipelines, broader dataset expansion with greater clinical variety, and integration of external knowledge sources for improved semantic and interpretive reasoning.

Conclusion

The EHR-ReasonCon benchmark, along with the EHR-Inspector framework, advances the field of automated EHR auditing by foregrounding the essential role of nuanced, clinically informed, and temporally aware reasoning in note–table consistency verification. The associated dataset and methods create new opportunities for both robust empirical evaluation and the development of more reliable, trustworthy clinical NLP systems, with implications for patient safety, automated chart review, and medico-legal informatics governance.

Figure 1: Schematic examples demonstrate the necessity of reasoning-intensive verification—beyond value matching—to assure consistency between narrative documentation and structured EHRs.

Figure 2: Multi-stage annotation pipeline emphasizing expert-informed tool-supported evidence retrieval and quality control.

Figure 3: EHR-Inspector framework modules for context segmentation, hierarchical anchor extraction, temporal scope filtering, and tool-augmented verification.

Figure 5: Distribution of inconsistency types, entity-level mismatch patterns, and note-type-specific error distributions, emphasizing temporal reasoning as the central challenge.