EHR-ReasonCon: Benchmark for Clinical Consistency

• EHR-ReasonCon is a reasoning-intensive benchmark for verifying consistency between clinical notes and structured EHR tables, overcoming the limitations of surface-level matching.
• It leverages expert-guided annotations on MIMIC-III and employs an LLM-based framework, EHR-Inspector, to segment notes, extract key entities, and resolve temporal references.
• The benchmark emphasizes clinical interpretation, event grouping, and temporal dynamics to ensure documentation integrity and improve patient safety.

EHR-ReasonCon is a reasoning-intensive benchmark for note–table consistency verification between clinical notes and structured EHR tables. Introduced in “Towards Error-Free EHRs: Reasoning-Intensive Consistency Verification Between Clinical Notes and Structured Tables in Electronic Health Records”, it is built on MIMIC-III with expert-guided annotations, comprises 8,048 entities derived from clinical notes, and is paired with EHR-Inspector, an LLM-based framework that segments notes, extracts anchor entities and temporal references, and uses table-exploration tools to verify consistency against structured tables (Kwon et al., 26 May 2026). The benchmark is explicitly motivated by the claim that note–table consistency verification cannot be solved by surface-level matching of numeric values or simple events, because real clinical documentation requires reasoning about clinical interpretation, event relations, and temporal changes.

1. Problem definition and conceptual scope

EHR-ReasonCon formalizes reasoning-intensive consistency verification between a clinical note and structured EHR tables. At a high level, the task is to read a clinical note, identify entities in the note that are verifiable against structured tables, determine the relevant time scope for each entity, retrieve structured evidence from the database, and decide whether the note’s claim is consistent or inconsistent with the structured data (Kwon et al., 26 May 2026).

The benchmark’s central claim is that surface matching is insufficient for three reasons. First, notes often state interpreted clinical conclusions, while tables store measurements; the paper’s example is that “anemia on admission” requires checking hemoglobin values against clinical thresholds rather than matching the literal word “anemia.” Second, notes may describe relationships among multiple events, as in microbiology verification that must align specimen, organism, and antibiotic susceptibility together. Third, notes frequently encode temporal dynamics, so verification may require reconstructing a timeline across multiple rows and tables.

This emphasis marks a departure from prior note–table consistency work. EHRCon defined the problem as checking whether an entity mentioned in a clinical note is consistent with the corresponding structured EHR tables and assembled 4,101 entities across 105 clinical notes, but it was described as focusing more on fragmentary surface matches such as values or simple events (Kwon et al., 2024). EHR-ReasonCon is presented as broader and harder, because it includes interpreted clinical states, richer timing fields, and event grouping across multiple rows.

2. Corpus design, schema coverage, and benchmark composition

EHR-ReasonCon is built on MIMIC-III, not MIMIC-IV. The reason given is note diversity, especially physician notes and nursing notes, and the fact that MIMIC-IV shifts note dates differently and omits some note types (Kwon et al., 26 May 2026). The benchmark uses 105 clinical notes: 38 discharge summaries, 33 physician notes, and 34 nursing notes. It is aligned against 14 structured tables from MIMIC-III, including Chartevents, Labevents, Prescriptions, Inputevents_cv / Inputevents_mv, Outputevents, Procedureevents_mv, Microbiologyevents, Diagnoses_icd, Procedures_icd, D_items, D_icd_diagnoses, D_icd_procedures, and D_labitems.

The benchmark contains 8,048 annotated entities extracted from those notes. The appendix is described as emphasizing that EHR-ReasonCon spans 58 columns and goes beyond simple numeric or exact-coded matches. Included entity classes comprise basic numeric / value-based items like lab values and vitals, timing fields such as chart time, start/end time, and admission/discharge-relative timing, procedural metadata like location, route, and duration, microbiology interpretation fields such as organism and susceptibility interpretation, and interpreted clinical states, where note wording must be mapped to underlying structured evidence.

The label distribution is explicitly reported as 5,848 consistent and 2,200 inconsistent out of the 8,048 entities. This imbalance is not incidental: the task is binary at the entity level, but the paper defines inconsistency strictly. If even one column conflicts, or if there is no supporting record, the entity is labeled inconsistent. The result is a benchmark whose verification unit is not a string span alone, but an entity interpreted relative to table alignment, temporal scope, and evidential sufficiency.

3. Annotation protocol, expert involvement, and reliability

The annotation protocol is a major component of EHR-ReasonCon. It was developed with four clinical experts: a board-certified radiation oncologist, a board-certified general surgeon, a resident in anesthesiology, and an EHR technician (Kwon et al., 26 May 2026). The protocol was iteratively refined through pilot studies.

To make annotation systematic and reliable, the authors built eight specialized table-exploration tools, grouped into three categories. The first category is entity-to-table item alignment, consisting of lexical search and semantic search. The second is database exploration and value profiling, consisting of get item value distribution and analyze category trend. The third is temporal and conditional record retrieval, consisting of get item status history, get item value history, analyze value trend, and view general timeline.

Annotation was performed by eight trained annotators. For each note, two annotators independently labeled it, then resolved disagreements. Physicians adjudicated difficult cases, and an independent reviewer checked all 105 notes for dataset-wide consistency. The benchmark reports high reliability: NER F1 / agreement: 0.897 and consistency-label agreement: 0.888.

The consistency label is binary but semantically demanding. Consistent means that the narrative content of the note is supported by structured records. Inconsistent means that the note is contradicted by structured records or lacks supporting evidence in the relevant tables. The appendix states that consistency checking requires reasoning over commonsense mappings, clinical knowledge, temporal interpretation, and event grouping across multiple rows. This is the benchmark’s defining methodological commitment.

4. EHR-Inspector and the verification pipeline

EHR-Inspector is the proposed framework for solving EHR-ReasonCon, and it is explicitly described as mirroring the human annotation process (Kwon et al., 26 May 2026). The pipeline begins with note segmentation. Because clinical notes are long and often contain multiple clinical events, EHR-Inspector first uses an LLM to segment the note into topic-coherent spans. The stated purpose is to reduce context overload and make verification more local and manageable.

The second component is temporal reference extraction. Since segmentation can separate relative time expressions from their anchors, the framework extracts global temporal references from the entire note, such as admission, discharge, and surgery dates. Relative expressions such as “postoperative day 3”, “the next day”, and “three days after surgery” are then resolved to absolute or usable time references.

The third component is anchor entity extraction. The framework extracts only entities that can be aligned to structured tables, called anchor entities. This is done in two complementary ways. In patient-specific extraction, the model is given the subset of table items recorded for that patient, $L_P$, plus value-distribution profiles, and extracts note entities corresponding to those known patient-specific structured items. In ontology-guided hierarchical extraction, the model uses a hierarchical ontology built over the entire EHR item space, with 12 high-level groups, then subgroups, then items. The formal definition is:

$$E_{\mathrm{extract}} = E_{\mathrm{patient}} \cup E_{\mathrm{ontology}}$$

The fourth component is temporal scope filtering. Some extracted entities refer to past history, future plans, or events outside the current hospitalization. To avoid false inconsistencies, EHR-Inspector assigns each extracted entity a temporal status of Past, Active, or Plan, and retains only Active entities for verification:

$$E_{\mathrm{scope}} = \{ e \in E_{\mathrm{extract}} \mid C(e) = \text{Active} \}$$

The fifth component is tool-augmented consistency verification. The model uses the table-exploration tools to search for matching table items, inspect value distributions, retrieve rows by time, retrieve rows by value and time, inspect trends over time, and reuse prior verification evidence via a validation cache. The cache is described as especially useful when multiple extracted entities come from the same clinical observation, as in “Abd: soft, distended.” The final output is a consistent / inconsistent decision for each entity.

5. Evaluation protocol, baselines, and empirical performance

Because strict rule-based comparison is difficult when annotation spans and granularities may differ, EHR-ReasonCon uses LLM-as-a-judge evaluation, with the judge validated against human reviewers (Kwon et al., 26 May 2026). Two evaluator modes are defined. Harsh requires exact agreement with gold labels and achieved 99.46% agreement with author annotations. Lenient accepts clinically reasonable alternatives even if they differ from the exact gold annotation, and its agreement with four practitioners was 95.35%. The paper also checks evaluator bias using GPT-5 as an alternate judge and reports similar results.

The main comparison is against CheckEHR, the prior state-of-the-art framework for similar note–table verification (Kwon et al., 2024). Across multiple backbones, EHR-Inspector is reported to outperform CheckEHR under both Harsh and Lenient evaluation. The tested backbones are Gemini 2.5 Flash, Qwen3-32B, GPT-OSS 20B, and MedGemma 27B.

For the best-performing backbone in the main table, Gemini 2.5 Flash, EHR-Inspector achieves the following recall/precision values. Under Lenient evaluation: Discharge: 77.36 / 72.31, Physician: 79.58 / 71.81, Nursing: 72.53 / 71.59. Under Harsh evaluation: Discharge: 69.28 / 62.81, Physician: 73.14 / 66.11, Nursing: 65.26 / 62.59. The paper states that the corresponding CheckEHR scores are much lower, especially for recall.

The framework is also reported to generalize beyond the benchmark’s native setting. On EHRCon, EHR-Inspector reaches Lenient F1 86.41, beating CheckEHR by 24.99. On MIMIC-IV, performance shows a modest performance drop, and under perturbed schema conditions the framework still performs well despite unseen table and column names. These results are used to support the claim that EHR-Inspector is not overfit to one exact schema.

6. Component effects, failure modes, and significance within EHR reasoning

The ablation analyses identify several components as especially important. Note segmentation improves stability by reducing long-context confusion, and removing it lowers F1 in both Harsh and Lenient settings. Scope filtering increases precision; removing it increases recall but hurts precision, because the model over-extracts irrelevant entities when temporal scope is not controlled. The proposed entity extraction strategy is reported to outperform NER baselines and CheckEHR’s extraction, especially in balancing recall and precision. Among tool categories, temporal retrieval is singled out as especially valuable, and the paper states that time-related tools matter most for the reasoning-heavy setting (Kwon et al., 26 May 2026).

The paper identifies five error types: premature conclusion error, tool usage error, temporal reasoning error, medical knowledge error, and validation cache error. The dominant issue is premature conclusion, described as the model stopping verification before gathering enough evidence. This is interpreted as a systematic under-search problem. Additional observations are that stronger models have fewer validation cache errors, that MedGemma, despite being medically specialized, does not dominate, and that temporal reasoning remains difficult, especially in long, interleaved clinical narratives.

A further analytical comparison concerns tool usage traces. Humans use fewer, more stable tool paths, whereas LLMs explore a broader, more dispersed set of tool sequences. The authors note that human annotators often use optimized and conservative tool strategies, while the LLM tends to wander more and sometimes conclude too early. This distinction is central to the benchmark’s interpretation of current model limitations: performance is no longer constrained only by entity recognition or table lookup, but by verification planning and evidential discipline.

The benchmark’s broader significance is explicitly linked to patient safety, documentation integrity, downstream decision support, legal and administrative correctness, and preventing propagation of EHR errors. A plausible implication is that EHR-ReasonCon addresses the same structural gap identified in broader analyses of EHR-integrated clinical AI: many systems consume longitudinal EHR data but preserve limited support for explicit temporal reasoning across patient histories, and evaluation remains dominated by predictive performance rather than longitudinal interpretability (Yi et al., 7 Jun 2026). In that sense, EHR-ReasonCon treats note–table verification not as a narrow extraction task, but as a benchmark for clinically grounded reasoning over structured evidence, temporal scope, and documentation semantics.