---
title: 'EHRStruct: Structured EHR Benchmark'
url: https://www.emergentmind.com/topics/ehrstruct
type: topic
---

# EHRStruct: Structured EHR Benchmark

EHRStruct is a benchmark and experimental framework for evaluating large language models on **structured Electronic Health Record (EHR) tasks**, where the input is relational, coded, and numeric clinical data rather than free-text notes [2511.08206]. It was introduced to address the absence of standardized evaluation frameworks and clearly defined tasks for structured EHR reasoning, and it defines **11 representative tasks** with **2,200 task-specific evaluation samples** derived from **Synthea** and the **eICU Collaborative Research Database** [2511.08206]. In adjacent literature, the term “EHRStruct” is also used more loosely to denote explicit structural modeling of EHRs, including patient-centric blockchain architectures [2602.20830], hyperbolic question-answering over longitudinal visits [2604.21027], tokenized synthetic EHR generation [2411.13428], and hierarchical prototype learning across codes, visits, and patients [2508.18313]. This broader usage suggests that EHRStruct denotes both a specific benchmark and a recurring research concern: how to represent, manipulate, and reason over structured EHR data with fidelity to its relational, temporal, and coded organization.

## 1. Definition and terminological scope

In its canonical sense, EHRStruct is the framework introduced in “EHRStruct: A Comprehensive Benchmark Framework for Evaluating Large Language Models on Structured Electronic Health Record Tasks” [2511.08206]. The benchmark targets **structured EHR data stored in relational tables**, and its motivating question is how well LLMs can reason over such data when patient information is expressed through rows, columns, codes, measurements, and timestamps rather than narrative clinical notes [2511.08206]. The framework therefore evaluates not only retrieval and arithmetic over tables, but also clinically oriented inference such as mortality prediction, disease prediction, clinical identification, and medication recommendation [2511.08206].

The paper positions EHRStruct against a fragmented research landscape in which prior studies typically examine one or two isolated tasks, use incompatible datasets and serialization schemes, and provide limited analysis of whether failures arise from table understanding, numeric reasoning, temporal reasoning, or missing clinical knowledge [2511.08206]. EHRStruct addresses this by fixing the data sources, task definitions, prompt structures, and evaluation protocol, and by organizing tasks according to **scenario** and **cognitive level** [2511.08206].

At the same time, multiple other works use “EHRStruct” as a descriptive label for explicit EHR structuring. A secure blockchain architecture for patient-controlled sharing describes a concrete structural blueprint with four layers, actor roles, on-chain metadata, and off-chain IPFS storage [2602.20830]. HypEHR treats EHRs as hierarchical trajectories of visits and codes and embeds them in hyperbolic space for EHR question answering [2604.21027]. SynEHRgy defines a unified tokenization scheme that maps covariates, outcomes, ICD codes, and irregularly sampled time series into a single decoder-only transformer sequence [2411.13428]. ProtoEHR formalizes EHRs as a hierarchy of **medical codes**, **hospital visits**, and **patients**, and augments each level with prototypes [2508.18313]. This suggests that, beyond the proper noun of the benchmark, EHRStruct also names a broader structural perspective on EHR computation.

## 2. Structured EHR representation and task taxonomy

Within the benchmark, “structured EHR” denotes data stored in relational tables whose rows correspond to clinical events or records and whose columns encode fields such as patient identifiers, demographics, dates, units, coded diagnoses, measurements, or billing variables [2511.08206]. Representative fields include `ID`, `RACE`, `GENDER`, `INCOME`, `DATE`, `DESCRIPTION`, `VALUE`, `UNITS`, `START`, `STOP`, `SYSTEM`, `CODE`, `admissionweight`, `dischargeweight`, `cost`, and `tax` [2511.08206]. These tables are serialized into text for LLM input using **plain text conversion**, **special character separation**, **graph-structured representation**, and **natural language description** [2511.08206].

EHRStruct classifies tasks along two axes. The first is **Task Scenario**, which separates **Data-Driven** tasks solvable from tables alone from **Knowledge-Driven** tasks that require external clinical knowledge or interpretation [2511.08206]. The second is **Task Level**, which distinguishes **Understanding** from **Reasoning** [2511.08206]. The benchmark instantiates these axes through **6 task categories** and **11 task IDs** [2511.08206].

| Category | Task IDs | Function |
|---|---|---|
| Information Retrieval | D-U1, D-U2 | Data filtering based on field conditions |
| Data Aggregation | D-R1, D-R2, D-R3 | Count, Average, Sum over numeric fields |
| Arithmetic Computation | D-R4, D-R5 | Arithmetic reasoning over numeric field trends |
| Clinical Identification | K-U1 | Clinical code identification |
| Diagnostic Assessment | K-R1, K-R2 | Mortality prediction and disease prediction |
| Treatment Planning | K-R3 | Medication recommendation |

The Data-Driven tasks emulate cohort selection and arithmetic over structured fields. D-U1 and D-U2 require filtering rows or patients by conjunctions of field conditions, such as demographic attributes or discharge status [2511.08206]. D-R1, D-R2, and D-R3 require computing counts, averages, or sums over repeated observations for a patient [2511.08206]. D-R4 and D-R5 ask for explicit arithmetic combinations such as differences or totals of numeric fields [2511.08206]. These tasks isolate table parsing, value selection, and deterministic computation.

The Knowledge-Driven tasks are qualitatively different. K-U1 asks whether a patient has ever had a given condition based on **SNOMED-CT** codes [2511.08206]. K-R1 targets mortality prediction from structured observations and social or physiologic indicators [2511.08206]. K-R2 covers disease prediction and multi-disease classification from patient-level structured profiles or laboratory values [2511.08206]. K-R3 evaluates medication recommendation, including binary treatment suitability and multi-drug prescription decisions [2511.08206]. The benchmark therefore spans both low-level manipulation of relational tables and higher-level clinical inference.

This taxonomy aligns with broader structured-EHR research. MIMIC-style EHRs are commonly modeled as normalized relational schemas with time-stamped event tables linked through patient and stay identifiers, where analytic correctness depends on distinguishing units such as patient, admission, and ICU stay [2501.09640]. HypEHR formalizes input as a natural language question and a patient’s visit history, where each visit is a set of diagnosis codes, procedure codes, drug codes, time stamps, and other structured fields [2604.21027]. ProtoEHR likewise models EHRs as a hierarchy from codes to visits to patients [2508.18313]. These formulations reinforce the benchmark’s assumption that structured EHR reasoning cannot be reduced to generic text generation without preserving relational and temporal structure.

## 3. Benchmark construction and evaluation protocol

EHRStruct is built from two public EHR sources: **Synthea**, a synthetic EHR generator, and the **eICU Collaborative Research Database**, a real-world multi-center ICU database [2511.08206]. For each of the **11 tasks**, the authors create **100 evaluation samples per task per dataset**, yielding the benchmark total of **2,200 instances** [2511.08206]. Table selection is performed jointly by computer science researchers and a medical expert, and the process identifies the relevant tables and fields for each task [2511.08206]. GPT-4o is used to generate question-answer pairs conditioned on the task definition, table schema, and sampled table content, and these outputs are then validated by medical reviewers and technical reviewers [2511.08206].

Each task instance consists of a task-specific instruction and one or more serialized tables [2511.08206]. The prompt specifies the expected output format, such as an ID list, a scalar answer, or a binary label, enabling exact automated scoring [2511.08206]. The benchmark primarily evaluates **zero-shot** performance, but it also studies **few-shot** prompting with **1-shot**, **3-shot**, and **5-shot** examples and **LoRA-based fine-tuning** on Qwen-7B using **30 training Q–A table pairs per task** [2511.08206].

The evaluation metrics follow the task split. For Data-Driven tasks, the benchmark uses **Accuracy (ACC)** [2511.08206]. For Knowledge-Driven tasks, it uses **AUC (Area Under ROC Curve)** and notes that, for binary labels, AUC is equivalent to balanced accuracy defined as the mean of sensitivity and specificity [2511.08206]. This metric choice reflects the class-imbalance concerns typical in medical prediction settings [2511.08206].

The model suite is unusually broad. EHRStruct evaluates **20 LLMs**, including commercial general-purpose models such as **GPT-3.5 Turbo**, **GPT-4.1**, **Gemini 1.5**, **Gemini 2.0**, and **Gemini 2.5**; large open-source general models such as **DeepSeek-V2.5**, **DeepSeek-V3**, and several **Qwen** variants; and medical LLMs including **Huatuo**, **HEAL**, **Meditron-7B**, **MedAlpaca-13B**, **JMLR**, **PMC-LLaMA-13B**, **Med42-70B**, **Apollo**, and **CancerLLM** [2511.08206]. It also evaluates **11 LLM-based enhancement methods**, including **C.L.E.A.R.**, **TaT**, **TableMaster**, **TIDE**, **E⁵**, **GraphOTTER**, **H-STAR**, **Table-R1**, **LLM4Healthcare**, **DeLLiriuM**, and **EnsembleLLM** [2511.08206].

This benchmark design makes EHRStruct unusual among structured-EHR studies. Other works in the provided literature define concrete architectures or predictive models, but not unified multi-task LLM evaluations. HypEHR evaluates on **two MIMIC-IV-based EHR-QA benchmarks** with four answer types—Boolean, Concept, Numerical, and Integer count [2604.21027]. SynEHRgy evaluates synthetic data quality using fidelity, utility, and privacy metrics on MIMIC-III [2411.13428]. ProtoEHR evaluates five predictive tasks on MIMIC-III and MIMIC-IV [2508.18313]. EHRStruct instead standardizes a cross-model, cross-task evaluation space specifically for structured-table reasoning.

## 4. Empirical findings and EHRMaster

The central empirical result is that current LLM performance on structured EHR tasks is highly uneven. According to the benchmark, **general LLMs outperform medical LLMs** across nearly all tasks and both datasets [2511.08206]. Many medical LLMs frequently fail to produce valid outputs on Knowledge-Driven tasks, and even on Data-Driven tasks their accuracies are often low [2511.08206]. By contrast, large commercial general models perform substantially better.

The strongest baseline results come from **Gemini-2.5** and **GPT-4.1** [2511.08206]. On **Synthea**, Gemini-2.5 reaches **98% ACC** on D-U1, **58%** on D-U2, **92%** on D-R1, **82%** on D-R2, **83%** on D-R3, and **100% accuracy** on both D-R4 and D-R5 [2511.08206]. On **eICU**, Gemini-2.5 reaches **95%** on D-U1, **84%** on D-U2, **95%** on D-R1, and produces the top AUCs on **K-R2 (60.3)** and **K-R3 (62.1)** while also performing strongly on **K-R1 (61.1)** [2511.08206]. GPT-4.1 is consistently second-best on many Data-Driven tasks and obtains the top **K-R1** result on eICU with **63.3 AUC** [2511.08206].

The paper further shows that **Knowledge-Driven tasks are substantially harder** than Data-Driven ones [2511.08206]. Even strong general LLMs frequently remain only slightly above random on Knowledge-Driven AUCs, which indicates that the bottleneck is not merely table serialization but also missing clinical reasoning or external knowledge [2511.08206]. The input format analysis supports this interpretation: **natural language description** significantly helps Data-Driven reasoning, and **graph-structured representations** help Data-Driven understanding, but **no input format consistently improves Knowledge-Driven tasks** [2511.08206].

Few-shot prompting improves performance in a limited but systematic way. **1-shot** and **3-shot** prompting almost always improve over zero-shot, especially for Knowledge-Driven tasks, while **5-shot** often produces no further gain or slight degradation [2511.08206]. Fine-tuning experiments on Qwen-7B show that both **single-task fine-tuning** and **multi-task fine-tuning** substantially improve performance, with **multi-task fine-tuning consistently outperforming single-task** and giving particularly large gains on Data-Driven Reasoning tasks such as D-R2 and D-R3 [2511.08206].

To respond to the benchmark findings, the authors introduce **EHRMaster**, a **code-augmented method** built on top of an LLM [2511.08206]. EHRMaster has three stages: **Solution Planning**, **Concept Alignment**, and **Adaptive Execution** [2511.08206]. The planning stage converts the instruction into a natural-language reasoning plan [2511.08206]. The alignment stage maps plan concepts to actual table columns and values without hallucinating columns or values [2511.08206]. The execution stage decides whether to solve the task through **code execution** over a Pandas DataFrame or through **direct reasoning** [2511.08206]. This design treats the LLM as planner and code generator rather than as a calculator.

EHRMaster dominates the benchmark’s Data-Driven tasks. On **Gemini-1.5**, it reaches **100%** on **D-U1**, **D-U2**, **D-R4**, and **D-R5**, and at least **94%** on **D-R1–3** [2511.08206]. On **Gemini-2.0** and **Gemini-2.5**, it again achieves near-perfect or perfect performance on almost all Data-Driven tasks [2511.08206]. Its Knowledge-Driven gains are more uneven: for example, on Gemini-1.5, **K-U1 AUC rises to 89** from **57**, while some K-R improvements are small [2511.08206]. On Gemini-2.5, **K-R3** rises from **58.4** to **69.2 AUC**, but **K-R1** and **K-R2** may decline slightly relative to some baselines [2511.08206]. The results therefore indicate that code augmentation is highly effective for deterministic table reasoning but does not fully solve clinical knowledge limitations.

## 5. Relation to broader structured EHR research

EHRStruct belongs to a wider research program in which the central issue is not merely language modeling but the structural form of EHR data. Several works in the provided corpus instantiate this concern differently.

A secure access-control architecture proposes a patient-centric, permissioned-blockchain system with four logical layers—**Participant Enrollment Layer**, **Data Collection Layer**, **Data Storage Layer**, and **Data Sharing Layer**—and stores full EHR documents off-chain in provider databases and **IPFS**, while storing metadata, encrypted pointers, keys, authorizations, and audit data on-chain [2602.20830]. The architecture uses **Hyperledger Fabric**, **RAFT-based ordering service**, and smart contracts named `Owner`, `Registration`, and `EHR` [2602.20830]. This is an infrastructural interpretation of EHRStruct, where structure concerns identities, permissions, metadata, and audit trails rather than LLM evaluation.

At the modeling level, **HypEHR** addresses EHR question answering by embedding codes, visits, and questions in **hyperbolic space** [2604.21027]. It pretrains a patient encoder using **next-visit diagnosis prediction** and **hierarchy-aware regularization** tied to the ICD ontology, and on two MIMIC-IV-based EHR-QA benchmarks it approaches LLM-based methods with about **22M parameters** [2604.21027]. The paper explicitly argues that clinical ontologies and patient trajectories are hierarchical and therefore better represented in the **Lorentzian hyperboloid model** than in Euclidean space [2604.21027]. This is a geometric interpretation of EHRStruct, where structure means ontology depth, visit hierarchy, and question-conditioned traversal of patient history.

**SynEHRgy** offers a generative interpretation. It defines structured inpatient EHRs as mixed-type objects containing **covariates**, **outcomes**, **ICD-9 diagnosis and procedure codes**, and **irregularly sampled time series** [2411.13428]. Its main contribution is a unified tokenization scheme with special structure tokens such as `<s>`, `</visit>`, `</covars>`, `</labels>`, and `</ts>`, variable-specific quantized value tokens, and discretized time-interval tokens [2411.13428]. The resulting vocabulary contains **5,127 unique tokens**, and the model is a small GPT-2–style decoder-only transformer with **4 layers**, **4 attention heads**, **embedding dimension 384**, and **context length 1024** [2411.13428]. Here EHRStruct denotes a sequence design that translates heterogeneous, irregular clinical data into a flat autoregressive token stream.

**ProtoEHR** contributes a predictive and interpretability-oriented interpretation. It formalizes EHR data as a hierarchy of **codes**, **visits**, and **patients** and builds a **medical knowledge graph** from LLM-extracted semantic relations among medical codes [2508.18313]. It then combines **CompGCN**, prototype-based encoders at each level, a patient-level **Transformer**, and hierarchical fusion to predict mortality, readmission, length of stay, drug recommendation, and phenotype [2508.18313]. On MIMIC-III and MIMIC-IV, it is reported as best or second-best on **24/24 metrics** across the evaluated tasks and datasets [2508.18313]. This suggests that explicit multi-level structure remains useful even outside LLM-centric evaluation settings.

More classical relational interpretations also persist. A chapter centered on MIMIC-III describes EHRs as **longitudinal, event-based records** with irregular sampling, mixed structured and unstructured data types, and a normalized schema of tables such as `PATIENTS`, `ADMISSIONS`, `ICUSTAYS`, `CHARTEVENTS`, `LABEVENTS`, `DIAGNOSES_ICD`, and `PROCEDURES_ICD` [2501.09640]. A separate knowledge-graph study constructs semantic embedding vectors and a latent graphical block model from EHR co-occurrence statistics to cluster near-synonymous codes and infer conditional dependency structure among code groups [2305.19997]. These works reinforce a plausible implication: the benchmark EHRStruct emerged in a field where “structure” already had multiple operational meanings—relational, ontological, sequential, graph-theoretic, and architectural.

## 6. Limitations, controversies, and future directions

The benchmark paper itself identifies several limitations. EHRStruct currently relies on only **two datasets**, namely Synthea and eICU, which leaves out outpatient care, pediatrics, non-ICU inpatient settings, other specialties, and non-North-American settings [2511.08206]. Its **11 tasks across 6 categories** are broad but not exhaustive, and the benchmark does not yet cover more complex iterative treatment planning, richer longitudinal disease trajectory modeling, or multimodal settings combining structured fields with clinical notes or imaging [2511.08206]. The framework also intentionally isolates structured tasks and therefore does not evaluate joint reasoning over structured and unstructured data [2511.08206].

The empirical results expose a further misconception: domain-specific medical LLMs are not automatically superior on structured clinical data [2511.08206]. In this benchmark, general LLMs consistently outperform current medical LLMs, and many medical models fail even to produce valid outputs on some Knowledge-Driven tasks [2511.08206]. Another misconception is that better serialization alone solves structured reasoning; the format analysis shows that improved text formatting helps mostly on Data-Driven tasks, whereas Knowledge-Driven performance remains weak regardless of format [2511.08206]. This suggests that table understanding and clinical knowledge are separable bottlenecks.

A broader controversy concerns whether structured EHR reasoning should be handled by generic LLM prompting at all. EHRMaster’s success on Data-Driven tasks shows that explicit planning, concept alignment, and tool use can nearly solve deterministic table manipulation [2511.08206]. HypEHR, by contrast, argues that a compact geometry-aware model can approach LLM-based pipelines with far fewer parameters [2604.21027]. ProtoEHR shows that explicit code–visit–patient hierarchy and prototype structure improve predictive performance and interpretability [2508.18313]. SynEHRgy shows that a carefully designed tokenization plus a small GPT-like model can generate realistic mixed-type EHRs [2411.13428]. A plausible implication is that future “EHRStruct” systems will not converge on a single paradigm, but instead combine benchmarked LLM reasoning with structure-specific back ends.

The future directions stated in the benchmark paper include extending EHRStruct to **iterative, adaptive treatment planning** that incorporates **real-time patient responses**, expanding to more datasets and modalities, and evaluating more **agentic workflows** and tool-augmented pipelines [2511.08206]. Related work points to additional directions: blockchain-based governance and interoperability for patient-controlled sharing [2602.20830], hyperbolic representations for ontology-aligned QA [2604.21027], unified sequence representations for synthetic generation [2411.13428], and hierarchical prototype systems for interpretable prediction [2508.18313]. Taken together, these studies indicate that EHRStruct is best understood not only as a benchmark title, but as a focal point in contemporary research on how structured clinical data should be represented, evaluated, and operationalized.

Source: https://www.emergentmind.com/topics/ehrstruct