EHRStruct: Structured EHR Benchmark

• EHRStruct is a benchmark framework for evaluating structured EHR tasks using relational, coded, and numeric clinical data.
• It defines 11 tasks with 2,200 samples from Synthea and eICU, supporting zero-shot and few-shot LLM evaluations.
• Empirical findings reveal that general LLMs outperform medical models, while code-augmented methods like EHRMaster enhance deterministic table reasoning.

EHRStruct is a benchmark and experimental framework for evaluating LLMs on structured Electronic Health Record (EHR) tasks, where the input is relational, coded, and numeric clinical data rather than free-text notes (Yang et al., 11 Nov 2025). It was introduced to address the absence of standardized evaluation frameworks and clearly defined tasks for structured EHR reasoning, and it defines 11 representative tasks with 2,200 task-specific evaluation samples derived from Synthea and the eICU Collaborative Research Database (Yang et al., 11 Nov 2025). In adjacent literature, the term “EHRStruct” is also used more loosely to denote explicit structural modeling of EHRs, including patient-centric blockchain architectures (Kenaza et al., 24 Feb 2026), hyperbolic question-answering over longitudinal visits (Liu et al., 22 Apr 2026), tokenized synthetic EHR generation (Karami et al., 2024), and hierarchical prototype learning across codes, visits, and patients (Cai et al., 23 Aug 2025). This broader usage suggests that EHRStruct denotes both a specific benchmark and a recurring research concern: how to represent, manipulate, and reason over structured EHR data with fidelity to its relational, temporal, and coded organization.

1. Definition and terminological scope

In its canonical sense, EHRStruct is the framework introduced in “EHRStruct: A Comprehensive Benchmark Framework for Evaluating LLMs on Structured Electronic Health Record Tasks” (Yang et al., 11 Nov 2025). The benchmark targets structured EHR data stored in relational tables, and its motivating question is how well LLMs can reason over such data when patient information is expressed through rows, columns, codes, measurements, and timestamps rather than narrative clinical notes (Yang et al., 11 Nov 2025). The framework therefore evaluates not only retrieval and arithmetic over tables, but also clinically oriented inference such as mortality prediction, disease prediction, clinical identification, and medication recommendation (Yang et al., 11 Nov 2025).

The paper positions EHRStruct against a fragmented research landscape in which prior studies typically examine one or two isolated tasks, use incompatible datasets and serialization schemes, and provide limited analysis of whether failures arise from table understanding, numeric reasoning, temporal reasoning, or missing clinical knowledge (Yang et al., 11 Nov 2025). EHRStruct addresses this by fixing the data sources, task definitions, prompt structures, and evaluation protocol, and by organizing tasks according to scenario and cognitive level (Yang et al., 11 Nov 2025).

At the same time, multiple other works use “EHRStruct” as a descriptive label for explicit EHR structuring. A secure blockchain architecture for patient-controlled sharing describes a concrete structural blueprint with four layers, actor roles, on-chain metadata, and off-chain IPFS storage (Kenaza et al., 24 Feb 2026). HypEHR treats EHRs as hierarchical trajectories of visits and codes and embeds them in hyperbolic space for EHR question answering (Liu et al., 22 Apr 2026). SynEHRgy defines a unified tokenization scheme that maps covariates, outcomes, ICD codes, and irregularly sampled time series into a single decoder-only transformer sequence (Karami et al., 2024). ProtoEHR formalizes EHRs as a hierarchy of medical codes, hospital visits, and patients, and augments each level with prototypes (Cai et al., 23 Aug 2025). This suggests that, beyond the proper noun of the benchmark, EHRStruct also names a broader structural perspective on EHR computation.

2. Structured EHR representation and task taxonomy

Within the benchmark, “structured EHR” denotes data stored in relational tables whose rows correspond to clinical events or records and whose columns encode fields such as patient identifiers, demographics, dates, units, coded diagnoses, measurements, or billing variables (Yang et al., 11 Nov 2025). Representative fields include ID, RACE, GENDER, INCOME, DATE, DESCRIPTION, VALUE, UNITS, START, STOP, SYSTEM, [CODE](https://www.emergentmind.com/topics/karpathy-agent-code), admissionweight, dischargeweight, cost, and tax (Yang et al., 11 Nov 2025). These tables are serialized into text for LLM input using plain text conversion, special character separation, graph-structured representation, and natural language description (Yang et al., 11 Nov 2025).

EHRStruct classifies tasks along two axes. The first is Task Scenario, which separates Data-Driven tasks solvable from tables alone from Knowledge-Driven tasks that require external clinical knowledge or interpretation (Yang et al., 11 Nov 2025). The second is Task Level, which distinguishes Understanding from Reasoning (Yang et al., 11 Nov 2025). The benchmark instantiates these axes through 6 task categories and 11 task IDs (Yang et al., 11 Nov 2025).

Category	Task IDs	Function
Information Retrieval	D-U1, D-U2	Data filtering based on field conditions
Data Aggregation	D-R1, D-R2, D-R3	Count, Average, Sum over numeric fields
Arithmetic Computation	D-R4, D-R5	Arithmetic reasoning over numeric field trends
Clinical Identification	K-U1	Clinical code identification
Diagnostic Assessment	K-R1, K-R2	Mortality prediction and disease prediction
Treatment Planning	K-R3	Medication recommendation

The Data-Driven tasks emulate cohort selection and arithmetic over structured fields. D-U1 and D-U2 require filtering rows or patients by conjunctions of field conditions, such as demographic attributes or discharge status (Yang et al., 11 Nov 2025). D-R1, D-R2, and D-R3 require computing counts, averages, or sums over repeated observations for a patient (Yang et al., 11 Nov 2025). D-R4 and D-R5 ask for explicit arithmetic combinations such as differences or totals of numeric fields (Yang et al., 11 Nov 2025). These tasks isolate table parsing, value selection, and deterministic computation.

The Knowledge-Driven tasks are qualitatively different. K-U1 asks whether a patient has ever had a given condition based on SNOMED-CT codes (Yang et al., 11 Nov 2025). K-R1 targets mortality prediction from structured observations and social or physiologic indicators (Yang et al., 11 Nov 2025). K-R2 covers disease prediction and multi-disease classification from patient-level structured profiles or laboratory values (Yang et al., 11 Nov 2025). K-R3 evaluates medication recommendation, including binary treatment suitability and multi-drug prescription decisions (Yang et al., 11 Nov 2025). The benchmark therefore spans both low-level manipulation of relational tables and higher-level clinical inference.

This taxonomy aligns with broader structured-EHR research. MIMIC-style EHRs are commonly modeled as normalized relational schemas with time-stamped event tables linked through patient and stay identifiers, where analytic correctness depends on distinguishing units such as patient, admission, and ICU stay (Alkan et al., 16 Jan 2025). HypEHR formalizes input as a natural language question and a patient’s visit history, where each visit is a set of diagnosis codes, procedure codes, drug codes, time stamps, and other structured fields (Liu et al., 22 Apr 2026). ProtoEHR likewise models EHRs as a hierarchy from codes to visits to patients (Cai et al., 23 Aug 2025). These formulations reinforce the benchmark’s assumption that structured EHR reasoning cannot be reduced to generic text generation without preserving relational and temporal structure.

3. Benchmark construction and evaluation protocol

EHRStruct is built from two public EHR sources: Synthea, a synthetic EHR generator, and the eICU Collaborative Research Database, a real-world multi-center ICU database (Yang et al., 11 Nov 2025). For each of the 11 tasks, the authors create 100 evaluation samples per task per dataset, yielding the benchmark total of 2,200 instances (Yang et al., 11 Nov 2025). Table selection is performed jointly by computer science researchers and a medical expert, and the process identifies the relevant tables and fields for each task (Yang et al., 11 Nov 2025). GPT-4o is used to generate question-answer pairs conditioned on the task definition, table schema, and sampled table content, and these outputs are then validated by medical reviewers and technical reviewers (Yang et al., 11 Nov 2025).

Each task instance consists of a task-specific instruction and one or more serialized tables (Yang et al., 11 Nov 2025). The prompt specifies the expected output format, such as an ID list, a scalar answer, or a binary label, enabling exact automated scoring (Yang et al., 11 Nov 2025). The benchmark primarily evaluates zero-shot performance, but it also studies few-shot prompting with 1-shot, 3-shot, and 5-shot examples and LoRA-based fine-tuning on Qwen-7B using 30 training Q–A table pairs per task (Yang et al., 11 Nov 2025).

The evaluation metrics follow the task split. For Data-Driven tasks, the benchmark uses Accuracy (ACC) (Yang et al., 11 Nov 2025). For Knowledge-Driven tasks, it uses AUC (Area Under ROC Curve) and notes that, for binary labels, AUC is equivalent to balanced accuracy defined as the mean of sensitivity and specificity (Yang et al., 11 Nov 2025). This metric choice reflects the class-imbalance concerns typical in medical prediction settings (Yang et al., 11 Nov 2025).

The model suite is unusually broad. EHRStruct evaluates 20 LLMs, including commercial general-purpose models such as GPT-3.5 Turbo, GPT-4.1, Gemini 1.5, Gemini 2.0, and Gemini 2.5; large open-source general models such as DeepSeek-V2.5, DeepSeek-V3, and several Qwen variants; and medical LLMs including Huatuo, HEAL, Meditron-7B, MedAlpaca-13B, JMLR, PMC-LLaMA-13B, Med42-70B, Apollo, and CancerLLM (Yang et al., 11 Nov 2025). It also evaluates 11 LLM-based enhancement methods, including C.L.E.A.R., TaT, TableMaster, TIDE, E⁵, GraphOTTER, H-STAR, Table-R1, LLM4Healthcare, DeLLiriuM, and EnsembleLLM (Yang et al., 11 Nov 2025).

This benchmark design makes EHRStruct unusual among structured-EHR studies. Other works in the provided literature define concrete architectures or predictive models, but not unified multi-task LLM evaluations. HypEHR evaluates on two MIMIC-IV-based EHR-QA benchmarks with four answer types—Boolean, Concept, Numerical, and Integer count (Liu et al., 22 Apr 2026). SynEHRgy evaluates synthetic data quality using fidelity, utility, and privacy metrics on MIMIC-III (Karami et al., 2024). ProtoEHR evaluates five predictive tasks on MIMIC-III and MIMIC-IV (Cai et al., 23 Aug 2025). EHRStruct instead standardizes a cross-model, cross-task evaluation space specifically for structured-table reasoning.

4. Empirical findings and EHRMaster

The central empirical result is that current LLM performance on structured EHR tasks is highly uneven. According to the benchmark, general LLMs outperform medical LLMs across nearly all tasks and both datasets (Yang et al., 11 Nov 2025). Many medical LLMs frequently fail to produce valid outputs on Knowledge-Driven tasks, and even on Data-Driven tasks their accuracies are often low (Yang et al., 11 Nov 2025). By contrast, large commercial general models perform substantially better.

The strongest baseline results come from Gemini-2.5 and GPT-4.1 (Yang et al., 11 Nov 2025). On Synthea, Gemini-2.5 reaches 98% ACC on D-U1, 58% on D-U2, 92% on D-R1, 82% on D-R2, 83% on D-R3, and 100% accuracy on both D-R4 and D-R5 (Yang et al., 11 Nov 2025). On eICU, Gemini-2.5 reaches 95% on D-U1, 84% on D-U2, 95% on D-R1, and produces the top AUCs on K-R2 (60.3) and K-R3 (62.1) while also performing strongly on K-R1 (61.1) (Yang et al., 11 Nov 2025). GPT-4.1 is consistently second-best on many Data-Driven tasks and obtains the top K-R1 result on eICU with 63.3 AUC (Yang et al., 11 Nov 2025).

The paper further shows that Knowledge-Driven tasks are substantially harder than Data-Driven ones (Yang et al., 11 Nov 2025). Even strong general LLMs frequently remain only slightly above random on Knowledge-Driven AUCs, which indicates that the bottleneck is not merely table serialization but also missing clinical reasoning or external knowledge (Yang et al., 11 Nov 2025). The input format analysis supports this interpretation: natural language description significantly helps Data-Driven reasoning, and graph-structured representations help Data-Driven understanding, but no input format consistently improves Knowledge-Driven tasks (Yang et al., 11 Nov 2025).

Few-shot prompting improves performance in a limited but systematic way. 1-shot and 3-shot prompting almost always improve over zero-shot, especially for Knowledge-Driven tasks, while 5-shot often produces no further gain or slight degradation (Yang et al., 11 Nov 2025). Fine-tuning experiments on Qwen-7B show that both single-task fine-tuning and multi-task fine-tuning substantially improve performance, with multi-task fine-tuning consistently outperforming single-task and giving particularly large gains on Data-Driven Reasoning tasks such as D-R2 and D-R3 (Yang et al., 11 Nov 2025).

To respond to the benchmark findings, the authors introduce EHRMaster, a code-augmented method built on top of an LLM (Yang et al., 11 Nov 2025). EHRMaster has three stages: Solution Planning, Concept Alignment, and Adaptive Execution (Yang et al., 11 Nov 2025). The planning stage converts the instruction into a natural-language reasoning plan (Yang et al., 11 Nov 2025). The alignment stage maps plan concepts to actual table columns and values without hallucinating columns or values (Yang et al., 11 Nov 2025). The execution stage decides whether to solve the task through code execution over a Pandas DataFrame or through direct reasoning (Yang et al., 11 Nov 2025). This design treats the LLM as planner and code generator rather than as a calculator.

EHRMaster dominates the benchmark’s Data-Driven tasks. On Gemini-1.5, it reaches 100% on D-U1, D-U2, D-R4, and D-R5, and at least 94% on D-R1–3 (Yang et al., 11 Nov 2025). On Gemini-2.0 and Gemini-2.5, it again achieves near-perfect or perfect performance on almost all Data-Driven tasks (Yang et al., 11 Nov 2025). Its Knowledge-Driven gains are more uneven: for example, on Gemini-1.5, K-U1 AUC rises to 89 from 57, while some K-R improvements are small (Yang et al., 11 Nov 2025). On Gemini-2.5, K-R3 rises from 58.4 to 69.2 AUC, but K-R1 and K-R2 may decline slightly relative to some baselines (Yang et al., 11 Nov 2025). The results therefore indicate that code augmentation is highly effective for deterministic table reasoning but does not fully solve clinical knowledge limitations.

5. Relation to broader structured EHR research

EHRStruct belongs to a wider research program in which the central issue is not merely language modeling but the structural form of EHR data. Several works in the provided corpus instantiate this concern differently.

A secure access-control architecture proposes a patient-centric, permissioned-blockchain system with four logical layers—Participant Enrollment Layer, Data Collection Layer, Data Storage Layer, and Data Sharing Layer—and stores full EHR documents off-chain in provider databases and IPFS, while storing metadata, encrypted pointers, keys, authorizations, and audit data on-chain (Kenaza et al., 24 Feb 2026). The architecture uses Hyperledger Fabric, RAFT-based ordering service, and smart contracts named Owner, Registration, and EHR (Kenaza et al., 24 Feb 2026). This is an infrastructural interpretation of EHRStruct, where structure concerns identities, permissions, metadata, and audit trails rather than LLM evaluation.

At the modeling level, HypEHR addresses EHR question answering by embedding codes, visits, and questions in hyperbolic space (Liu et al., 22 Apr 2026). It pretrains a patient encoder using next-visit diagnosis prediction and hierarchy-aware regularization tied to the ICD ontology, and on two MIMIC-IV-based EHR-QA benchmarks it approaches LLM-based methods with about 22M parameters (Liu et al., 22 Apr 2026). The paper explicitly argues that clinical ontologies and patient trajectories are hierarchical and therefore better represented in the Lorentzian hyperboloid model than in Euclidean space (Liu et al., 22 Apr 2026). This is a geometric interpretation of EHRStruct, where structure means ontology depth, visit hierarchy, and question-conditioned traversal of patient history.

SynEHRgy offers a generative interpretation. It defines structured inpatient EHRs as mixed-type objects containing covariates, outcomes, ICD-9 diagnosis and procedure codes, and irregularly sampled time series (Karami et al., 2024). Its main contribution is a unified tokenization scheme with special structure tokens such as <s>, </visit>, </covars>, </labels>, and </ts>, variable-specific quantized value tokens, and discretized time-interval tokens (Karami et al., 2024). The resulting vocabulary contains 5,127 unique tokens, and the model is a small GPT-2–style decoder-only transformer with 4 layers, 4 attention heads, embedding dimension 384, and context length 1024 (Karami et al., 2024). Here EHRStruct denotes a sequence design that translates heterogeneous, irregular clinical data into a flat autoregressive token stream.

ProtoEHR contributes a predictive and interpretability-oriented interpretation. It formalizes EHR data as a hierarchy of codes, visits, and patients and builds a medical knowledge graph from LLM-extracted semantic relations among medical codes (Cai et al., 23 Aug 2025). It then combines CompGCN, prototype-based encoders at each level, a patient-level Transformer, and hierarchical fusion to predict mortality, readmission, length of stay, drug recommendation, and phenotype (Cai et al., 23 Aug 2025). On MIMIC-III and MIMIC-IV, it is reported as best or second-best on 24/24 metrics across the evaluated tasks and datasets (Cai et al., 23 Aug 2025). This suggests that explicit multi-level structure remains useful even outside LLM-centric evaluation settings.

More classical relational interpretations also persist. A chapter centered on MIMIC-III describes EHRs as longitudinal, event-based records with irregular sampling, mixed structured and unstructured data types, and a normalized schema of tables such as PATIENTS, ADMISSIONS, ICUSTAYS, CHARTEVENTS, LABEVENTS, DIAGNOSES_ICD, and PROCEDURES_ICD (Alkan et al., 16 Jan 2025). A separate knowledge-graph study constructs semantic embedding vectors and a latent graphical block model from EHR co-occurrence statistics to cluster near-synonymous codes and infer conditional dependency structure among code groups (Lu et al., 2023). These works reinforce a plausible implication: the benchmark EHRStruct emerged in a field where “structure” already had multiple operational meanings—relational, ontological, sequential, graph-theoretic, and architectural.

6. Limitations, controversies, and future directions

The benchmark paper itself identifies several limitations. EHRStruct currently relies on only two datasets, namely Synthea and eICU, which leaves out outpatient care, pediatrics, non-ICU inpatient settings, other specialties, and non-North-American settings (Yang et al., 11 Nov 2025). Its 11 tasks across 6 categories are broad but not exhaustive, and the benchmark does not yet cover more complex iterative treatment planning, richer longitudinal disease trajectory modeling, or multimodal settings combining structured fields with clinical notes or imaging (Yang et al., 11 Nov 2025). The framework also intentionally isolates structured tasks and therefore does not evaluate joint reasoning over structured and unstructured data (Yang et al., 11 Nov 2025).

The empirical results expose a further misconception: domain-specific medical LLMs are not automatically superior on structured clinical data (Yang et al., 11 Nov 2025). In this benchmark, general LLMs consistently outperform current medical LLMs, and many medical models fail even to produce valid outputs on some Knowledge-Driven tasks (Yang et al., 11 Nov 2025). Another misconception is that better serialization alone solves structured reasoning; the format analysis shows that improved text formatting helps mostly on Data-Driven tasks, whereas Knowledge-Driven performance remains weak regardless of format (Yang et al., 11 Nov 2025). This suggests that table understanding and clinical knowledge are separable bottlenecks.

A broader controversy concerns whether structured EHR reasoning should be handled by generic LLM prompting at all. EHRMaster’s success on Data-Driven tasks shows that explicit planning, concept alignment, and tool use can nearly solve deterministic table manipulation (Yang et al., 11 Nov 2025). HypEHR, by contrast, argues that a compact geometry-aware model can approach LLM-based pipelines with far fewer parameters (Liu et al., 22 Apr 2026). ProtoEHR shows that explicit code–visit–patient hierarchy and prototype structure improve predictive performance and interpretability (Cai et al., 23 Aug 2025). SynEHRgy shows that a carefully designed tokenization plus a small GPT-like model can generate realistic mixed-type EHRs (Karami et al., 2024). A plausible implication is that future “EHRStruct” systems will not converge on a single paradigm, but instead combine benchmarked LLM reasoning with structure-specific back ends.

The future directions stated in the benchmark paper include extending EHRStruct to iterative, adaptive treatment planning that incorporates real-time patient responses, expanding to more datasets and modalities, and evaluating more agentic workflows and tool-augmented pipelines (Yang et al., 11 Nov 2025). Related work points to additional directions: blockchain-based governance and interoperability for patient-controlled sharing (Kenaza et al., 24 Feb 2026), hyperbolic representations for ontology-aligned QA (Liu et al., 22 Apr 2026), unified sequence representations for synthetic generation (Karami et al., 2024), and hierarchical prototype systems for interpretable prediction (Cai et al., 23 Aug 2025). Taken together, these studies indicate that EHRStruct is best understood not only as a benchmark title, but as a focal point in contemporary research on how structured clinical data should be represented, evaluated, and operationalized.