EHRMaster: Reliable EHR Task Execution

• EHRMaster is a planning and execution framework that enhances structured EHR task reliability by converting natural language queries into schema-aligned code execution.
• It achieves near-perfect accuracy on Data-Driven tasks and competitive performance on Knowledge-Driven tasks by reducing schema hallucinations and arithmetic errors.
• The framework employs a two-step process of planning then alignment to enforce strict output formats and minimize execution mistakes in processing tabular EHR data.

EHRMaster is a code-augmented, planning-and-execution framework for making LLMs reliable on structured electronic health record tasks. It was introduced alongside EHRStruct, a benchmark that defines 11 representative tasks over structured EHR data, spans Data-Driven and Knowledge-Driven scenarios as well as Understanding and Reasoning levels, and provides 2,200 curated evaluation samples from Synthea and eICU. EHRMaster converts a natural-language question over a serialized EHR table into an explicit plan, aligns that plan to exact schema elements, and then selects either constrained Python/pandas execution over a preloaded DataFrame or direct label-only reasoning, with the goal of reducing schema hallucinations, arithmetic mistakes, and output-format violations (Yang et al., 11 Nov 2025).

1. Benchmark origin and problem setting

EHRMaster emerged from a specific empirical diagnosis: base LLMs and prior LLM-based enhancement methods were not reliably handling structured EHR data. The motivating benchmark, EHRStruct, is described as the first comprehensive benchmark explicitly targeting structured EHR tasks. It standardizes task definitions, input formats, prompting protocols, and evaluation, and evaluates 20 advanced and representative LLMs together with 11 LLM-based enhancement methods across 11 tasks derived from two widely used sources: Synthea and the eICU Collaborative Research Database (Yang et al., 11 Nov 2025).

The benchmark organizes structured EHR work into two scenarios. Data-Driven tasks focus on table-centric filtering, aggregation, and arithmetic over relational records. Knowledge-Driven tasks require clinically grounded identification, mortality prediction, disease prediction based on clinical profiles, or treatment planning from structured evidence. This distinction is central to the design of EHRMaster because it separates problems that are predominantly schema- and arithmetic-bound from problems that remain dependent on clinical inference.

EHRStruct also reports several findings that directly motivate the framework. General LLMs outperform medical LLMs on structured EHR tasks; input format matters; few-shot prompting helps up to 1–3 shots; multi-task finetuning outperforms single-task finetuning; and existing enhancement methods are scenario-specific and fail to generalize across all 11 tasks. A common misconception is therefore that domain branding alone is sufficient for structured EHR competence. The benchmark instead indicates that schema grounding, numeric correctness, and controlled execution are more decisive than nominal medical specialization in this setting (Yang et al., 11 Nov 2025).

2. Core architecture and execution model

EHRMaster is a planning–alignment–execution method. Its defining move is to front-load schema-grounded reasoning before answer generation. Rather than asking an LLM to respond directly to a question over tabular EHR content, the framework first generates a natural-language plan, then rewrites that plan using the exact fields and value vocabularies present in the table, and only then decides whether to execute code or return a direct binary decision (Yang et al., 11 Nov 2025).

The execution substrate is deliberately narrow. The framework uses Python with pandas, with a preloaded DataFrame named df. It does not use SQL in its core execution. Generated code must operate only on df, reference only existing columns, avoid import statements, and assign the final answer to a variable named result. For code columns such as SNOMED-CT or ICD9, explicit conversion to string is enforced before string operations. These constraints are not incidental; they are the operational mechanism by which EHRMaster suppresses schema hallucinations and enforces benchmark-specific answer formatting.

The workflow is task-aware. In Data-Driven/Understanding tasks, planning summarizes target fields and filters. In Data-Driven/Reasoning tasks, planning specifies the operation, such as count, mean, sum, or arithmetic, together with the filtering logic. In Knowledge-Driven/Understanding tasks, planning recognizes the clinical concept and states an exact check over the [CODE](https://www.emergentmind.com/topics/confident-ordinary-differential-editing-code) column constrained to SYSTEM == 'SNOMED-CT'. In Knowledge-Driven/Reasoning tasks, planning outlines clinical logic, such as mortality risk signs, diagnostic cues, or medication appropriateness, and identifies the fields that support the decision. The subsequent Concept Alignment stage rewrites that logic using exact column names and observed values from the table, eliminating synonyms and approximations.

Adaptive Execution is then controlled by a decision rule. If the aligned logic requires filtering, aggregation, or arithmetic across rows or columns, EHRMaster chooses Code Execution. If the task is a binary decision over structured evidence where the benchmark prescribes a textual label and the computation is trivial or qualitative, it chooses Direct Reasoning. This operationalizes program-of-thought: the system first produces a plan, then a schema-aligned plan, and then code or a decision, yielding stepwise supervision without revealing chain-of-thought to the user (Yang et al., 11 Nov 2025).

Temporal and relational dependencies are handled inside the same abstraction. The framework consumes tables serialized as TSV and as a pandas DataFrame. Time and relational dependencies are represented through planning and alignment operations such as locating admission and discharge events, filtering rows by timestamps, or relating rows via shared identifiers, and are then realized through pandas filtering, grouping, and sorting. EHRMaster therefore does not introduce a graph or SQL layer, but it does support cross-row operations through pandas-based execution.

3. Task taxonomy, datasets, and evaluation protocol

EHRStruct’s task space is defined by scenario and cognitive level. The 11 tasks are distributed across Data-Driven understanding and reasoning, and Knowledge-Driven understanding and reasoning. Accuracy is used for all Data-Driven tasks, while AUC is used for all Knowledge-Driven tasks. The paper notes that with binary predictions, AUC is mathematically equivalent to balanced accuracy, and that this choice mitigates class imbalance concerns common in medical data (Yang et al., 11 Nov 2025).

Category	Tasks	Metric
Data-Driven / Understanding	D-U1 filtering based on conditions; D-U2 filtering with complementary conditions	Accuracy
Data-Driven / Reasoning	D-R1 count; D-R2 average; D-R3 sum; D-R4 arithmetic difference; D-R5 arithmetic sum	Accuracy
Knowledge-Driven / Understanding	K-U1 clinical identification	AUC
Knowledge-Driven / Reasoning	K-R1 mortality prediction; K-R2 disease prediction; K-R3 treatment planning	AUC

The benchmark uses two datasets with deliberately different characteristics. Synthea provides synthetic patients with a risk-free privacy profile and standardized tables. eICU provides real-world, multi-institutional ICU data with rich structured schemas. For each of the 11 tasks, 100 evaluation samples are constructed from each dataset, yielding 200 samples per task and 2,200 instances overall. The question–answer pairs are double-validated: medical reviewers assess correctness and plausibility, while technical reviewers assess faithfulness to task definitions and schemas.

Input representation is itself part of the benchmark. Four serialization styles are compared in zero-shot settings: plain text conversion, special character separation, graph-structured representation, and natural language description. Few-shot settings include 0-, 1-, 3-, and 5-shot configurations. Finetuning analyses compare single-task and multi-task LoRA on Qwen-7B, with learning rate 1e-4, 3 epochs, batch size 8, validation split 10%, LoRA rank 8, alpha 32, dropout 0.05, and 30 QA-table training pairs per task, disjoint from evaluation (Yang et al., 11 Nov 2025).

These design choices matter because they fix the interpretive frame in which EHRMaster operates. The method is not evaluated as a free-form clinical assistant; it is evaluated against strict task-specific output contracts such as D-R2: [number] or binary labels. Its architecture is therefore inseparable from the benchmark’s attention to formatting, schema exactness, and reproducibility.

4. Empirical performance and characteristic error profile

The principal empirical result is asymmetric. On Data-Driven tasks, EHRMaster attains near-perfect or perfect accuracy across Gemini backbones and outperforms the best competing enhancement method for most tasks. On Knowledge-Driven tasks, it produces competitive improvements, but gains are mixed, reflecting the difficulty of clinical inference when only table content is available and no explicit external knowledge is injected (Yang et al., 11 Nov 2025).

The quantitative summary on Synthea shows this pattern clearly. With Gemini 1.5, EHRMaster achieves D-U1=100, D-U2=100, D-R1=96, D-R2=96, D-R3=94, D-R4=100, D-R5=100, K-U1=89, K-R1=62.3, K-R2=54.0, and K-R3=54.7, compared with previous SOTA values of 76, 79, 80, 78, 73, 85, 93, 57, 61.3, 56.4, and 54.2, respectively. With Gemini 2.5, it achieves D-U1=100, D-U2=100, D-R1=97, D-R2=95, D-R3=97, D-R4=100, D-R5=100, K-U1=60, K-R1=59.3, K-R2=55.1, and K-R3=69.2, compared with previous SOTA values of 100, 89, 94, 85, 85, 100, 100, 57, 66.3, 61.2, and 58.4 (Yang et al., 11 Nov 2025).

The sensitivity analyses reinforce the benchmark-level findings. Natural-language descriptions boost Data-Driven reasoning, while graph-like prompts help Data-Driven understanding. No universally helpful format emerges for Knowledge-Driven tasks. Few-shot prompting is often best at 1–3 shots, while 5-shot prompting can degrade due to context interference. Multi-task LoRA improves most tasks more than single-task finetuning. These results clarify that EHRMaster’s gains are not solely attributable to stronger prompting; they arise from a tighter coupling of schema grounding and execution.

Its error profile is correspondingly specific. Baseline LLMs exhibit schema hallucinations, arithmetic errors, label-format violations, and knowledge-driven brittleness, particularly in mapping clinical concepts such as SNOMED-CT or synthesizing signals for diagnosis and treatment. EHRMaster strongly reduces schema and arithmetic mistakes through code execution on df. Its remaining weaknesses are concentrated in K-R1, K-R2, and K-R3, where clinically correct heuristics and external medical knowledge are needed and where improvements remain model- and task-dependent. A second common misconception is therefore that controlled code execution is sufficient for full clinical reasoning. The reported evidence does not support that claim (Yang et al., 11 Nov 2025).

5. Expansion into a broader EHR system blueprint

Although EHRMaster is most precisely defined by the EHRStruct method, related syntheses use the name as a broader systems label for interoperable, privacy-aware, multimodal EHR reasoning. In that broader usage, the term denotes not only table reasoning, but also EHR mining, FHIR-native summarization, multimodal query generation, identity linkage, temporal abstraction, privacy preservation, and visualization.

Domain	Representative mechanism	Paper
EHR mining and analytics	OMOP CDM, HL7 FHIR, cohort design, confounding control, feature engineering	(Yadav et al., 2017)
Privacy-aware summarization	FHIR-native retrieval, clinical context package, summaries grounded only in retrieved evidence	(Kazemzadeh et al., 4 Jan 2026)
Multimodal querying	UMLS-based normalization, SBERT/FAISS template retrieval, Text_Func, execution-and-repair loop	(Zhang, 25 Nov 2025)
Data harmonization	variable sheet, variable details sheet, derived-variable registry via recodeflow	(Aminoleslami et al., 2024)
Identity and mortality linkage	centralized MPI; deterministic token linkage to external death data	(Jayatissa et al., 2018, Peralta et al., 2022)
Privacy-preserving learning	federated learning, differential privacy, secure aggregation guidance	(Ganadily et al., 2024)
Faithful summarization	reference revision, metric calibration, sentence-level entity planning	(Adams, 2024)
Immersive visualization	FHIR-centered XR visualization with imaging and AI segmentation	(Marteau et al., 5 Dec 2025)

This broader interpretation is technically coherent because the surrounding literature repeatedly identifies the same infrastructural pressures. EHR mining work emphasizes longitudinal, irregular, sparse, and confounded data; interoperability layers such as OMOP CDM and HL7 FHIR; and the need for careful cohort anchoring, missingness handling, and evaluation design (Yadav et al., 2017). EHRSummarizer extends this logic into a privacy-aware, FHIR-native architecture that retrieves high-yield FHIR R4 resources, normalizes them into a clinical context package, and constrains summarization to evidence present in that package while avoiding diagnostic or treatment recommendations (Kazemzadeh et al., 4 Jan 2026).

A complementary line of work addresses direct information access over mixed structured and unstructured EHR content. TQGen-EHRQuery builds natural-language-to-query generation for tables, radiology reports, and discharge summaries using a UMLS-grounded medical knowledge module, SBERT/FAISS template matching, a callable Text_Func, and an execution-and-repair loop (Zhang, 25 Nov 2025). EHRs Data Harmonization Platform formalizes harmonization logic through a variable sheet, a variable details sheet, and a derived-variable registry, so that mappings, missingness semantics, and custom feature derivations can be versioned and shared without exposing PHI (Aminoleslami et al., 2024).

Other syntheses extend the same label to identity, device, and temporal infrastructures. Master Patient Index work frames identity resolution as centralized cross-system indexing with merge and unmerge logic, blocking rules, and HL7/IHE interoperability (Jayatissa et al., 2018). Stanford’s death-linkage study shows that no single linkage rule provides both coverage and accuracy across EHR and LADMF, which is highly relevant for any EHRMaster-style mortality ascertainment layer (Peralta et al., 2022). Further adjacent work adds privacy-preserving learning with federated learning and differential privacy (Ganadily et al., 2024), faithful hospital-course summarization via revised references and sentence-level entity planning (Adams, 2024), and extended-reality visualization built around FHIR resources, volumetric imaging, and AI segmentation (Marteau et al., 5 Dec 2025).

6. Limitations, governance, and future directions

The primary limitation of EHRMaster in its original form is clear: Knowledge-Driven tasks remain challenging. The benchmark states that clinical inference often needs robust external medical knowledge or calibrated decision rules not captured by table-only contexts. This is why EHRMaster’s improvements on K-* tasks are competitive but mixed, even when Data-Driven performance approaches ceiling levels (Yang et al., 11 Nov 2025).

A second limitation is representational scope. The framework handles common temporal and relational patterns through pandas filters, grouping, and sorting, but extremely long trajectories or multi-table joins may require dedicated retrieval or SQL/graph layers. Table size and context limits remain an issue, and chunking strategies are not detailed. Error propagation is also possible: mistakes in planning or alignment can cascade, even if code execution catches many numeric errors.

Deployment introduces another layer of constraint. The paper does not specify sandboxing, resource limits, timeouts, or PHI-handling policies. The recommended operational pattern is therefore a locked-down container with no network, limited CPU, memory, and runtime, disallowed imports and file system access, schema validation for column references, exception handling with fallback or re-prompting, and strict output-format validators. For clinical use, the guidance is explicit that knowledge-driven outputs influencing care should remain clinician-reviewed, with traceability from plan to aligned plan to code to final answer (Yang et al., 11 Nov 2025).

The broader EHRMaster ecosystem sharpens these concerns rather than dissolving them. Privacy-aware summarization work explicitly constrains generation to retrieved evidence and treats operational monitoring as a safety requirement (Kazemzadeh et al., 4 Jan 2026). Hospital-course summarization research shows that fine-tuned LLMs are highly prone to entity hallucinations and can cover fewer salient entities unless explicit entity planning is introduced (Adams, 2024). Federated learning and differential privacy work further indicates that privacy-preserving EHR modeling requires not only local training, but clipping, noise calibration, privacy accounting, secure aggregation, and auditability (Ganadily et al., 2024).

The future directions named for the original method are twofold. First, EHRStruct should be extended toward iterative, adaptive treatment planning that reacts to real-time changes such as drug reactions and physiologic trajectories. Second, structured reasoning should be unified with external clinical knowledge, potentially via retrieval or structured knowledge integration, to strengthen K-* performance (Yang et al., 11 Nov 2025). In the broader literature, this suggests an eventual convergence of schema-grounded execution, evidence-constrained summarization, multimodal retrieval, and governance-grade privacy controls into a single EHR orchestration layer rather than a standalone benchmark solver.