Accelerating Reproducible Research in Synthetic EHR Generation

Abstract: The generation of high-fidelity synthetic Electronic Health Records (EHR) is crucial for advancing medical research while preserving patient privacy. However, head-to-head comparison of existing generative models is hindered by disjointed codebases, incompatible data loaders, conflicting library dependencies, and inconsistent evaluation protocols. To address these gaps, we introduce a lightweight, end-to-end benchmarking framework for reproducible synthetic EHR evaluation, organized as a unified pipeline spanning data ingestion, standardized model training, and architecture-agnostic evaluation. Our current implementation targets the generation of longitudinal ICD diagnosis codes -- the most commonly studied modality in this literature -- and is built on the community-maintained PyHealth library. We reimplement and unify strong baselines (MedGAN, CorGAN, PromptEHR, HALO) under full ICD-9 vocabulary granularity, and add a lightweight GPT-2 baseline from the general-purpose sequence-modeling literature. We contribute a rigorous, architecture-agnostic privacy-utility evaluation suite that applies identically to GAN- and transformer-based generators, and report bootstrapped confidence intervals across all metrics. We further analyze the poor long-tailed performance of existing models and discuss the extensibility of our framework beyond diagnosis codes. By lowering the engineering barrier to running, extending, and evaluating under a single pipeline, we introduce a starting point for community-driven reproducibility and benchmarking synthetic EHR models.

• The paper establishes a unified, architecture-agnostic benchmarking framework that standardizes the evaluation of synthetic EHR generative models.
• It compares flat and sequential models over the full ICD-9 code space, highlighting critical trade-offs in statistical fidelity and privacy risk.
• Empirical results show that transformer-based models like GPT-2 can match specialized architectures while better handling rare, long-tailed diagnosis codes.

A Unified Benchmarking Framework for Synthetic EHR Generation

Motivation and Problem Setting

Generation of synthetic Electronic Health Records (EHRs) is an active research area, driven by privacy regulations that restrict data sharing and a need for reproducible, high-fidelity data to fuel medical AI development. While numerous generative models—ranging from GAN-based structures like MedGAN and CorGAN to autoregressive transformers such as PromptEHR and HALO—have been proposed, meaningful benchmarking and reproducibility remain unsolved challenges. Disjointed codebases, inconsistent formats, divergent evaluation protocols, and truncated clinical vocabularies hinder apples-to-apples comparison, masking failure modes and limiting progress.

The problem is exacerbated by common practices such as dimensionality reduction (e.g., truncating ICD-9 codes to 3 digits) to manage high-dimensional sparsity, which destroys the rare, clinically informative patterns needed for precision medicine. Existing evaluations further complicate comparisons by employing non-standardized, architecture-dependent metrics and code subsets, preventing the field from converging on best practices for both utility and privacy.

Benchmarking Framework and Methodological Advances

To address these systemic limitations, the framework introduced in "Accelerating Reproducible Research in Synthetic EHR Generation" (2606.06990) provides an end-to-end, architecture-agnostic evaluation and benchmarking pipeline. Built atop the community-driven PyHealth library, the pipeline ensures all generative models can be trained, evaluated, and compared through a unified API, making reproducible research practical.

Key characteristics include:

• Full Granularity ICD-9 Coverage: All baselines are restored and extended to operate over the entire 6,955-code ICD-9 vocabulary, rather than the truncated 3-digit standard.
• Unified Data Schema: Regardless of the generative approach—flattened (vectorized) or sequential (visit-level)—all outputs are converted into a standardized long-format (subject_id, visit_id, code), enabling direct comparison of distributional fidelity, structural correctness, and privacy metrics.

  Figure 1: Illustration of data transformation pipelines—flattened and sequential approaches—and the unifying long-format schema applied for evaluation.

Explicit support for both static matrix models (MedGAN, CorGAN) and sequence models (PromptEHR, HALO, GPT-2) enables rigorous study of the trade-offs between models that preserve and discard temporal structure. The framework also enables rapid integration of new models and modalities beyond diagnosis codes, extending to future ICD standards and richer EHR representations.

Model Architectures and Augmentations

Five principal generative models are reimplemented and benchmarked:

• MedGAN: Restoration of count-matrix mode (omitted in recent ports), enabling integer-valued output across the full ICD-9 space. The architecture is reconfigured for the expanded vocabulary and supports both binary and count data for robust modeling of healthcare utilization.
• CorGAN: Expansion to an 8-layer convolutional architecture to accommodate the high-dimensional code space. Encoder-decoder structures with binary cross-entropy ensure joint modeling of code correlations, but experimental results show this approach struggles with long-tail code sparsity.
• PromptEHR: BART-based conditional generation, leveraging demographic context via prompt-based conditioning and introducing a frequency-guided decoding mechanism that adaptively blends model probabilities with empirical code frequencies.
• HALO: A production-capable hierarchical transformer model that factorizes EHR generation across visit and intra-visit code levels, capturing both the sequential nature of patient history and the set-structure of events within visits.
• GPT-2 Baseline: A lightweight, domain-agnostic autoregressive transformer, adapted for EHR sequence modeling, providing an essential baseline that challenges the necessity of domain-specific architectural complexity.

A core innovation is the extensive effort to ensure all models are compatible with the full clinical vocabulary and are benchmarked identically—both in terms of pre/post-processing and evaluation criteria.

Evaluation Protocol: Metrics and Methodology

The evaluation pipeline encompasses both fidelity (utility) and privacy metrics that are strictly architecture-independent. For statistical utility, the principal components are:

• Marginal Code Prevalence ($R^2$, Pearson’s $\rho$, RMSE): Quantifies alignment between real and synthetic distributions for every medical code.
• Discriminator-based Fidelity: A binary classifier (LSTM for sequential outputs, Random Forest for flat models) differentiates between real and synthetic samples, with scores near 0.5 indicating indistinguishability and thus high-fidelity synthesis.

Privacy assessment is executed via:

• Nearest Neighbor Adversarial Accuracy Risk (NNAAR): Detects overfitting by measuring whether synthetic data points are closer to the training set than the general population, controlling for memorization.
• Membership Inference Attack (MIA): An attacker model attempts to discern if a particular patient was included in the real training data, with accuracy near random chance as the ideal.

  Figure 2: ICD-9 code distribution in MIMIC-III, visualizing the heavy-tailed nature and the dominance of a minority of common codes compared to the large universe of rare diagnoses.

This comprehensive, bootstrapped evaluation is critical for surfacing trade-offs and failure modes obfuscated by insufficient or metric-specific analysis.

Empirical Results and Analysis

The most salient empirical findings are as follows:

• Sequential vs. Flat Models: While flat models (MedGAN, CorGAN) excel at reproducing marginal code prevalence (e.g., MedGAN $R^2$ ≈ 0.98), they exhibit inferior joint fidelity (discriminative scores far from 0.5) and structural inconsistencies (unable to model visits). In contrast, sequential models (HALO, GPT-2, PromptEHR) better capture both temporal dependencies and joint distributions, yielding superior discriminative scores (HALO: 0.327, GPT-2: 0.355).
• Long-Tailed Code Failure Modes: All models deteriorate in their handling of rare diagnosis codes (lower prevalence percentiles). Only HALO and PromptEHR maintain moderate $R^2$ up to the 99th percentile of the ICD-9 distribution, though no approach achieves robust tail coverage.

  Figure 3: Impact of long-tailed code distributions on generative utility and privacy. Only HALO and PromptEHR retain reasonable prevalence $R^2$ as rare codes are increasingly included in the evaluation.

• GPT-2 Baseline Competitiveness: The non-domain-specific GPT-2 baseline matches or surpasses specialized models in fidelity, challenging assumptions about the necessity for bespoke architectures in synthetic EHR modeling. Despite slightly lower vocabulary coverage, GPT-2 achieves indistinguishability scores superior to PromptEHR and HALO, while maintaining competitive privacy risk and marginal utility.

  Figure 4: Rank-frequency (Zipf) plot of patient-level ICD-9 code prevalence, highlighting that only HALO adequately tracks the long tail of rare codes, whereas CorGAN and PromptEHR truncate prematurely.

• Privacy Preservation: Across all architectures, privacy risk via NNAAR and MIA remains low, demonstrating negligible overfitting or data leakage.

Structural and Distributional Fidelity

Supplementary structural ablations reinforce the separation between model classes:

• Sequential models produce realistic distributions of visits per patient and codes per visit but under-represent longer visits and patient histories due to sequence-length contraction under MLE training.
• Flat models trivially match codes-per-patient statistics by construction but collapse all visits into a pseudo-visit, losing temporal resolution.

Co-occurrence analysis further confirms that only HALO and MedGAN closely approximate real-world pairwise associations among prevalent codes (cosine similarity >0.97), whereas CorGAN and PromptEHR exhibit degraded joint structure.

Implications and Future Research Directions

Practical Implications: The adoption of this framework significantly lowers the engineering barrier for reproducing, extending, and comparing diverse generative models. By standardizing both the pipeline and evaluation, it paves the way for systematic advances in synthetic healthcare data generation and improves scientific rigor in benchmarking.

Theoretical Implications: The empirical trade-off between marginal alignment and joint distribution fidelity highlights the inherent limitations of flattened models for high-dimensional, sequential medical data, even when marginal utility metrics are near-perfect. Future work must address the modeling of the inherently long-tailed, sparse structure of real-world EHRs.

Future Directions:

• Architectural Innovations: The persistent failure to model the ICD-9 tail (rare codes) signals a need for architectures—potentially diffusion-based or with explicit tail modeling—that can robustly synthesize rare clinical patterns without overfitting.
• Beyond ICD-9: Migration toward richer modalities (ICD-10, labs, medications, SDOH) and new generative paradigms (e.g., diffusion transformers) will better match real-life clinical analytics scenarios.
• Metric and Evaluation Expansion: Scalability and theoretical rigor of high-dimensional fidelity and privacy metrics remain open problems; research into approximation-based and sample-level metrics is active [alaa2022faithful, shi2024tabdiff].
• Downstream Task Evaluation: Assessing the clinical utility of synthetic data for downstream tasks such as predicting risk or treatment response with full code granularity is essential for validating utility beyond statistical reproduction.

Conclusion

This work establishes a robust, extensible framework for benchmarking synthetic EHR generators, restoring and extending seminal models for full clinical vocabulary, and introducing architecture-agnostic, reproducible evaluation. Sequential models, particularly those based on transformers, achieve more faithful joint and structural fidelity than classical GAN-based approaches, though even state-of-the-art methods underperform on rare code synthesis. The strong performance of the GPT-2 baseline underscores the potential of generic sequence modeling for healthcare synthesis tasks and challenges the field to justify additional complexity. The open framework is positioned as a "living substrate" for the research community, providing a rigorous foundation for innovation in the generation and evaluation of synthetic clinical data.