CEHR-GPT: Decoder-Based EHR Modeling
- CEHR-GPT is a family of decoder-oriented transformer models that linearize and tokenizes longitudinal patient records for clinical modeling.
- The architecture leverages time-token encoding and autoregressive generation to support tasks such as synthetic data creation, risk prediction, and phenotyping.
- CEHR-GPT unifies feature representation and zero-shot forecasting while enabling integration of structured codes and unstructured clinical notes.
CEHR-GPT denotes a line of decoder-oriented transformer methods for electronic health records that linearize longitudinal patient histories into token sequences and then use autoregressive generation, discriminative heads, or both for clinical modeling. In the literature, the name refers both to a specific synthetic-EHR generator with chronological patient timelines and OMOP conversion (Pang et al., 2024) and to a later scalable multi-task foundation model that unifies feature representation, zero-shot prediction, and synthetic data generation within a single architecture (Pang et al., 3 Sep 2025). Closely related work also uses CEHR-GPT as a design pattern for adapting descriptor-based phenotyping, clinical-note classification, and multimodal EHR modeling to GPT-style decoders (Munoz-Farre et al., 2022).
1. Terminological scope and historical lineage
CEHR-GPT emerged against a background in which EHR modeling had already bifurcated into structured-code sequence models and free-text clinical LLMs. An early causal-transformer precursor, MedGPT, used MedCAT to extract SNOMED-CT disorder concepts from clinical narratives, built patient-wise timelines, prepended age tokens, truncated or padded sequences to length , and achieved precision of $0.344$, $0.552$, and $0.640$ for top-$1$, top-$3$, and top-$5$ future-disorder prediction on King’s College Hospital data, outperforming an LSTM baseline (Kraljevic et al., 2021). A different precursor, sEHR-CE, recast structured EHR codes as natural-language descriptors and fine-tuned PubMedBERT for cohort expansion without relying on curated terminology mappings (Munoz-Farre et al., 2022).
Within this lineage, the 2024 CEHR-GPT paper defined the model as a chronology-preserving generator for “fully timestamped, multi-domain patient histories” that can be converted to the OMOP Common Data Model (Pang et al., 2024). The 2025 CEHR-GPT foundation-model paper generalized the scope further, presenting a “general-purpose foundation model for EHR data” with three capabilities—feature representation, zero-shot prediction, and synthetic data generation—organized around time-token-based learning (Pang et al., 3 Sep 2025). This suggests that CEHR-GPT is best understood not as a single frozen architecture but as a family of GPT-style EHR systems centered on longitudinal tokenization, temporal encoding, and decoder-based modeling.
| System | Primary modality | Stated capability |
|---|---|---|
| CEHR-GPT | Structured OMOP timelines | Synthetic EHR generation |
| CEHR-GPT | Structured EHR timelines | Representation, zero-shot prediction, synthesis |
| sEHR-CE-derived “CEHR-GPT” recipe | Structured codes rendered as text | Cohort expansion and phenotyping |
2. Patient-sequence representations and temporal encoding
The defining CEHR-GPT representation in the synthetic-generation work begins with a four-token demographic prompt: start-year, age at first visit, gender, and race. Each clinic encounter is then bracketed by visit-start and visit-end tokens, annotated with a visit-type token, and separated by an artificial time token encoding the number of days between visits; intervals longer than days collapse to a “Long-Term” token. Within inpatient stays, same-day groups are separated by an IATT token carrying the in-hospital day offset, and inpatient blocks end with a discharge-facility code such as DC_HOME (Pang et al., 2024). The encoder-decoder conversion is designed so that all event dates satisfy
where $0.344$0 is a constant shift, so relative timing is preserved (Pang et al., 2024).
The later CEHR-GPT foundation model retains the same basic logic but scales it to a $0.344$1-token context window and removes positional embeddings entirely. Each patient record is flattened into four demographic tokens—[start_year], [start_age], [gender], [race]—followed by visits bracketed by [VS] and [VE], each beginning with a visit-type token [VT] and ending with an ATT token such as D10 or [LT] for gaps $0.344$2 days (Pang et al., 3 Sep 2025). A central claim of this formulation is that time should be encoded as a token rather than as an additive embedding: because each ATT is a full token, the self-attention mechanism can learn arbitrary interactions among time and event embeddings (Pang et al., 3 Sep 2025).
A separate serialization strategy appears in the sEHR-CE recipe for a GPT-style adaptation. There, a patient’s longitudinal history is a time-ordered sequence of codes $0.344$3, each replaced by an English descriptor $0.344$4, and concatenated into a single document,
$0.344$5
with standard BERT-style token, positional, and segment embeddings (Munoz-Farre et al., 2022). In that construction the model never sees raw Read or ICD codes—only text descriptors—so terminology alignment is delegated to the transformer rather than to manual code maps (Munoz-Farre et al., 2022).
3. Decoder architectures and learning objectives
The original CEHR-GPT generator uses a decoder-only transformer with $0.344$6 layers, $0.344$7 attention heads, embedding dimension $0.344$8, feed-forward hidden dimension $0.344$9, learned positional embeddings of size $0.552$0, and dropout $0.552$1 on attention and FFN sublayers (Pang et al., 2024). Training minimizes the standard next-token cross-entropy loss
$0.552$2
so the model learns both event continuation and time progression because explicit time markers are part of the token stream (Pang et al., 2024).
The 2025 CEHR-GPT foundation model adopts a larger GPT-2–style decoder stack with $0.552$3 layers, $0.552$4 attention heads, $0.552$5-dimensional embeddings, and no positional embeddings (Pang et al., 3 Sep 2025). Its distinctive addition is a pair of time-aware auxiliary heads applied only to ATT tokens. The Time Decomposition head splits each ATT embedding into year, month, and day components and predicts the true decomposed gap, while the Time-to-Event head parameterizes a Gamma distribution over the actual day gap $0.552$6 (Pang et al., 3 Sep 2025). The overall pretraining objective combines next-token prediction with these ATT-regularization terms:
$0.552$7
This explicitly couples chronological language modeling with temporal calibration of the time tokens (Pang et al., 3 Sep 2025).
Related decoder-based classifiers show how CEHR-GPT ideas extend beyond pure generation. A GPT-2 small clinical text classifier froze the embedding layers and transformer blocks $0.552$8, kept only the final transformer block, final LayerNorm, and a lightweight classification head trainable, and reduced the trainable parameter count to approximately $0.552$9 million, or about $0.640$0 of the model (Irany, 29 Jan 2026). By contrast, the multimodal GDP model combines a CNN-plus-transformer encoder for structured time series, a BioClinicalBERT note encoder, cross-modal attention, and a $0.640$1-layer, $0.640$2B-parameter LLaMA-based decoder trained with generative narrative loss, masked feature prediction, and next time-step prediction (Sivarajkumar et al., 22 Aug 2025). A plausible implication is that CEHR-GPT has broadened from a chronology-preserving generator into a more general decoder-centric design space spanning structured, textual, and multimodal EHR inputs.
4. Phenotyping, risk prediction, and note-based clinical inference
In descriptor-based cohort expansion, the sEHR-CE framework first performed MLM fine-tuning on $0.640$3 K UK Biobank patients using AdamW with learning rate $0.640$4, batch size $0.640$5, and $0.640$6 epochs, then carried out multi-label cohort-expansion training with a $0.640$7-fold patient-level stratified split, PubMedBERT’s $0.640$8-token limit, AdamW at learning rate $0.640$9, warmup of $1$0 of total steps, and $1$1 epochs (Munoz-Farre et al., 2022). To address comorbidity and label leakage, concept-description pairs lexically linked to a positive label were masked $1$2 of the time, kept $1$3, or replaced $1$4, and records with multiple positive phenotypes were replicated with phenotype-specific masking and loss masking (Munoz-Farre et al., 2022). On the four held-out phenotypes—type 2 diabetes mellitus, heart failure, breast cancer, and prostate cancer—the text model reported average Recall@0.5 of $1$5 and AUPRC of $1$6, compared with $1$7 and $1$8 for BEHRT and $1$9 and $3$0 for sEHR-CE-codes (Munoz-Farre et al., 2022). In the type 2 diabetes use case, unlabeled controls with predicted probability at or above the $3$1th percentile of the control distribution, approximately $3$2, yielded about $3$3 putative missed cases out of about $3$4 controls, and these individuals had elevated HbA$3$5c, higher polygenic risk scores, higher visit rates, and lower survival (Munoz-Farre et al., 2022).
Clinical-note CEHR-GPT patterns are also used as feature extractors. In mortality prediction from discharge summaries, GPT-4o-mini was prompted with three questions—risk of death, risk of readmission, and overall health—each scored from $3$6 to $3$7, and the three parsed values were used as logistic-regression features alongside structured EMR features for $3$8 first-time CCU or CVICU admissions (Anderson et al., 14 Apr 2025). On the test set, the combined EMR+GPT model reached AUCs of $3$9 for $5$0-year mortality, $5$1 for $5$2-day mortality, and $5$3 for $5$4-day readmission, while the top-decile PPV for $5$5-year mortality rose from $5$6 for EMR only to $5$7 for EMR+GPT (Anderson et al., 14 Apr 2025).
A more explicitly decoder-based clinical text classifier adapted GPT-2 small to MIMIC-IV-Note radiology reports with sequence length $5$8, batch size $5$9, 0 epochs, learning rate 1, weight decay 2, and a WeightedRandomSampler for imbalance mitigation (Irany, 29 Jan 2026). On the 3 k-report any-disease POS+UNC aggregate task, head-only training achieved test AUROC 4, F1 5, and test accuracy about 6, selective fine-tuning achieved AUROC 7, F1 8, and test accuracy 9, and full fine-tuning achieved AUROC 0, F1 1, and test accuracy 2 (Irany, 29 Jan 2026). In cognitive-impairment staging from unstructured notes, zero-shot GPT-4o reached quadratic weighted Cohen’s kappa of 3 for five-level global CDR scoring in memory-clinic notes and 4 for three-level NC/MCI/Dementia classification over a three-year outpatient window, rising to 5 on high-confidence physician cases (Leng et al., 13 Feb 2025). These note-centric systems are not identical to the timeline-based CEHR-GPT foundation models, but they occupy the same methodological neighborhood: decoder-based inference over longitudinal clinical text for phenotyping and prediction.
5. Synthetic EHR generation, utility, and privacy
Synthetic data generation is the most literal instantiation of CEHR-GPT. In the 2024 chronology-based model, OMOP tables are converted into token sequences, synthetic sequences are generated by autoregressive sampling with top-6 or top-7 filtering, and the output is decoded back into OMOP through a date-cursor engine that advances on ATT or IATT tokens, opens and closes visits, and inserts domain-table records with the current cursor date (Pang et al., 2024). Any sequence violating the grammar is discarded; in practice, 8 of samples passed validation (Pang et al., 2024). Training used 9 million CUIMC-NYP patients, a 0-token context window, batch size 1, learning rate 2, and two epochs on two GPUs (Pang et al., 2024).
Evaluation in that work was organized into marginal distributions, pairwise co-occurrence, predictive utility, and privacy risk. Top-3 4 sampling and top-5 were closest to the real-data lower bound for pairwise co-occurrence, while vanilla chronological GPT without ATT/IATT performed worst (Pang et al., 2024). In downstream predictive tasks such as heart-failure readmission, hospitalization risk, COPD readmission, AFib-to-stroke, and CAD-to-CABG, the weighted distance between synthetic-trained and real-trained models was approximately 6–7, with zero as the ideal (Pang et al., 2024). Privacy metrics included membership inference F1 at most 8, attribute inference F1 around 9, meaningful identity disclosure risk approximately $0.344$00, and nearest-neighbor adversary attack risk around $0.344$01, all below the nominal $0.344$02 threshold (Pang et al., 2024).
The later CEHR-GPT foundation model reported that synthetic-only logistic regression on heart-failure readmission, hospitalization, AFib stroke, and CABG was within $0.344$03–$0.344$04 of real-data models in AUROC, and that augmenting real data with synthetic data yielded modest but consistent gains (Pang et al., 3 Sep 2025). Treatment-pathway replication for hypertension, diabetes, and depression cohorts produced synthetic prevalences within $0.344$05 absolute of real, and all four privacy attack scores remained well below the $0.344$06 thresholds (Pang et al., 3 Sep 2025).
Related synthetic-EHR decoder work broadens the comparison set. SynEHRgy tokenized covariates, ICD codes, and irregular time series into a single sequence, trained a small GPT-2–style model with $0.344$07 layers, hidden dimension $0.344$08, $0.344$09 attention heads, vocabulary size $0.344$10, and causal loss, and reported strong fidelity and privacy metrics on MIMIC-III (Karami et al., 2024). In its CEHR-GPT benchmark, bigram, trigram, and sequential-bigram Pearson correlations over the top-$0.344$11 ICD n-grams were $0.344$12, $0.344$13, and $0.344$14; time-series fidelity reached Precision $0.344$15, Recall $0.344$16, Coverage $0.344$17, and $0.344$18; and privacy metrics reached Wasserstein distance $0.344$19, JSD $0.344$20, and AUROC $0.344$21 (Karami et al., 2024). These results reinforce a recurring theme in CEHR-GPT research: the synthetic-data problem is evaluated not only by token realism but by downstream utility and attack-based privacy audits.
6. Generalization, multimodality, and open technical questions
The CEHR-GPT foundation model explicitly addressed external transfer by extending its tokenizer with more than $0.344$22 new OMOP concept IDs from the Stanford ehrshot dataset so that no events were dropped (Pang et al., 3 Sep 2025). Across $0.344$23 binary and multiclass tasks, fine-tuned CEHR-GPT-T outperformed all baselines on “New Diagnosis” tasks by an average AUROC gain of $0.344$24 over the best baseline and matched or exceeded the best “Patient Outcomes” models, while linear-probing CEHR-GPT-L remained competitive, especially on rare-event tasks (Pang et al., 3 Sep 2025). This is notable because the same pretrained model family was used for feature extraction, zero-shot forecasting via generated continuations, and unconditional synthetic-data generation (Pang et al., 3 Sep 2025).
Multimodal expansion appears in GDP, which natively encodes structured EHR time series with a CNN-transformer encoder, fuses them with unstructured clinical notes through cross-modal attention, and uses a LLaMA-based decoder for both prediction and generation (Sivarajkumar et al., 22 Aug 2025). On MIMIC-IV, GDP reported heart-failure AUROC $0.344$25, type 2 diabetes AUROC $0.344$26, and $0.344$27-day readmission AUROC $0.344$28; for narrative generation, GDP-Instruct reached ROUGE-L $0.344$29 and BERTScore-F1 $0.344$30, and in blinded human evaluation it scored highest on faithfulness, fluency, and overall clinical utility (Sivarajkumar et al., 22 Aug 2025). A plausible implication is that future CEHR-GPT systems will increasingly be multimodal rather than purely timeline-based.
Another extension direction is semantic and ontological enrichment. HiSGT constructs a hierarchy graph over clinical codes, derives hierarchy-aware embeddings with a GNN, extracts semantic embeddings from code descriptions using a pre-trained clinical LLM, and fuses these with transformer token embeddings (Zhou et al., 28 Feb 2025). On MIMIC-IV, HiSGT reported unigram $0.344$31 versus $0.344$32 for GPT, bigram $0.344$33 versus $0.344$34, classification accuracy $0.344$35 versus $0.344$36, and attribute-inference F1 around $0.344$37 for both HiSGT and GPT (Zhou et al., 28 Feb 2025). This suggests that one limitation of flat code-token generation is the absence of explicit hierarchy and semantics in the token space.
Several limitations recur across the CEHR-GPT literature. Sequence-length ceilings remain consequential: the 2024 synthetic CEHR-GPT excluded patients whose sequences exceed $0.344$38 tokens, the radiology classifier truncated to $0.344$39 tokens, and the foundation model truncated at $0.344$40 tokens (Pang et al., 2024). Time granularity is often day-level rather than hour- or minute-level, and some variants exclude laboratory measurements or free text from the core input representation (Pang et al., 3 Sep 2025). Zero-shot forecasting through autoregressive sampling is computationally expensive, and privacy evaluations are usually empirical rather than based on formal differential privacy guarantees (Pang et al., 3 Sep 2025). The net result is a field that has established a coherent decoder-based paradigm for longitudinal EHR modeling, but not yet a single settled architecture.