---
title: 'CEHR-GPT: Decoder-Based EHR Modeling'
url: https://www.emergentmind.com/topics/cehr-gpt
type: topic
---

# CEHR-GPT: Decoder-Based EHR Modeling

CEHR-GPT denotes a line of decoder-oriented transformer methods for electronic health records that linearize longitudinal patient histories into token sequences and then use autoregressive generation, discriminative heads, or both for clinical modeling. In the literature, the name refers both to a specific synthetic-EHR generator with chronological patient timelines and OMOP conversion [2402.04400] and to a later scalable multi-task foundation model that unifies feature representation, zero-shot prediction, and synthetic data generation within a single architecture [2509.03643]. Closely related work also uses CEHR-GPT as a design pattern for adapting descriptor-based phenotyping, clinical-note classification, and multimodal EHR modeling to GPT-style decoders [2211.17121].

## 1. Terminological scope and historical lineage

CEHR-GPT emerged against a background in which EHR modeling had already bifurcated into structured-code sequence models and free-text clinical language models. An early causal-transformer precursor, MedGPT, used MedCAT to extract SNOMED-CT disorder concepts from clinical narratives, built patient-wise timelines, prepended age tokens, truncated or padded sequences to length $T=50$, and achieved precision of $0.344$, $0.552$, and $0.640$ for top-$1$, top-$3$, and top-$5$ future-disorder prediction on King’s College Hospital data, outperforming an LSTM baseline [2107.03134]. A different precursor, sEHR-CE, recast structured EHR codes as natural-language descriptors and fine-tuned PubMedBERT for cohort expansion without relying on curated terminology mappings [2211.17121].

Within this lineage, the 2024 CEHR-GPT paper defined the model as a chronology-preserving generator for “fully timestamped, multi-domain patient histories” that can be converted to the OMOP Common Data Model [2402.04400]. The 2025 CEHR-GPT foundation-model paper generalized the scope further, presenting a “general-purpose foundation model for EHR data” with three capabilities—feature representation, zero-shot prediction, and synthetic data generation—organized around time-token-based learning [2509.03643]. This suggests that CEHR-GPT is best understood not as a single frozen architecture but as a family of GPT-style EHR systems centered on longitudinal tokenization, temporal encoding, and decoder-based modeling.

| System | Primary modality | Stated capability |
|---|---|---|
| CEHR-GPT | Structured OMOP timelines | Synthetic EHR generation |
| CEHR-GPT | Structured EHR timelines | Representation, zero-shot prediction, synthesis |
| sEHR-CE-derived “CEHR-GPT” recipe | Structured codes rendered as text | Cohort expansion and phenotyping |

## 2. Patient-sequence representations and temporal encoding

The defining CEHR-GPT representation in the synthetic-generation work begins with a four-token demographic prompt: start-year, age at first visit, gender, and race. Each clinic encounter is then bracketed by visit-start and visit-end tokens, annotated with a visit-type token, and separated by an artificial time token $D_t$ encoding the number of days between visits; intervals longer than $1{,}080$ days collapse to a “Long-Term” token. Within inpatient stays, same-day groups are separated by an IATT token carrying the in-hospital day offset, and inpatient blocks end with a discharge-facility code such as `DC_HOME` [2402.04400]. The encoder-decoder conversion is designed so that all event dates satisfy
$$
T(D_i)=T(D_i')+C_i,
$$
where $C_i$ is a constant shift, so relative timing is preserved [2402.04400].

The later CEHR-GPT foundation model retains the same basic logic but scales it to a $4{,}096$-token context window and removes positional embeddings entirely. Each patient record is flattened into four demographic tokens—`[start_year]`, `[start_age]`, `[gender]`, `[race]`—followed by visits bracketed by `[VS]` and `[VE]`, each beginning with a visit-type token `[VT]` and ending with an ATT token such as `D10` or `[LT]` for gaps $\ge 1080$ days [2509.03643]. A central claim of this formulation is that time should be encoded as a token rather than as an additive embedding: because each ATT is a full token, the self-attention mechanism can learn arbitrary interactions among time and event embeddings [2509.03643].

A separate serialization strategy appears in the sEHR-CE recipe for a GPT-style adaptation. There, a patient’s longitudinal history is a time-ordered sequence of codes $(\theta_1,\dots,\theta_t)$, each replaced by an English descriptor $\xi_{\theta_i}$, and concatenated into a single document,
$$
s=[CLS]\,W(\xi_{\theta_1})\,W(\xi_{\theta_2})\,\dots\,W(\xi_{\theta_t})\,[SEP],
$$
with standard BERT-style token, positional, and segment embeddings [2211.17121]. In that construction the model never sees raw Read or ICD codes—only text descriptors—so terminology alignment is delegated to the transformer rather than to manual code maps [2211.17121].

## 3. Decoder architectures and learning objectives

The original CEHR-GPT generator uses a decoder-only transformer with $16$ layers, $8$ attention heads, embedding dimension $d_{\text{model}}=128$, feed-forward hidden dimension $128$, learned positional embeddings of size $128$, and dropout $0.1$ on attention and FFN sublayers [2402.04400]. Training minimizes the standard next-token cross-entropy loss
$$
\mathcal{L}=-\sum_{t=1}^{T}\log P(x_t\mid x_{1:t-1}),
$$
so the model learns both event continuation and time progression because explicit time markers are part of the token stream [2402.04400].

The 2025 CEHR-GPT foundation model adopts a larger GPT-2–style decoder stack with $16$ layers, $12$ attention heads, $768$-dimensional embeddings, and no positional embeddings [2509.03643]. Its distinctive addition is a pair of time-aware auxiliary heads applied only to ATT tokens. The Time Decomposition head splits each ATT embedding into year, month, and day components and predicts the true decomposed gap, while the Time-to-Event head parameterizes a Gamma distribution over the actual day gap $\Delta T_{ij}$ [2509.03643]. The overall pretraining objective combines next-token prediction with these ATT-regularization terms:
$$
L=\sum_{i,j}\Bigl(L^{\mathrm{ntp}}_{ij}+\mathbbm{1}[\text{token}_{ij}\in\mathrm{ATT}]\cdot(TTE_{ij}+TD_{ij})\Bigr).
$$
This explicitly couples chronological language modeling with temporal calibration of the time tokens [2509.03643].

Related decoder-based classifiers show how CEHR-GPT ideas extend beyond pure generation. A GPT-2 small clinical text classifier froze the embedding layers and transformer blocks $1\dots 11$, kept only the final transformer block, final LayerNorm, and a lightweight classification head trainable, and reduced the trainable parameter count to approximately $7.08$ million, or about $6\%$ of the model [2601.21955]. By contrast, the multimodal GDP model combines a CNN-plus-transformer encoder for structured time series, a BioClinicalBERT note encoder, cross-modal attention, and a $24$-layer, $3.2$B-parameter LLaMA-based decoder trained with generative narrative loss, masked feature prediction, and next time-step prediction [2508.16054]. A plausible implication is that CEHR-GPT has broadened from a chronology-preserving generator into a more general decoder-centric design space spanning structured, textual, and multimodal EHR inputs.

## 4. Phenotyping, risk prediction, and note-based clinical inference

In descriptor-based cohort expansion, the sEHR-CE framework first performed MLM fine-tuning on $138$ K UK Biobank patients using AdamW with learning rate $4\times10^{-5}$, batch size $32$, and $5$ epochs, then carried out multi-label cohort-expansion training with a $5$-fold patient-level stratified split, PubMedBERT’s $512$-token limit, AdamW at learning rate $1\times10^{-5}$, warmup of $25\%$ of total steps, and $3$ epochs [2211.17121]. To address comorbidity and label leakage, concept-description pairs lexically linked to a positive label were masked $80\%$ of the time, kept $10\%$, or replaced $10\%$, and records with multiple positive phenotypes were replicated with phenotype-specific masking and loss masking [2211.17121]. On the four held-out phenotypes—type 2 diabetes mellitus, heart failure, breast cancer, and prostate cancer—the text model reported average Recall@0.5 of $0.68\pm0.14$ and AUPRC of $0.61$, compared with $0.58\pm0.17$ and $0.53$ for BEHRT and $0.55\pm0.19$ and $0.50$ for sEHR-CE-codes [2211.17121]. In the type 2 diabetes use case, unlabeled controls with predicted probability at or above the $98$th percentile of the control distribution, approximately $0.85$, yielded about $2{,}020$ putative missed cases out of about $113{,}500$ controls, and these individuals had elevated HbA$_1$c, higher polygenic risk scores, higher visit rates, and lower survival [2211.17121].

Clinical-note CEHR-GPT patterns are also used as feature extractors. In mortality prediction from discharge summaries, GPT-4o-mini was prompted with three questions—risk of death, risk of readmission, and overall health—each scored from $1$ to $100$, and the three parsed values were used as logistic-regression features alongside structured EMR features for $14{,}011$ first-time CCU or CVICU admissions [2504.12338]. On the test set, the combined EMR+GPT model reached AUCs of $0.886$ for $1$-year mortality, $0.884$ for $90$-day mortality, and $0.720$ for $90$-day readmission, while the top-decile PPV for $1$-year mortality rose from $43\%$ for EMR only to $55\%$ for EMR+GPT [2504.12338].

A more explicitly decoder-based clinical text classifier adapted GPT-2 small to MIMIC-IV-Note radiology reports with sequence length $T=1024$, batch size $8$, $10$ epochs, learning rate $5\times10^{-5}$, weight decay $0.1$, and a WeightedRandomSampler for imbalance mitigation [2601.21955]. On the $50$ k-report any-disease POS+UNC aggregate task, head-only training achieved test AUROC $0.62$, F1 $0.61$, and test accuracy about $67.5\%$, selective fine-tuning achieved AUROC $0.96$, F1 $0.85$, and test accuracy $91.0\%$, and full fine-tuning achieved AUROC $0.99$, F1 $0.99$, and test accuracy $96.2\%$ [2601.21955]. In cognitive-impairment staging from unstructured notes, zero-shot GPT-4o reached quadratic weighted Cohen’s kappa of $0.83$ for five-level global CDR scoring in memory-clinic notes and $0.91$ for three-level NC/MCI/Dementia classification over a three-year outpatient window, rising to $0.96$ on high-confidence physician cases [2502.09715]. These note-centric systems are not identical to the timeline-based CEHR-GPT foundation models, but they occupy the same methodological neighborhood: decoder-based inference over longitudinal clinical text for phenotyping and prediction.

## 5. Synthetic EHR generation, utility, and privacy

Synthetic data generation is the most literal instantiation of CEHR-GPT. In the 2024 chronology-based model, OMOP tables are converted into token sequences, synthetic sequences are generated by autoregressive sampling with top-$k$ or top-$p$ filtering, and the output is decoded back into OMOP through a date-cursor engine that advances on ATT or IATT tokens, opens and closes visits, and inserts domain-table records with the current cursor date [2402.04400]. Any sequence violating the grammar is discarded; in practice, $98\%$ of samples passed validation [2402.04400]. Training used $2.3$ million CUIMC-NYP patients, a $512$-token context window, batch size $32$, learning rate $2\times10^{-4}$, and two epochs on two GPUs [2402.04400].

Evaluation in that work was organized into marginal distributions, pairwise co-occurrence, predictive utility, and privacy risk. Top-$p$ $95\%$ sampling and top-$k=300$ were closest to the real-data lower bound for pairwise co-occurrence, while vanilla chronological GPT without ATT/IATT performed worst [2402.04400]. In downstream predictive tasks such as heart-failure readmission, hospitalization risk, COPD readmission, AFib-to-stroke, and CAD-to-CABG, the weighted distance between synthetic-trained and real-trained models was approximately $0.05$–$0.10$, with zero as the ideal [2402.04400]. Privacy metrics included membership inference F1 at most $0.13$, attribute inference F1 around $0.03$, meaningful identity disclosure risk approximately $0.002$, and nearest-neighbor adversary attack risk around $-0.001$, all below the nominal $0.333$ threshold [2402.04400].

The later CEHR-GPT foundation model reported that synthetic-only logistic regression on heart-failure readmission, hospitalization, AFib stroke, and CABG was within $1$–$3\%$ of real-data models in AUROC, and that augmenting real data with synthetic data yielded modest but consistent gains [2509.03643]. Treatment-pathway replication for hypertension, diabetes, and depression cohorts produced synthetic prevalences within $0.03$ absolute of real, and all four privacy attack scores remained well below the $0.333$ thresholds [2509.03643].

Related synthetic-EHR decoder work broadens the comparison set. SynEHRgy tokenized covariates, ICD codes, and irregular time series into a single sequence, trained a small GPT-2–style model with $N=4$ layers, hidden dimension $384$, $4$ attention heads, vocabulary size $5{,}127$, and causal loss, and reported strong fidelity and privacy metrics on MIMIC-III [2411.13428]. In its CEHR-GPT benchmark, bigram, trigram, and sequential-bigram Pearson correlations over the top-$1{,}000$ ICD n-grams were $0.81$, $0.62$, and $0.68$; time-series fidelity reached Precision $0.64$, Recall $0.51$, Coverage $0.58$, and $\mathrm{MSE}_{\mathrm{corr}}=0.018$; and privacy metrics reached Wasserstein distance $0.11$, JSD $0.05$, and AUROC $0.50$ [2411.13428]. These results reinforce a recurring theme in CEHR-GPT research: the synthetic-data problem is evaluated not only by token realism but by downstream utility and attack-based privacy audits.

## 6. Generalization, multimodality, and open technical questions

The CEHR-GPT foundation model explicitly addressed external transfer by extending its tokenizer with more than $5{,}000$ new OMOP concept IDs from the Stanford ehrshot dataset so that no events were dropped [2509.03643]. Across $15$ binary and multiclass tasks, fine-tuned CEHR-GPT-T outperformed all baselines on “New Diagnosis” tasks by an average AUROC gain of $1.0\%$ over the best baseline and matched or exceeded the best “Patient Outcomes” models, while linear-probing CEHR-GPT-L remained competitive, especially on rare-event tasks [2509.03643]. This is notable because the same pretrained model family was used for feature extraction, zero-shot forecasting via generated continuations, and unconditional synthetic-data generation [2509.03643].

Multimodal expansion appears in GDP, which natively encodes structured EHR time series with a CNN-transformer encoder, fuses them with unstructured clinical notes through cross-modal attention, and uses a LLaMA-based decoder for both prediction and generation [2508.16054]. On MIMIC-IV, GDP reported heart-failure AUROC $0.923$, type 2 diabetes AUROC $0.817$, and $30$-day readmission AUROC $0.627$; for narrative generation, GDP-Instruct reached ROUGE-L $0.135$ and BERTScore-F1 $0.545$, and in blinded human evaluation it scored highest on faithfulness, fluency, and overall clinical utility [2508.16054]. A plausible implication is that future CEHR-GPT systems will increasingly be multimodal rather than purely timeline-based.

Another extension direction is semantic and ontological enrichment. HiSGT constructs a hierarchy graph over clinical codes, derives hierarchy-aware embeddings with a GNN, extracts semantic embeddings from code descriptions using a pre-trained clinical language model, and fuses these with transformer token embeddings [2502.20719]. On MIMIC-IV, HiSGT reported unigram $R^2=0.989$ versus $0.969$ for GPT, bigram $R^2=0.967$ versus $0.935$, classification accuracy $0.905$ versus $0.883$, and attribute-inference F1 around $0.045$ for both HiSGT and GPT [2502.20719]. This suggests that one limitation of flat code-token generation is the absence of explicit hierarchy and semantics in the token space.

Several limitations recur across the CEHR-GPT literature. Sequence-length ceilings remain consequential: the 2024 synthetic CEHR-GPT excluded patients whose sequences exceed $512$ tokens, the radiology classifier truncated to $1024$ tokens, and the foundation model truncated at $4{,}096$ tokens [2402.04400]. Time granularity is often day-level rather than hour- or minute-level, and some variants exclude laboratory measurements or free text from the core input representation [2509.03643]. Zero-shot forecasting through autoregressive sampling is computationally expensive, and privacy evaluations are usually empirical rather than based on formal differential privacy guarantees [2509.03643]. The net result is a field that has established a coherent decoder-based paradigm for longitudinal EHR modeling, but not yet a single settled architecture.

Source: https://www.emergentmind.com/topics/cehr-gpt