- The paper presents BERT-LER, combining percentile-based laboratory values within a transformer architecture for improved interpretability of clinical predictions on large-scale EHR records. Testing on public benchmarks, PLAB-LER performed well, specifically excelling on EHRShot laboratory tasks, using expert attribution insights into risk factors.
- In the model architecture, lab measurements are discretized into bins, offering scalability and capturing lab-specific meanings. On the asthmosis task, BERT-LER identifies meaningful signals like elevated nucleated erythrocyte ratios predicting exacerbation.
- Model attribution beyond coding also offers scalability and robustness, identifying clinical risk factors like obesity and race, although researchers acknowledge potential biases in lab feature associations.
BERT-LER is a BERT-style encoder model for coded electronic health record (EHR) timelines that combines two elements rarely treated jointly in prior work: laboratory-value representation that preserves graded quantitative information, and token-level interpretability via Integrated Gradients (IG). Pre-trained on a de-identified TriNetX Dataworks cohort of approximately 75 million patients, the model was evaluated on the 14 few-shot tasks of the public EHRShot benchmark and on a prospectively specified asthma severity progression study, where it achieved performance competitive with — and on laboratory-related tasks exceeding — the released CLMBR foundation model, while its attributions recovered clinically established risk factors (2608.20315).
Motivation and positioning
Structured EHR transformers such as Med-BERT, BEHRT, TransformEHR, and CLMBR have demonstrated strong clinical prediction performance, but two gaps persist. First, continuous laboratory measurements are difficult to incorporate into discrete token vocabularies without discarding value magnitude; Labrador, which pre-trains a transformer on lab trajectories alone within MIMIC, reported limited transfer gains (Bellamy et al., 2023). Second, existing interpretability approaches either do not connect features to outcomes (attention-score methods in TransformEHR) or operate on substantially smaller cohorts (~10–15k patients for ExBEHRT and M-BEHRT) without examining percentile-encoded lab values through the same attribution framework. BERT-LER's stated contribution is not a novel algorithmic primitive but the integration of percentile-based lab encoding and IG attribution at scale, over a broad structured vocabulary, validated on both a public benchmark and a prospectively defined clinical protocol.
Model architecture and laboratory encoding
The model follows the masked language modeling (MLM) pre-training recipe of TransformEHR, using a standard 12-layer/12-head BERT configuration with relative key-query positional embeddings and a 768-dimensional hidden size. Each patient timeline is encoded as three aligned arrays: coded event tokens (ICD-10/9 diagnoses, NDC/RxNorm medications, LOINC labs, CPT/HCPCS procedures, CVX vaccines, demographics), position IDs representing days since the first event, and percentile IDs.
Laboratory results are discretized into ten population-level percentile bins computed once over the full TriNetX reference population. Each lab measurement occupies one sequence position whose embedding is the sum of a lab-code embedding (e.g., a LOINC code) and a shared percentile embedding. This factorization requires roughly N+M embeddings for N tests and M bins, versus N×M for a per-combination vocabulary — the latter being what CLMBR's LOINC vocabulary effectively uses (~10× more lab tokens). The factorized design scales cheaply to finer bin resolution, at the cost that test-specific meaning of "high" must be learned in later layers rather than at the input, which the authors note may limit resolution for non-monotonic lab interpretations. Open-ended tail bins handle outliers, and missing data require no imputation since unobserved events simply never appear as tokens. During MLM, when a lab-code token is masked, the percentile embedding remains visible to the model.
A diagnosis-only MED-BERT baseline and an ablation variant retaining lab tokens but removing percentile-value embeddings were trained under identical hyperparameter sweeps.
Evaluation design
Two complementary settings were used. On EHRShot (6,739 Stanford patients, canonical splits), UMLS mapping translated SNOMED-coded EHRShot concepts into the ICD10CM-based pre-training vocabulary. On the asthma tasks, cohorts were drawn from the disjoint evaluation half of TriNetX (no pre-training overlap) with explicit inclusion criteria: mild-to-moderate asthma per GINA Step 2–4 therapy, 365-day baseline and follow-up windows ending January 2018 (pre-COVID-19), exclusions for other pulmonary disease and biologic use, and three mutually exclusive outcome profiles (exacerbation, symptoms without exacerbation, controlled) defining three tasks: binary loss-of-control, three-way multiclass, and exacerbation among symptomatic patients. Splits were 70/15/15 with stratified sampling, early stopping, ROC-AUC (and PR-AUC for asthma), and bootstrap confidence intervals from 1,000 resamples. Notably, count-based XGBoost and logistic regression baselines had access to the same lab-value information as BERT-LER (lab events keyed by code and percentile bin), differing only in representation — so BERT-LER's gains cannot be attributed to information advantage alone.
On EHRShot, BERT-LER outperformed CLMBR on all four laboratory prediction tasks with non-overlapping or near-non-overlapping confidence intervals in several cases — most strikingly hyponatremia (0.867 vs. 0.733) and anemia (0.910 vs. 0.891) — and on lupus diagnosis (0.81 vs. 0.69). Diagnosis and ICU-outcome tasks were comparable between the two models within statistical uncertainty; neither MED-BERT nor RETAIN beat both on any task. The lab-ablation variant dropped sharply on lab tasks (hyponatremia fell from 0.77 to 0.65), directly implicating the percentile-value embeddings in the laboratory-task gains.
On the asthma tasks, BERT-LER achieved the best PR-AUC and ROC-AUC across all three tasks:
| Task |
Model |
PR-AUC |
ROC-AUC |
| Loss-of-control |
BERT-LER |
0.599 |
0.682 |
|
XGBoost |
0.554 |
0.647 |
|
Med-BERT |
0.509 |
0.622 |
| Multiclass |
BERT-LER |
0.627 |
0.667 |
|
XGBoost |
0.611 |
0.656 |
| Exacerbation |
BERT-LER |
0.232 |
0.724 |
|
XGBoost |
0.204 |
0.688 |
The exacerbation task showed uniformly low PR-AUC due to class imbalance (~10% positives), yet BERT-LER's margin over baselines was largest there, suggesting longitudinal sequence modeling is particularly valuable for rare outcomes. Across all tasks, the lab-ablation variant performed between MED-BERT and full BERT-LER, indicating that both the occurrence and the magnitude of laboratory measurements contribute predictive signal beyond diagnosis codes.
CLMBR was excluded from the asthma comparison because its vocabulary codes diagnoses in SNOMEDCT_US while these cohorts use ICD10CM/ICD9CM, requiring substantial remapping — a practical portability limitation of the released foundation model worth noting independently of this paper's results.
Explainability
IG attributions are computed post-hoc against a zero-embedding baseline, with category-mean baselines used for demographic tokens and median-percentile baselines for labs to avoid positional bias; IG was chosen over perturbation methods such as SHAP for scalability, since its cost is independent of sequence length. For lab features specifically, attribution is run on the combined lab-plus-percentile embedding, and ordinal regression coefficients over decile bins rank lab tests by directional effect.
On the EHRShot anemia task, aggregated attributions recovered established epidemiology: children and the very elderly, women, gut functional disorders, high RBC-concentration and blood-oxygen tests toward normal results, and chemotherapy agents including gemcitabine toward abnormal ones. Hydroxyurea received strong attribution toward low values despite limited literature support — an example where the method surfaces hypotheses the authors flag as unverified. RETAIN's top features for the same task showed no clear link to anemia outcomes.
For asthma loss of control, top-ranked tokens aligned with known risk factors: severity codes, frequent SABA use (albuterol), LABA–ICS combinations, montelukast, COPD comorbidity, obesity, and Black/African American race (consistent with documented disparities), with adolescents 10–19 showing a negative association. Lab-level analysis added interpretable physiological signals: elevated nucleated erythrocyte ratios and base excess positively correlated with worsening (markers of marrow stress, hypoxemia, and ventilatory strain), while higher eGFR, hematocrit, oxygen saturation, vitamin D, and iron correlated negatively. These align with comparison-model features and published literature, supporting the claim that token-level attribution can remain clinically meaningful rather than merely performative.
The authors explicitly caution, however, that lab attributions are subject to confounding by indication and informed-presence bias: test ordering encodes care setting and acuity, so associations may reflect why and where a test was ordered rather than direct physiology. The percentile ablation partially disentangles occurrence from value effects, but directional lab findings are presented as hypothesis-generating only. A further caveat is that several separately ranked medication tokens correspond to the same agent (multiple RxNorm/NDC codes per drug), so medication attributions should be read jointly.
Limitations and open questions
Beyond the informed-presence bias above, the paper concedes several limitations. No confounder-invariant learning is applied, so attributions are predictive associations, not causal effects. Attribution validation is qualitative; no formal quantitative test of faithfulness or stability is performed, and identity-versus-value analyses are deferred. The shared percentile embedding may limit resolution for strongly non-monotonic lab interpretations. Novel clinical codes absent from training cannot be interpreted. Asthma performance is lower than on several EHRShot tasks, attributed to the difficulty of forecasting rare progression events and smaller fine-tuning cohorts. Two cohort exclusion criteria (death and severe-asthma biologic initiation) condition on post-baseline variables, biasing the cohort toward less severe trajectories; a time-to-event formulation with censoring is identified as a natural alternative left unexplored. Finally, whether factorized percentile embeddings scale gracefully to finer bin counts in terms of learned representation quality — beyond parameter-count advantages — remains empirically unresolved here.
Conclusion
This paper demonstrates that a single EHR transformer pipeline can simultaneously preserve graded laboratory information through percentile-factorized embeddings and produce token-level IG attributions grounded in input medical events, at a pre-training scale (75 million patients) larger than most prior explainable EHR models. Its strongest quantitative claims are the consistent superiority on EHRShot laboratory tasks and the largest margins on the hardest, most imbalanced asthma exacerbation task, with the lab-value ablation isolating percentile magnitude as a substantive signal source. The work leaves open the quantitative validation of attribution faithfulness, causal handling of care-setting confounding in lab ordering, and time-to-event modeling of asthma progression.
