TALE-EHR: Temporal Transformer for EHR

• TALE-EHR is a Transformer-based framework that integrates explicit temporal structure with language-derived clinical semantics for modeling longitudinal EHR data.
• It introduces a novel time-aware attention mechanism using continuous temporal kernels to handle irregular event intervals and complex clinical patterns.
• The architecture leverages pre-trained LLM embeddings for standardized code descriptions, demonstrating enhanced performance on benchmarks like MIMIC-IV and PIC.

TALE-EHR denotes a line of electronic health record modeling that combines explicit temporal structure with language-derived clinical semantics. In its explicit 2025 formulation, TALE-EHR is a Transformer-based framework for longitudinal EHR data that introduces a time-aware attention mechanism over irregular event sequences and uses embeddings of standardized code descriptions derived from a pre-trained LLM, specifically BGE, to encode heterogeneous clinical concepts (Yu et al., 20 Jul 2025). In adjacent usage, the label also denotes a broader class of approaches that textualize structured EHR and reuse LLMs as general EHR encoders, including frozen embedding pipelines over Markdown-serialized records (Hegselmann et al., 24 Feb 2025). Across these usages, the unifying objective is to preserve temporality, semantic richness, and interoperability in patient representations for downstream clinical prediction.

1. Definition, scope, and problem setting

TALE-EHR addresses three recurrent difficulties in longitudinal EHR analysis: heterogeneity across coding systems and data modalities, irregular time intervals between events, and complex temporal patterns spanning acute and chronic processes (Yu et al., 20 Jul 2025). The explicit TALE-EHR framework is evaluated on MIMIC-IV and PIC. For MIMIC-IV, the processed cohort contains 342,917 patients, 12,232 unique codes, and approximately 619 million event occurrences; tasks include next-visit medical code prediction, disease onset prediction for nine diseases, 30-day hospital readmission, and next-visit length of stay greater than 7 days. For PIC, the dataset contains 12,868 patient records and 2,607 unique codes; tasks include pneumonia prediction, heart malformation prediction, in-hospital mortality, and ICU length of stay greater than 7 days (Yu et al., 20 Jul 2025).

The explicit formulation treats the EHR as a marked temporal point process. For patient $i$, the history is written as

$$\mathcal{H}^{(i)} = \{(t_1^{(i)}, c_1^{(i)}), \dots, (t_{m_i}^{(i)}, c_{m_i}^{(i)})\},$$

where $t_j^{(i)}$ is the event timestamp and $c_j^{(i)}$ is a discrete event mark. TALE-EHR then learns a history representation $\mathbf{h}_t^{(i)}$ used for both event timing prediction and event-type prediction (Yu et al., 20 Jul 2025).

A broader, TALE-EHR-style interpretation appears in work that converts structured EHR data into text and uses general LLMs as encoders. One such formulation serializes demographics, labs, vitals, visits, conditions, medications, and procedures into Markdown, computes an embedding $z=f(x)$ from frozen LLM encoders, and trains a downstream logistic regression classifier $\hat y=\sigma(w^\top z+b)$ without EHR-specific pretraining (Hegselmann et al., 24 Feb 2025). This broader usage suggests that TALE-EHR is not only a named architecture but also a methodological orientation: temporalized EHR representation through language-aware encoding.

2. Temporal point-process architecture and time-aware attention

In the 2025 TALE-EHR model, individual codes are the atomic events rather than visits. Diagnoses, procedures, medications, labs, chart events, inputs and outputs, and DRG or other hospital-specific codes are represented as event pairs $(t_j, c_j)$ (Yu et al., 20 Jul 2025). The model uses Transformer-style self-attention at two levels: code-level time-aware self-attention to encode each event, and history-level time-aware attention to summarize all prior events into $\mathbf{h}_t^{(i)}$.

For each code $c$, a fixed semantic embedding $$\mathcal{H}^{(i)} = \{(t_1^{(i)}, c_1^{(i)}), \dots, (t_{m_i}^{(i)}, c_{m_i}^{(i)})\},$$0 is projected into queries, keys, and values: $$\mathcal{H}^{(i)} = \{(t_1^{(i)}, c_1^{(i)}), \dots, (t_{m_i}^{(i)}, c_{m_i}^{(i)})\},$$1 The standard scaled dot-product attention

$$\mathcal{H}^{(i)} = \{(t_1^{(i)}, c_1^{(i)}), \dots, (t_{m_i}^{(i)}, c_{m_i}^{(i)})\},$$2

is modified by a learnable time-dependent weighting function $$\mathcal{H}^{(i)} = \{(t_1^{(i)}, c_1^{(i)}), \dots, (t_{m_i}^{(i)}, c_{m_i}^{(i)})\},$$3 (Yu et al., 20 Jul 2025). At the code level, the attention kernel is

$$\mathcal{H}^{(i)} = \{(t_1^{(i)}, c_1^{(i)}), \dots, (t_{m_i}^{(i)}, c_{m_i}^{(i)})\},$$4

so continuous temporal gaps directly modulate attention scores. The temporal weighting function is parameterized as a sigmoid of a polynomial,

$$\mathcal{H}^{(i)} = \{(t_1^{(i)}, c_1^{(i)}), \dots, (t_{m_i}^{(i)}, c_{m_i}^{(i)})\},$$5

with the default setting using degree 5 polynomial. This construction is the central technical distinction of TALE-EHR: time is not appended merely as an input token or embedding but enters the attention computation itself as a continuous kernel (Yu et al., 20 Jul 2025).

History-level attention applies temporal weighting a second time. The patient representation is computed as

$$\mathcal{H}^{(i)} = \{(t_1^{(i)}, c_1^{(i)}), \dots, (t_{m_i}^{(i)}, c_{m_i}^{(i)})\},$$6

where the weights $$\mathcal{H}^{(i)} = \{(t_1^{(i)}, c_1^{(i)}), \dots, (t_{m_i}^{(i)}, c_{m_i}^{(i)})\},$$7 combine content relevance and temporal distance from the prediction time. This yields a multi-scale temporal representation intended to distinguish short-term and long-term effects, including differing temporal signatures of acute and chronic diseases (Yu et al., 20 Jul 2025).

Relative to prior EHR sequence models, TALE-EHR differs from time-aware RNNs such as T-LSTM or KIT-LSTM, which apply time decay to hidden states; from HiTANet, which uses hierarchical attention with time encoding; and from CEHR-BERT, which uses artificial time tokens and time2vec embeddings. The distinguishing claim is that continuous time gaps are modeled via a learned kernel function that directly modulates attention scores (Yu et al., 20 Jul 2025).

3. Semantic representation of clinical concepts

A second defining component of TALE-EHR is its use of language-model-derived embeddings for structured clinical codes. For each code $$\mathcal{H}^{(i)} = \{(t_1^{(i)}, c_1^{(i)}), \dots, (t_{m_i}^{(i)}, c_{m_i}^{(i)})\},$$8, a standardized description $$\mathcal{H}^{(i)} = \{(t_1^{(i)}, c_1^{(i)}), \dots, (t_{m_i}^{(i)}, c_{m_i}^{(i)})\},$$9 is retrieved from source-specific manuals and ontologies. ICD codes use WHO official descriptions; RxNorm uses NLM RxNorm descriptions; PheCodes use PheWAS catalog descriptions; CCS and DRG use their respective description tables; and local MIMIC codes use institutional description manuals (Yu et al., 20 Jul 2025). The description text is then embedded by BGE: $t_j^{(i)}$0 These embeddings are fixed during TALE-EHR training and provide the semantic basis for the attention projections $t_j^{(i)}$1, $t_j^{(i)}$2, and $t_j^{(i)}$3 (Yu et al., 20 Jul 2025).

The preprocessing pipeline explicitly reduces raw vocabulary sparsity by code roll-up. Procedures from ICD-9/10 PCS and HCPCS are mapped to CCS; diagnoses from ICD-9/10 CM are mapped to PheCodes where possible plus a subset of frequent raw ICD codes; medications are mapped from NDC to RxNorm; and ICU chart codes may be split further when frequent low-cardinality values occur (Yu et al., 20 Jul 2025). This makes TALE-EHR a code-sequence model with language-derived semantics rather than a raw text model.

A related but broader TALE-EHR-style strategy is to textualize the entire patient record. In the frozen-encoder formulation, structured EHR data are converted into Markdown with a header, demographics, aggregated recent body metrics, vital signs, lab results, visit summaries, non-visit events, and detailed visits in reverse chronological order; task-specific instructions are prepended; and the resulting document is fed to LLM embedding models such as GTE-Qwen2 and LLM2Vec (Hegselmann et al., 24 Feb 2025). A closely aligned hierarchical variant, PaReGTA, converts each visit into short templated text such as “62 days after previous, meds: ...” or “62 days after previous, comorbidities: ...”, fine-tunes GTE-base-v1.5 with unsupervised SimCSE, and aggregates visit embeddings with hybrid temporal pooling (Yoon et al., 23 Feb 2026). These systems preserve the central TALE-EHR intuition that language-space representations can mediate between local coding systems and clinically meaningful semantics.

4. Training objectives, benchmarks, and empirical performance

TALE-EHR is trained in two stages. Pretraining combines a temporal point-process loss for event times with focal loss for event-type prediction; fine-tuning uses task-specific MLP heads with weighted binary cross-entropy for downstream prediction (Yu et al., 20 Jul 2025). The pretraining stage lasts 10 epochs, the fine-tuning stage 5 epochs, Adam uses learning rate $t_j^{(i)}$4 during pretraining, and fine-tuning applies a smaller learning rate $t_j^{(i)}$5 to pretrained parameters with higher learning rate for task-specific heads. Event sequences are truncated to length 1024, and downstream models use up to 1024 events before the target time as input history (Yu et al., 20 Jul 2025).

Across tasks, TALE-EHR is compared against LSTM, RETAIN, RETAINEX, Dipole, HiTANet, and CEHR-BERT. On MIMIC-IV next-visit code prediction, TALE-EHR achieves Acc@5 of $t_j^{(i)}$6, Acc@10 of $t_j^{(i)}$7, Acc@20 of $t_j^{(i)}$8, macro F1 of $t_j^{(i)}$9, and Recall of $c_j^{(i)}$0. The strongest baseline listed, CEHR-BERT, reaches Acc@5 of $c_j^{(i)}$1, Acc@20 of $c_j^{(i)}$2, F1 of $c_j^{(i)}$3, and Recall of $c_j^{(i)}$4 (Yu et al., 20 Jul 2025).

On disease prediction in MIMIC-IV, TALE-EHR reports representative results including arteriosclerosis AUROC $c_j^{(i)}$5, AUPRC $c_j^{(i)}$6, F1 $c_j^{(i)}$7; type 2 diabetes AUROC $c_j^{(i)}$8, AUPRC $c_j^{(i)}$9, F1 $\mathbf{h}_t^{(i)}$0; acute kidney injury AUROC $\mathbf{h}_t^{(i)}$1, AUPRC $\mathbf{h}_t^{(i)}$2, F1 $\mathbf{h}_t^{(i)}$3; heart failure AUROC $\mathbf{h}_t^{(i)}$4, AUPRC $\mathbf{h}_t^{(i)}$5, F1 $\mathbf{h}_t^{(i)}$6; and hypertension AUROC $\mathbf{h}_t^{(i)}$7, AUPRC $\mathbf{h}_t^{(i)}$8, F1 $\mathbf{h}_t^{(i)}$9. The abstract reports average AUROC of about $z=f(x)$0 across diseases (Yu et al., 20 Jul 2025). On clinical benchmarks, 30-day readmission reaches AUROC $z=f(x)$1, AUPRC $z=f(x)$2, F1 $z=f(x)$3, while next-visit length of stay greater than 7 days reaches AUROC $z=f(x)$4, AUPRC $z=f(x)$5, F1 $z=f(x)$6 (Yu et al., 20 Jul 2025). On PIC, pneumonia reaches AUROC $z=f(x)$7, AUPRC $z=f(x)$8, F1 $z=f(x)$9; heart malformations AUROC $\hat y=\sigma(w^\top z+b)$0, AUPRC $\hat y=\sigma(w^\top z+b)$1, F1 $\hat y=\sigma(w^\top z+b)$2; mortality AUROC $\hat y=\sigma(w^\top z+b)$3, AUPRC $\hat y=\sigma(w^\top z+b)$4, F1 $\hat y=\sigma(w^\top z+b)$5; and ICU length of stay greater than 7 days AUROC $\hat y=\sigma(w^\top z+b)$6, AUPRC $\hat y=\sigma(w^\top z+b)$7, F1 $\hat y=\sigma(w^\top z+b)$8 (Yu et al., 20 Jul 2025).

Setting	TALE-EHR	Comparator
MIMIC-IV code prediction Acc@5	$\hat y=\sigma(w^\top z+b)$9	CEHR-BERT $(t_j, c_j)$0
30-day readmission AUROC	$(t_j, c_j)$1	CEHR-BERT $(t_j, c_j)$2
Heart failure AUROC	$(t_j, c_j)$3	CEHR-BERT $(t_j, c_j)$4
PIC mortality AUROC	$(t_j, c_j)$5	CEHR-BERT $(t_j, c_j)$6

Ablation studies identify the time-aware mechanism as the dominant contributor. Removing time-awareness reduces arteriosclerosis AUROC from $(t_j, c_j)$7 to $(t_j, c_j)$8 and heart failure from $(t_j, c_j)$9 to $\mathbf{h}_t^{(i)}$0. Replacing LLM embeddings with random embeddings also lowers performance, for example arteriosclerosis from $\mathbf{h}_t^{(i)}$1 to $\mathbf{h}_t^{(i)}$2 and type 2 diabetes from $\mathbf{h}_t^{(i)}$3 to $\mathbf{h}_t^{(i)}$4, though these drops are smaller than removing time-awareness (Yu et al., 20 Jul 2025). Piecewise time bins degrade performance relative to the continuous polynomial kernel, and polynomial order 10 is slightly worse than order 5, which is interpreted in the source as likely mild overfitting (Yu et al., 20 Jul 2025).

5. Relation to adjacent TALE-EHR-style methods

The broader literature represented here places TALE-EHR among several technically distinct but conceptually connected EHR representation strategies. The frozen-encoder LLM approach of “LLMs are Powerful Electronic Health Record Encoders” does not name TALE-EHR, but it is explicitly characterized as “TALE-EHR via frozen LLM embeddings.” It converts longitudinal structured records into Markdown with code descriptions, prepends task instructions such as “Will the patient stay in the hospital for more than 7 days,” and uses frozen embedding models followed by logistic regression. On EHRSHOT, GTE-Qwen2-7B attains overall macro AUROC $\mathbf{h}_t^{(i)}$5 versus $\mathbf{h}_t^{(i)}$6 for CLMBR-T-Base, and concatenating GTE-Qwen2-7B with CLMBR reaches $\mathbf{h}_t^{(i)}$7, indicating complementary information between general-purpose LLM embeddings and EHR-specific pretraining (Hegselmann et al., 24 Feb 2025).

PaReGTA extends the textualization strand in a hierarchical direction. It converts each visit into concept-specific text with explicit temporal cues, adapts GTE-base-v1.5 using unsupervised SimCSE, and aggregates visit embeddings with hybrid temporal pooling that combines exponential recency weighting and attention over informative visits. On 39,088 migraine patients from All of Us, PaReGTA-Gap with LightGBM attains AUC $\mathbf{h}_t^{(i)}$8, compared with $\mathbf{h}_t^{(i)}$9 for count Bag-of-Codes and $c$0 for one-hot encoding; RETAIN and T-LSTM were reported as failing to converge to reliable validation performance in that cohort (Yoon et al., 23 Feb 2026). This suggests one practical branch of TALE-EHR research emphasizes visit-level text plus lightweight temporal aggregation rather than end-to-end sequence training.

TAPER is an earlier multimodal precursor. It learns visit-level representations by combining a causal transformer over visit-level medical codes, BERT-based note embeddings summarized by a GRU autoencoder, and demographics. On MIMIC-III, the full representation improves over ClinicalBERT and other baselines for readmission, mortality, and length-of-stay prediction, while encoding time only through sinusoidal positional embeddings and causal masking, not explicit continuous temporal gaps (Darabi et al., 2019). TEE4EHR represents a different but related event-level strand: a transformer event encoder with point-process loss that models laboratory test ordering patterns and informative missingness in continuous time, then transfers the frozen encoder to downstream outcome prediction (Karami et al., 2024). TAML, by contrast, is time-aware in task space rather than sequence architecture: it treats multiple future horizons as separate meta-learning tasks and introduces a Temporal Information Sharing Strategy to augment positives across adjacent windows (Liu et al., 2023). HeaRT occupies the visualization end of the spectrum, using a BERT-based Japanese clinical information extraction system plus graph-based temporal reasoning to map free-text EHR narratives into patient timelines rendered in a Gantt-chart-like interface (Yada et al., 2023).

A separate adjacent literature concerns temporal granularity in causal inference rather than representation learning. “Targeted Learning with Daily EHR Data” shows that, for longitudinal TMLE on EHR data, moving from 90-day to 5-day intervals is computationally feasible with long-format TMLE and modern ML but increases variance despite reducing coarsening error (Sofrygin et al., 2017). A plausible implication is that TALE-EHR-style predictive models and causal EHR workflows face a shared design question: how much temporal resolution can be retained before estimation becomes unstable or computationally impractical.

6. Interpretability, limitations, and open directions

TALE-EHR includes several interpretability analyses. The learned temporal weighting curves $c$1 vary by disease: acute kidney injury shows rapid decay with time, while type 2 diabetes and heart failure decay more gradually, with a long-tail effect in which distant events retain small but non-zero weights (Yu et al., 20 Jul 2025). UMAP projections of history representations show clearer separation between positive and negative cases than an LSTM baseline, and the architecture supports inspection of which codes at which times receive the highest attention weights (Yu et al., 20 Jul 2025). Related systems develop complementary interpretability mechanisms: TEE4EHR aggregates self-attention across patients to estimate event-event interaction matrices, and PaReGTA-RSS measures representation shift after removing clinically defined factors such as medications, comorbidities, or recent temporal windows (Karami et al., 2024, Yoon et al., 23 Feb 2026).

The limitations of TALE-EHR proper are explicit. It focuses on structured codes and does not incorporate free-text notes, imaging, or raw numeric lab and vital values as separate continuous features. Its semantic component depends on the quality and coverage of standardized code descriptions, especially for local codes. Event-level pairwise temporal attention can be computationally intensive as histories grow, although fine-tuning truncates sequences to 1024 events (Yu et al., 20 Jul 2025). In the broader TALE-EHR-style literature, additional limitations recur: manual serialization choices can bias performance across tasks; context length and naive pooling can discard long-range information in textual encoders; and single-site evaluation leaves claims of coding-system robustness only partially tested (Hegselmann et al., 24 Feb 2025). PaReGTA further notes that its interpretability is predictive rather than causal and that its reported migraine experiments are not strictly prospective because they include periods after chronic diagnosis (Yoon et al., 23 Feb 2026).

The forward research agenda is correspondingly clear. TALE-EHR itself proposes multimodal extension to clinical notes and imaging, adaptation to more institutions and coding systems through text-based embeddings, and richer time kernels or joint modeling of time and marks (Yu et al., 20 Jul 2025). The textualization strand proposes serialization-free processing of raw EHR structures, better pooling mechanisms than mean pooling, larger context windows, zero-shot or few-shot prompting, and multi-institution benchmarks (Hegselmann et al., 24 Feb 2025). PaReGTA suggests coupling visit-level LLM encoders with deeper time-aware architectures and extending representation-shift explanations to multi-label and multi-timepoint settings (Yoon et al., 23 Feb 2026). Taken together, these directions indicate that TALE-EHR is best understood not as a single fixed model family, but as an emerging synthesis of continuous-time sequence modeling, language-derived clinical semantics, and modular patient representation learning for heterogeneous EHR data.