---
title: Electrocardiogram-Language Models (ELMs)
url: https://www.emergentmind.com/topics/electrocardiogram-language-models-elms
type: topic
---

# Electrocardiogram-Language Models (ELMs)

Electrocardiogram-Language Models (ELMs) are a heterogeneous class of models that couple electrocardiographic signals, electrocardiographic knowledge, or both with language modeling, بحيث ECG data can be queried, summarized, classified, retrieved, or forecast in natural language. The term is used across at least three partially overlapping paradigms: multimodal systems that condition text generation on ECG signals, domain-specific large language models adapted to electrocardiography, and self-supervised models that treat cardiac waveforms or rhythms as language-like token sequences [2410.14464] [2510.18339] [2602.23060]. This terminological breadth is not incidental: it reflects a research area in which “language” may denote textual clinical reasoning, symbolic ECG tokenization, or joint ECG–text representation learning.

## 1. Conceptual scope and historical formation

The intellectual basis for ELMs predates current multimodal LLM practice. “ECG Language Processing (ELP)” explicitly framed an ECG record as a sequence of heartbeats analogous to sentences, with P-wave, QRS complex, T-wave, and U-wave morphologies serving as word-like units; that work clustered wave snippets into a vocabulary, embedded token sequences, and modeled them with CNNs or Bi-LSTM architectures for heartbeat classification and atrial fibrillation detection [2006.08841]. Subsequent work extended the language analogy from symbolic sequence modeling to multimodal learning between ECGs and reports, including retrieval systems that aligned ECG images with clinical text through ViT–BERT-style architectures [2304.06286].

A later generation of work consolidated the modern ELM label but did not converge on a single canonical architecture. Some papers define ELMs as multimodal foundation models that jointly represent raw ECG waveforms and generate natural-language interpretations or reports, as in few-shot ECG question answering and long-context forecasting [2410.14464] [2602.15677]. Others use the term for domain-adapted LLMs specialized for electrocardiography literature and clinical knowledge rather than direct signal ingestion [2510.18339]. A parallel line treats ECG as a structured language for self-supervised learning, with rhythm-aware tokenization and masked prediction objectives [2602.23060].

| Family | Representative papers | Core formulation |
|---|---|---|
| Signal-as-language SSL | [2006.08841], [2502.10707], [2602.23060] | ECG waves, beats, or rhythms are tokenized and modeled with language-style objectives |
| Multimodal ECG-conditioned generation | [2410.14464], [2403.04945], [2602.15677] | ECG embeddings or ECG tokens are fused with an LLM to answer questions or generate reports |
| Domain-adapted and retrieval-grounded ECG LLMs | [2510.18339], [2505.03781], [2510.00261] | Textual ECG knowledge, rules, or retrieved documents ground ECG-oriented reasoning |

This diversity suggests that ELM is best understood as a research program rather than a single model family. A plausible implication is that comparisons across papers require attention to what is being aligned: waveform-to-language, report-to-language, or cardiology text-to-language.

## 2. Representation strategies

Representation design is the defining architectural fault line in ELM research. One branch uses explicit symbolic tokenization. ELP clustered normalized wave snippets into a vocabulary with \(K=20\) total clusters and converted ECGs into sequences of integer tokens [2006.08841]. HeartLang advanced this idea by introducing the QRS-Tokenizer, which centers segmentation on detected QRS complexes, forms heartbeat “words,” pads or truncates to a sentence length \(l=256\), and learns a vector-quantized vocabulary \(\mathcal V\) with \(k=8\,192\) entries of dimension \(d=128\), of which \(5\,394\) were used in validation [2502.10707]. RhythmBERT further imposed physiological structure by delineating P, QRS, and T segments, learning separate convolutional autoencoders with \(d_P=d_T=12\) and \(d_{QRS}=24\), clustering latents into \(k_P=13\), \(k_{QRS}=11\), and \(k_T=10\), and combining these discrete rhythm tokens with continuous morphology embeddings projected to \(d_{\text{model}}=196\) [2602.23060].

A second branch relies on learned ECG encoders that produce continuous embeddings for an LLM. In the few-shot ECG question-answering ELM, a self-supervised ECG encoder outputs a sequence \(E=\{e_1,\dots,e_K\}\), which is mapped by the trainable “Meta Mapper” into a prefix \(P\) prepended to textual token embeddings for a frozen decoder-only LLM; the default mapper uses 4 attention layers, 8 heads, and dropout \(0.5\) [2410.14464]. MEIT likewise converts a \(12\)-lead, \(10\) s ECG \(X_e \in \mathbb R^{M\times T}\) into ECG embeddings \(H_e\) and injects them as keys and values in every causal-attention layer of the LLM, rather than adding a separate cross-modal head [2403.04945]. CLIC uses a different multimodal design: a ResNet18-based ECG encoder produces a \(512\)-dimensional vector \(h^{ECG}\), a frozen ClinicalBERT text encoder produces a \(768\)-dimensional \(h^T\), and the concatenated \(1280\)-dimensional representation is passed to an MLP classifier [2605.19132].

A third branch minimizes modality-specific machinery. ECG-Byte replaces a pretrained ECG encoder with an adapted byte-pair encoding tokenizer. One implementation quantizes ECGs into an alphabet of size \(|\mathcal A|=26\), applies a default \(N_{\mathrm{merge}}=3500\), obtains a final vocabulary of approximately \(5856\), and reports a median encoded length of approximately \(800\) tokens per \(2\) s segment with a \(9\times\) compression ratio [2412.14373]. A later unified benchmark used \(5000\) merges on \(300\,000\) full preprocessed ECGs and reported a final vocabulary of size approximately \(10\,000\) [2505.18847]. ELF goes further by removing the ECG encoder entirely: a raw \(12\)-lead, \(5\) s, \(250\) Hz signal \(X \in \mathbb R^{C\times L}\) is flattened and mapped by a single linear projection \(z = Wx + b\), and the resulting vector is inserted as a “<signal>” token embedding in the LLM input sequence [2601.18798].

Image-based representations remain a viable but separate lineage. ECG retrieval work encoded denoised PTB-XL ECGs as Markov Transition Field, Gramian Angular Field, Recurrence Plot, or All-Grid images, then aligned them with reports using ViT and BERT projections into a \(256\)-dimensional normalized joint space [2304.06286]. The later unified representation benchmark found that symbolic representations achieved the greatest number of statistically significant wins over both signal and image inputs across six public datasets and five evaluation metrics [2505.18847].

## 3. Learning objectives and adaptation regimes

Training strategies in ELMs range from classic masked modeling to instruction tuning, meta-learning, retrieval conditioning, and curriculum learning. In domain-specific textual adaptation, ECG-LLM fine-tuned open-weight Llama 3.1 \(8\)B and \(70\)B models on approximately \(30\,000\) synthetic question–answer pairs generated from \(150+\) ECG-focused journal articles and textbooks. Fine-tuning used LoRA adapters on attention, feed-forward, and output projections, covering approximately \(3.7\%\) of model parameters, with AdamW and paged-32 quantization, cosine decay from \(1\mathrm{e}{-4}\) to \(1\mathrm{e}{-6}\), batch size \(8\) examples per H100 GPU, \(2\) epochs, weight decay \(0.01\), global norm clipping \(\le 1.0\), and cross-entropy on answer tokens only:
\[
L_{CE} = -\sum_t y_t \log p_t .
\]
The reported best LoRA setting was \(r=256, \alpha=128\) [2510.18339].

Few-shot multimodal ELMs use a different regime. The meta-learning QA model freezes the LLM backbone, trains only the fusion module, and optimizes under MAML with inner-loop task adaptation
\[
\theta_i' = \theta - \alpha \nabla_\theta L_{T_i}(f_\theta; D_i^s)
\]
and outer-loop meta-optimization
\[
L_{\text{meta}}(\theta) = \sum_{T_i \sim p(T)} L_{T_i}( f_{\theta_i'}; D_i^q ).
\]
This setup is explicitly designed for tasks with limited labeled ECG-question-answer triples and unseen attribute–answer classes [2410.14464].

Instruction tuning dominates report-generation ELMs. MEIT converts each ECG–report pair into a chat-style template and computes autoregressive cross-entropy only on the target report tokens, masking the prompt and ECG positions. The paper states that no separate contrastive or additional alignment loss is used; alignment is induced through shared cross-attention fusion and the autoregressive objective. Fine-tuning employs LoRA with rank \(128\), alpha \(64\), dropout \(0.1\), batch size \(64\) with gradient accumulation \(2\), sequence length \(256\), \(5\) epochs, and a linear learning-rate schedule with \(3\%\) warm-up [2403.04945].

Self-supervised ECG-language pretraining has increasingly become multi-objective. ESI pairs ECGs with LLM-generated, textbook-anchored text and combines a symmetric InfoNCE contrastive loss with a captioning loss over approximately \(660\)k pairs from PTB-XL, Chapman-Shaoxing, and MIMIC-IV-ECG [2405.19366]. MELP extends this to three scales—token, beat, and rhythm—by combining captioning, local contrastive alignment, and global contrastive alignment in
\[
L_{total} = L_{rhythm} + \lambda_1 L_{token} + \lambda_2 L_{beat}
\]
with default \(\lambda_1=2.0\) and \(\lambda_2=0.2\), trained on \(760\)k high-quality ECG–text pairs from MIMIC-IV-ECG [2506.21803]. RhythmBERT, by contrast, uses a standard BERT-style masked language modeling objective on \(20\%\) of discrete rhythm tokens after pretraining the tokenizer and transformer on \(800\,035\) MIMIC-IV-ECG recordings [2602.23060]. HeartLang separates form-level pretraining from rhythm-level pretraining through vector-quantized heartbeat reconstruction followed by masked ECG sentence prediction [2502.10707].

The most elaborate curriculum appears in CAMEL. Its frozen MedGemma-4B backbone is adapted with LoRA modules of rank \(128\) through five stages: ECG autoencoding on over \(1\) billion \(1\) s segments, multiple choice and short answer instruction tuning on \(20\) M examples, statistics questions, \(420\)K multi-turn dialogues, and a forecasting stage with \(17\)K examples from Icentia11k [2602.15677]. This suggests that, in the ELM literature, language-style supervision is increasingly used not only to map ECGs to text but also to scaffold clinical reasoning behaviors.

## 4. Tasks, evaluation methodology, and empirical performance

ELM evaluation is unusually heterogeneous. ECG-LLM explicitly uses a four-layer evaluation stack: multiple-choice accuracy on Full (\(27\,774\) MCQs), Special (\(1\,219\)), and Checked (\(534\)) sets; automatic text similarity with BLEU-4, ROUGE-1/2/L F1, and BERTScore F1; LLM-as-judge using Deepseek R1 on question–candidate–reference triples; and human expert review on \(10\) factual and \(40\) complex questions with graded labels from \(0\) to \(1.0\), with \(1\,000\)-replicate bootstrapping and \(95\%\) confidence intervals [2510.18339]. Under this protocol, Llama 3.1 \(70\)B + FT scored \(90.2\%\), \(92.0\%\), and \(88.2\%\) on Special, Full, and Checked multiple-choice sets, respectively; its text-similarity scores were ROUGE-1 \(0.4270\), ROUGE-2 \(0.2449\), ROUGE-L \(0.3764\), BLEU-4 \(0.1289\), and BERTScore \(0.3904\). Yet in LLM-as-judge Claude 3.7 answered \(273\) of \(417\) questions correctly versus \(260\) for \(70\)B+FT, and human expert review on complex questions favored RAG \(8\)B, base \(70\)B, and Claude 3.7, while \(70\)B+FT incurred a slight drop with \(3\) errors [2510.18339].

Few-shot ECG QA uses task-centric meta-test protocols rather than broad generative evaluation. The multimodal meta-learning ELM reports, in the \(5\)-way \(5\)-shot setting with LLaMA-3.1-8B, accuracies of \(84.6\%\), \(77.3\%\), and \(69.6\%\) on Single-Verify, Single-Choose, and Single-Query question types, respectively. Cross-domain adaptation from PTB-XL to MIMIC-IV-ECG improves from \(76.3/49.1/10.4\) without meta-adapt to \(89.7/85.7/39.7\) with meta-adapt in the \(2\)-way \(5\)-shot setting [2410.14464].

Report-generation benchmarks stress lexical and semantic overlap metrics. MEIT, trained on PTB-XL and MIMIC-IV-ECG, evaluated nine open-source decoder-only LLMs with nine metrics. On MIMIC-IV-ECG, LLaMA-2 and Mistral-Instruct led most metrics, with BLEU-4 approximately \(0.58/0.576\), METEOR approximately \(0.775/0.768\), ROUGE-L approximately \(0.745/0.751\), and BERTScore-F1 approximately \(0.744/0.747\). Zero-shot instruction-tuned models retained approximately \(90\%\) of their PTB-XL fine-tuned performance and outperformed “no-IT” by approximately \(15\)–\(20\%\) averaged over BLEU-3, BLEU-4, METEOR, and ROUGE-L [2403.04945].

Open-source RAG studies focus on grounded natural language generation. The RAG pipeline for ELMs reports, on ECG-Chat Instruct, an ECG-Byte baseline BLEU-4 of \(22.85 \pm 0.18\%\) and Accuracy \(9.65 \pm 0.29\%\), improving to BLEU-4 \(38.10 \pm 0.05\%\) and Accuracy \(18.27 \pm 0.05\%\) with RAG. On ECG-QA MIMIC-IV, BLEU-4 rises from \(15.52\) to \(49.07\) and Accuracy from \(27.21\%\) to \(57.77\%\). The ablations show that RAG only at inference yields BLEU-4 \(32.33\%\) and Accuracy \(14.96\%\), RAG only at training degrades below baseline to \(20.08\%\) and \(7.17\%\), and using RAG at both training and inference gives the best result [2510.00261].

Forecasting widens the task definition of ELMs beyond interpretation. CAMEL reports zero-shot evaluation over \(6\) tasks and \(9\) datasets, with an absolute \(+7.0\%\) average gain on ECGBench over prior SOTA ELMs and \(+12.4\%\) over fully supervised models on ECGForecastBench, including \(73.4\%\) macro-F1 at \(w=300\) s and \(h=60\) s, compared with \(56.9\%\) for XGB, \(53.6\%\) for CNN, and \(52.1\%\) for GPT-5.2 [2602.15677]. This broadening of scope implies that “ELM performance” cannot be reduced to a single benchmark family.

## 5. Retrieval, grounding, explainability, and deployment

Grounding mechanisms are central because free-form clinical language generation is vulnerable to hallucination. ECG-LLM implements a RAG pipeline over the same ECG literature used for fine-tuning, using recursive \(1{,}024\)-token chunks with \(100\)-token overlap, PubMedBERT embeddings of dimension \(768\), a Chroma vector database with approximate HNSW indexing, cosine similarity, retrieval of the top-\(20\) chunks, and reranking to top-\(5\) via dual-encoder rescoring [2510.18339]. The open-source RAG study generalizes this design space by indexing ECG features and reports in FAISS IndexIVFFlat, retrieving top-\(k\) nearest neighbors under \(L_2\) or cosine similarity, and showing that \(k=1\) to \(5\) is sufficient, while larger \(k\) adds noise [2510.00261].

ALFRED adds a more explicit diagnostic scaffold. It defines an ELM as the set \(\{\)Feature-Extraction Module + Rule Module + Retrieval Module + LLM Head\(\}\), uses an expert-curated knowledge base split into \(\le 1{,}024\)-character chunks, indexes \(1{,}024\)-dimensional embeddings in Milvus HNSW with \(M=10\) and \(efConstruction=100\), retrieves \(k=3\) passages for features and \(k=5\) for disease terms with a similarity threshold \(0.25\), and prompts GPT-4o-Mini to emit a JSON diagnosis over the five PTB-XL superclasses with explanations grounded in features and retrieved definitions [2505.03781]. On PTB-XL fold \(10\), the proposed ELM attained PPV \(0.443\), NPV \(0.834\), sensitivity \(0.477\), and specificity \(0.797\), versus \(0.326\), \(0.805\), \(0.356\), and \(0.754\) for the base framework without rules [2505.03781]. This architecture makes interpretability procedural rather than purely post hoc.

Textual context itself can function as a grounding modality. CLIC converts acquisition-time metadata into language either through a deterministic template or through an LLM-guided paragraph, then fuses the resulting text embedding with ECG features. On PTB-XL, ECG-Only achieved Accuracy \(0.714 \pm 0.017\) and Macro-F1 \(0.604 \pm 0.019\), ECG+Attr reached \(0.773 \pm 0.005\) and \(0.668 \pm 0.006\), CLIC-DET reached \(0.800 \pm 0.008\) and \(0.704 \pm 0.015\), and CLIC-LLM reached \(0.770 \pm 0.003\) and \(0.662 \pm 0.007\); for the minority CD class, F\(_1\) improved from \(0.308\) in ECG-Only to \(0.490\) in CLIC-DET [2605.19132]. The result indicates that controlled clinical text can be a stronger anchor than linguistically richer but more variable LLM-generated context under frozen text-encoder settings.

Explainability has also been explored through signal-to-token formulations. For intracardiac EGMs, tokenization into \(V=250\) discrete bins, vocabulary extension with signal and AFib label tokens, and joint MLM plus classification training enabled attention visualization, integrated gradients, and counterfactual analyses. Clinical Longformer achieved sensitivity \(99.9\%\), specificity \(99.6\%\), PPV \(99.9\%\), NPV \(99.3\%\), and accuracy \(99.7\%\) on AFib classification in the reported internal dataset [2402.01115]. Although EGMs are not standard surface ECGs, this line of work suggests that language-style tokenization can expose token-level interpretability not available in dense latent connectors.

Deployment concerns are unusually salient in ECG settings because local inference is often feasible. ECG-LLM reports that all fine-tuned and RAG models can be containerized on premise on NVIDIA A100/H100 hardware, with no PHI stored, retrieval over an encrypted vector database, optional ephemeral RAG contexts, static \(70\)B+FT inference at approximately \(180\) ms/token on A100, an additional approximately \(200\) ms/query for vector search and embedding in the RAG pipeline, resource requirements of \(2\times\)A100 \(80\) GB or \(4\times\)H100 for \(70\)B+FT, and a single \(40\) GB GPU for \(8\)B+RAG [2510.18339]. This makes privacy-preserving, locally deployable ELMs a distinct practical objective rather than a secondary implementation detail.

## 6. Limitations, misconceptions, and open directions

A common misconception is that high ELM benchmark scores necessarily imply strong ECG grounding. ELF directly challenges this assumption. On PTB-XL, training and inference with real ECG, zero tensors, or text only changed accuracy only modestly, from \(31.43\) to \(28.05\) to \(30.05\), and training on text alone while inferring on real ECG increased accuracy to \(37.09\) [2601.18798]. Similar perturbation behavior was observed for other models. This suggests that current benchmarks often permit substantial reliance on language priors, benchmark artifacts, or answer-format regularities rather than waveform-derived understanding.

A second misconception is that automatic metrics provide a stable ordering of systems. ECG-LLM shows strong disagreement across evaluation layers: fine-tuning dominates multiple-choice and text-overlap metrics, whereas human expert evaluation on complex questions prefers Claude 3.7 and RAG variants [2510.18339]. The few-shot meta-learning paper likewise notes that generative answers evaluated by overlap metrics may miss clinical correctness nuances [2410.14464]. These results suggest that ELM assessment requires task-specific triangulation rather than a single proxy score.

A third misconception is that more elaborate natural-language context is always better. CLIC demonstrates the opposite under its frozen-ClinicalBERT setup: deterministic template text outperforms LLM-generated contextual descriptions on both Accuracy and Macro-F1 [2605.19132]. A plausible implication is that lexical consistency and controlled semantics may matter more than narrative richness when the downstream fusion mechanism is simple concatenation.

Open directions in the literature are correspondingly diverse. RhythmBERT identifies multi-lead extension, dynamic rhythm generation, fine-grained interpretability aligning learned tokens with expert-defined morphologies, and cross-modal integration with metadata as key next steps [2602.23060]. MELP points toward external medical knowledge bases and more explicit clinically interpretable token-level objectives [2506.21803]. The few-shot multimodal ELM proposes multi-ECG comparison, additional modalities such as echo and X-ray, lightweight adapters or prompt tuning, and uncertainty quantification with human-in-the-loop verification [2410.14464]. CAMEL emphasizes longer context windows, alternative tokenization, and multimodal prognostication with other continuous vitals or structured records [2602.15677].

Taken together, these papers indicate that the field is moving in two simultaneous directions. One direction seeks stronger physiological inductive bias through rhythm-aware tokenization, multi-scale supervision, and longer temporal context. The other seeks stronger clinical grounding through retrieval, rules, metadata text, and privacy-preserving local deployment. Whether these strands ultimately converge into a single dominant ELM paradigm remains unresolved, but the literature already establishes that electrocardiography can be modeled as language at the levels of signal structure, clinical knowledge, and interactive reasoning.

Source: https://www.emergentmind.com/topics/electrocardiogram-language-models-elms