CardiacCLIP: Multimodal Cardiac AI via Contrastive Learning

• CardiacCLIP is a multimodal framework that aligns cardiac imaging and clinical reports using contrastive learning and supervised adaptation.
• It employs innovations like Multi Frame Learning and EchoZoom to enhance few-shot left ventricular ejection fraction prediction from echocardiography videos.
• Extensions to 3D CT and ECG modalities demonstrate its versatile application across various cardiac imaging tasks.

CardiacCLIP denotes a cardiac adaptation of the Contrastive Language-Image Pre-training paradigm in which cardiac videos, images, volumes, or physiological signals are aligned with clinical language in a shared representation space. In published usage, the name refers directly to a video-based CLIP adaptation for few-shot left ventricular ejection fraction (LVEF) prediction from echocardiography (Du et al., 21 Sep 2025). Closely related work treats multi-view echocardiography video-LLMs, 3D cardiac CT vision-LLMs, and ECG report-aligned signal-LLMs as near-direct realizations of the same design logic: modality-specific encoders, clinical text encoders, cosine-similarity alignment, and transfer to low-label or zero-shot cardiac tasks (Takizawa et al., 26 Apr 2025, Hu et al., 29 Jul 2025, Yu et al., 25 May 2026).

1. Terminological scope and research lineage

The immediate antecedent of CardiacCLIP is EchoCLIP, a multimodal echocardiography foundation model trained on 1,032,975 cardiac ultrasound videos and corresponding expert interpretations from 224,685 studies and 99,870 patients. EchoCLIP established that a CLIP-style joint space between echocardiographic images and reports can support zero-shot LVEF estimation, device detection, retrieval, and longitudinal reasoning (Christensen et al., 2023).

Subsequent work specialized this paradigm in different ways. The 2025 CardiacCLIP paper reformulated echocardiographic LVEF prediction as video-based few-shot ordinal regression built on CLIP representations, while introducing Multi Frame Learning (MFL) and EchoZoom (Du et al., 21 Sep 2025). A separate 2025 echocardiography paper replaced the image encoder of a CLIP model with a video encoder and extended the input from a single apical four-chamber frame to five standard views and full video sequences, explicitly presenting the model as a CLIP-style design for cardiac ultrasound interpretation (Takizawa et al., 26 Apr 2025). Parallel extensions appeared in cardiac CT and electrocardiography: Cardiac-CLIP for 3D cardiac CT used a two-stage 3D MAE plus contrastive training pipeline (Hu et al., 29 Jul 2025), and ECGCLIP aligned ECG waveforms with expert reports at scale using signal-language contrastive learning (Yu et al., 25 May 2026).

Model	Modality and supervision	Representative emphasis
EchoCLIP / EchoCLIP-R (Christensen et al., 2023)	Apical-4-chamber echo videos with expert interpretations	Zero-shot LVEF, device detection, retrieval, longitudinal similarity
Video CLIP model (Takizawa et al., 26 Apr 2025)	Five-view echo videos with Japanese reports	ViViT video encoder, multi-view mean pooling
CardiacCLIP (Du et al., 21 Sep 2025)	Echo videos with LVEF interval prompts	Few-shot LVEF prediction via MFL, EchoZoom, ordinal regression
Cardiac-CLIP (Hu et al., 29 Jul 2025)	3D cardiac CT with radiology reports	3D MAE pre-training and soft-label contrastive learning
ECGCLIP (Yu et al., 25 May 2026)	ECG waveforms with expert reports	Signal-language CMA plus ECG-only UMA
CLIC (Lucafo et al., 18 May 2026)	ECG with contextual text	Supervised late fusion, explicitly not CLIP-style

This lineage suggests that CardiacCLIP is best understood not only as a single model name but also as a recurring multimodal design pattern in cardiac AI.

2. Core formulation: joint embeddings, prompts, and supervision regimes

In its canonical CLIP-style form, CardiacCLIP-like systems learn a shared embedding space in which a cardiac sample and its paired report are close under cosine similarity. The multi-view echocardiography video-LLM computes video embeddings $v_i$, report embeddings $t_i$, and optimizes a symmetric contrastive loss over video-to-report and report-to-video directions, with cosine similarity

$$\text{sim}(v_i, t_j) = \frac{v_i^\top t_j}{\|v_i\|\|t_j\|}.$$

The same general structure appears in EchoCLIP, which follows OpenCLIP, and in ECGCLIP, whose Cross-Modal Alignment loss is a symmetric InfoNCE objective over ECG-report pairs (Takizawa et al., 26 Apr 2025, Christensen et al., 2023, Yu et al., 25 May 2026).

The few-shot echocardiographic CardiacCLIP model is architecturally CLIP-derived but changes the downstream learning objective. Rather than retaining contrastive pre-training as the task loss, it uses CLIP visual and text towers for supervised coarse-to-fine ordinal regression of LVEF. The coarse stage predicts a distribution over LVEF bins using text embeddings as classifier weights, and the fine stage predicts an intra-bin shift. Its combined objective is

$L_{OR} = L_{CE} + L_{MAE},$

with final prediction

$$y^{*} = \sum_{i=1}^{k} p_i \cdot \frac{b_i}{1 + \delta_i}.$$

This is a central distinction: CardiacCLIP, in the narrow sense of (Du et al., 21 Sep 2025), is a CLIP adaptation rather than a purely contrastive training recipe.

Cardiac CT work introduces a further variation. Cardiac-CLIP preserves the contrastive vision-language structure but replaces hard one-to-one targets with pathology-vector-derived soft labels. For a batch of cases with pathology vectors $\mathbf{y}_i \in \{-1,1\}^D$, the soft-label matrix is

$$\tilde{\mathbf{A}}_{ij} = \frac{\mathbf{y}_i \cdot \mathbf{y}_j}{\|\mathbf{y}_i\| \cdot \|\mathbf{y}_j\|},$$

and the contrastive loss is supervised by this semantic affinity matrix rather than an identity matrix. This directly encodes graded clinical similarity among multi-label cardiac CT findings (Hu et al., 29 Jul 2025).

A recurring technical consequence is that “CardiacCLIP” does not denote one fixed objective. In the literature it spans pure contrastive training, contrastive pre-training followed by supervised adaptation, and multimodal prompting schemes that retain CLIP’s text-conditioned classification semantics.

3. Echocardiographic CardiacCLIP for few-shot LVEF prediction

The model introduced as “CardiacCLIP” in 2025 targets few-shot LVEF prediction from echocardiography videos and is built on a ConvNeXt-Base CLIP visual encoder and a CLIP text transformer (Du et al., 21 Sep 2025). It addresses three technical deficits of image-only echo-LLMs: lack of temporal aggregation, weak localization of left-ventricular structure, and poor label efficiency in small-data settings.

Its temporal module, Multi Frame Learning, is a Multiple Instance Learning-style attention pooler over frame embeddings. For frame features

$$F = [f_1, f_2, \dots, f_B] \in \mathbb{R}^{B \times C},$$

frame scores are computed as

$s_i = W_3 \tanh\big( W_2 \tanh( W_1 f_i ) \big),$

weights are normalized by

$$\alpha_i = \frac{\exp(s_i)}{\sum_{j=1}^{B} \exp(s_j)},$$

and the aggregated video representation is

$t_i$0

This is not temporal self-attention over frame tokens; it is learned scalar attention over independently encoded frames.

Its spatial module, EchoZoom, is a multi-resolution input scaling strategy that uses the same pretrained CLIP visual encoder at $t_i$1 and $t_i$2. A $t_i$3 frame is split into four non-overlapping $t_i$4 sub-images, all processed through the same encoder, and the resulting features are fused across scales. The paper states that EchoZoom does not add extra learnable parameters (Du et al., 21 Sep 2025).

The few-shot protocol discretizes LVEF into integer classes 1–100 and constructs 1/2/4/8-shot subsets from the EchoNet-Dynamic training set of 7,465 videos. Dataset sizes are 84, 162, 307, and 570 samples for 1-, 2-, 4-, and 8-shot settings, respectively. Training uses 48 sampled frames per clip with temporal stride 2, RAdam, 100 epochs, batch size 2, and initial learning rate $t_i$5, cosine-decayed to zero (Du et al., 21 Sep 2025).

Empirically, the model improves over both standard supervised video models and CLIP-derived baselines. On EchoNet-Dynamic, MAE is 7.25 in the 1-shot setting, compared with 9.32 for EchoNet and 7.91 for NumCLIP; 7.11 in 2-shot, compared with 9.17 for EchoNet and 7.56 for NumCLIP; 6.79 in 4-shot, compared with 7.49 for EchoNet and 7.68 for NumCLIP; and 6.42 in 8-shot, compared with 6.81 for EchoNet and 6.96 for NumCLIP. In 1-shot RMSE, CardiacCLIP achieves 9.06 versus 12.11 for EchoNet and 9.89 for NumCLIP (Du et al., 21 Sep 2025).

Ablation results isolate the source of improvement. In 1-shot MAE, a NumCLIP-like base model yields 7.91, adding MFL gives 7.42, adding EchoZoom gives 7.50, and combining both gives 7.25. Heavier temporal modules degrade performance: MFL with GRU gives 8.26, multi-head attention over frames gives 10.47, and multi-head attention plus GRU gives 8.3. Frame-length ablation identifies 48 frames as optimal: 16 frames gives 7.89, 36 gives 7.64, 48 gives 7.25, 54 gives 7.72, 64 gives 7.38, 96 gives 7.92, and 128 gives 7.94 (Du et al., 21 Sep 2025).

These results indicate that, in few-shot echocardiography, parameter-efficient temporal aggregation and multi-scale spatial reuse of the pretrained visual tower are more effective than heavier sequence models.

4. Multi-view video-language echocardiography and report retrieval

A separate line of work broadens the CardiacCLIP concept from single-view apical four-chamber videos to full multi-view echocardiography. The 2025 “Video CLIP Model for Multi-View Echocardiography Interpretation” trains on 60,747 cases from The University of Tokyo Hospital, with 747,900 training videos across five standard transthoracic views: parasternal long-axis (LAX), parasternal short-axis (SAX), apical two-chamber (2CH), apical three-chamber (3CH), and apical four-chamber (4CH) (Takizawa et al., 26 Apr 2025).

The visual encoder is ViViT, taking sequences of 32 frames and producing a 512-dimensional video embedding. The text encoder is BERT-base Japanese v3, with the text context expanded from 77 tokens to 256 to accommodate long echocardiography reports in Japanese. Multi-view fusion is not implemented through separate branches or view tokens. Instead, each video $t_i$6 is passed through a shared encoder $t_i$7, yielding $t_i$8, and the study-level representation is the arithmetic mean

$t_i$9

This late-fusion design naturally supports variable numbers of clips and missing views (Takizawa et al., 26 Apr 2025).

Evaluation uses cross-modal retrieval rather than disease labels. The proposed MultiVideo model achieves MCMRR 595 for video-to-report and 584 for report-to-video, with $$\text{sim}(v_i, t_j) = \frac{v_i^\top t_j}{\|v_i\|\|t_j\|}.$$0 values of 10.9% and 10.3%, respectively. Restricting the same trained model to 4CH at inference gives MCMRR 705 and 695, with $$\text{sim}(v_i, t_j) = \frac{v_i^\top t_j}{\|v_i\|\|t_j\|}.$$1 of 8.4% and 8.0%. A single-view video model trained only on 4CH yields MCMRR 676 and 686, with $$\text{sim}(v_i, t_j) = \frac{v_i^\top t_j}{\|v_i\|\|t_j\|}.$$2 of 8.8% and 7.1%. An image-only single-frame CLIP analogue yields MCMRR 1315 and 1246, with $$\text{sim}(v_i, t_j) = \frac{v_i^\top t_j}{\|v_i\|\|t_j\|}.$$3 of 2.3% and 3.4% (Takizawa et al., 26 Apr 2025).

Two technical conclusions are explicit. First, switching from single-frame to video roughly doubles retrieval performance. Second, using multiple views at inference provides an additional improvement over single-view video. The MultiVideo-4CH versus SingleVideo comparison further suggests that most of the multi-view benefit arises from actually using multiple views at inference rather than from multi-view training alone (Takizawa et al., 26 Apr 2025).

This trajectory extends, rather than replaces, EchoCLIP’s earlier frame-based strategy. EchoCLIP trained on apical-4-chamber videos but treated videos as bags of frames, sampling a single random frame per epoch for training and averaging over 10 frames at inference for tasks such as zero-shot LVEF regression. EchoCLIP achieved external LVEF MAE of 7.1%, internal PAP MAE of 10.8 mmHg, and AUCs between 0.84 and 0.98 for device detection, while EchoCLIP-R introduced a custom report tokenizer that reduced mean report length from $$\text{sim}(v_i, t_j) = \frac{v_i^\top t_j}{\|v_i\|\|t_j\|}.$$4 GPT-2 BPE tokens to $$\text{sim}(v_i, t_j) = \frac{v_i^\top t_j}{\|v_i\|\|t_j\|}.$$5 tokens and improved mean cross-modal retrieval rank from 571.3 to 206.1 (Christensen et al., 2023).

Taken together, these echocardiographic studies show a progression from image-language alignment, to retrieval-optimized long-context tokenization, to explicit video modeling, and finally to multi-view study-level aggregation.

5. Extensions to cardiac CT, ECG, and contextual cardiac language

Cardiac-CLIP generalizes the CardiacCLIP principle to 3D cardiac CT. It uses a two-stage pre-training strategy: a 3D masked autoencoder on 130,889 CT scans followed by contrastive alignment on 11,106 paired cardiac CT scans and radiology reports. Reports are standardized into seven abnormality sentences using GPT-4o, pathology vectors are constructed for coronary stenosis, coronary calcification, aortic calcification, atherosclerosis, cardiomegaly, pericardial effusion, and pulmonary arterial hypertension, and a soft-label similarity matrix supervises contrastive learning. In downstream evaluation, the model is reported to outperform Merlin, CT-CLIP, M3DNet, and a 3D-ViT baseline across classification and retrieval tasks; on prospective prediction of acute coronary syndrome it reaches AUROC 0.802 after fine-tuning, compared with 0.529 for fully supervised 3D-ViT, and on functional coronary stenosis it reaches mean AUROC $$\text{sim}(v_i, t_j) = \frac{v_i^\top t_j}{\|v_i\|\|t_j\|}.$$6 under 5-fold cross-validation (Hu et al., 29 Jul 2025).

ECGCLIP provides the signal-language analogue. It was pre-trained on 2,837,962 ECG studies from 1,324,856 patients and evaluated on 89 downstream tasks spanning 45 ECG diagnoses, 39 echocardiographic targets, and 5 rare cardiac diseases. Its Cross-Modal Alignment loss aligns ECGs and expert reports, while Uni-Modal Alignment adds a second symmetric InfoNCE loss between two dropout-perturbed views of the same ECG, without waveform-level augmentation. On the internal test set, ECGCLIP-R34 achieves PRAUC 0.900 for atrial fibrillation and 0.383 for ST-segment elevation myocardial infarction, and also improves performance on rare diseases such as Ebstein anomaly, constrictive pericarditis, dextrocardia, and cardiac amyloidosis, with internal PRAUC values of 0.253, 0.175, 0.121, and 0.201, respectively. It also matches or exceeds full-dataset baseline performance with only 10% of training data (Yu et al., 25 May 2026).

CLIC occupies an adjacent but conceptually distinct position. It fuses raw 12-lead ECG signals with contextual patient and acquisition information that has been converted into language and encoded by frozen ClinicalBERT. Its best-performing configuration, CLIC-DET, uses deterministic template-based text rather than LLM-generated narrative reports and achieves macro-F1 $$\text{sim}(v_i, t_j) = \frac{v_i^\top t_j}{\|v_i\|\|t_j\|}.$$7 and general accuracy $$\text{sim}(v_i, t_j) = \frac{v_i^\top t_j}{\|v_i\|\|t_j\|}.$$8 on PTB-XL, outperforming ECG-only and ECG+Attr baselines overall. However, the model is explicitly not trained with a CLIP-style contrastive objective; it uses supervised late fusion and BCE classification only (Lucafo et al., 18 May 2026).

These studies show that the core CardiacCLIP idea is modality-agnostic. It can be instantiated in 2D frame-based echo, video echo, 3D CT, and ECG waveforms, but the exact encoder class, supervision regime, and task formulation vary substantially.

6. Technical trade-offs, misconceptions, and open directions

A common misconception is that CardiacCLIP necessarily means end-to-end contrastive pre-training on paired cardiac data. The literature is more heterogeneous. EchoCLIP, the multi-view video-language echocardiography model, Cardiac-CLIP for CT, and ECGCLIP all use explicitly contrastive alignment objectives, whereas the named CardiacCLIP model for LVEF prediction uses CLIP backbones but replaces contrastive optimization with supervised ordinal regression, and CLIC is multimodal yet explicitly non-CLIP in training objective (Christensen et al., 2023, Takizawa et al., 26 Apr 2025, Hu et al., 29 Jul 2025, Du et al., 21 Sep 2025, Lucafo et al., 18 May 2026).

A second misconception is that richer or freer text is always better. EchoCLIP-R shows that a custom echocardiography report tokenizer can greatly improve retrieval while degrading continuous zero-shot estimation, with EF MAE worsening to approximately 16.9–17.0% and device-detection AUCs dropping relative to EchoCLIP. CLIC similarly finds that controlled template-based contextual text produces more stable downstream performance than LLM-generated narratives, even though the latter are semantically richer (Christensen et al., 2023, Lucafo et al., 18 May 2026).

A third misconception is that heavier temporal modeling is always advantageous for cardiac video. The few-shot CardiacCLIP ablations show the opposite in low-label echocardiography: simple MIL-style attention pooling over frame embeddings outperforms GRU and multi-head attention variants, and the multi-view echo retrieval study shows that late fusion by mean pooling across view-specific embeddings already yields a substantial gain over single-view or image-only settings (Du et al., 21 Sep 2025, Takizawa et al., 26 Apr 2025).

The principal limitations are similarly consistent across papers. EchoCLIP trains on a single healthcare system and a single view; the multi-view video-LLM is single-center and single-language; CardiacCLIP for LVEF is validated only on apical four-chamber EchoNet-Dynamic; Cardiac-CLIP relies on largely non-public clinical CT cohorts and report standardization; ECGCLIP, despite broad external validation, is pre-trained predominantly on one large Chinese hospital cohort. Across modalities, the literature repeatedly identifies the need for multi-center scaling, broader modality coverage, improved handling of view identity or structured context, and clinically usable interpretability (Christensen et al., 2023, Takizawa et al., 26 Apr 2025, Hu et al., 29 Jul 2025, Du et al., 21 Sep 2025, Yu et al., 25 May 2026).

Future directions are already explicit in these works. The multi-view echocardiography study points to inter-hospital collaboration and LLMs that incorporate video for end-to-end report generation (Takizawa et al., 26 Apr 2025). Cardiac-CLIP suggests extension to other cardiovascular imaging modalities and integration with EHR data (Hu et al., 29 Jul 2025). ECGCLIP motivates broader multimodal cardiac foundation models that could align ECG, echocardiography, MRI, and text in a unified latent space (Yu et al., 25 May 2026). A plausible implication is that the term CardiacCLIP will increasingly designate not one architecture, but a family of cardiac foundation models whose common feature is the use of clinically grounded language as a supervisory scaffold for cardiac representation learning.