DeepCORO-CLIP: Multi-View Coronary Angiography Model

• The paper introduces DeepCORO-CLIP, the first multi-view, video-text foundation model for coronary angiography, achieving state-of-the-art performance across multiple diagnostic tasks.
• It employs a two-phase training protocol using contrastive learning and task-specific fine-tuning with attention-based pooling across multiple video projections.
• Validated on large internal and external cohorts, the model demonstrates significant improvements in lesion detection, stenosis quantification, and transfer learning for prognostic applications.

DeepCORO-CLIP is a multi-view foundation model for coronary angiography video-text analysis designed to support comprehensive study-level interpretation rather than isolated frame or projection analysis. It was trained with video-text contrastive learning on 203,808 angiography videos from 28,117 patients across 32,473 studies at the Montreal Heart Institute and externally validated on 4,249 studies from the University of California, San Francisco. The model integrates multiple projections with attention-based pooling for diagnostic, prognostic, and disease progression tasks, including significant stenosis detection, chronic total occlusion, intracoronary thrombus, coronary calcification, one-year major adverse cardiovascular events, and left ventricular ejection fraction estimation (Harrabi et al., 18 Mar 2026).

1. Clinical problem and model scope

Coronary angiography is described as the reference standard for evaluating coronary artery disease, yet visual interpretation remains variable between readers. Existing artificial intelligence methods are characterized as typically analyzing single frames or projections and focusing mainly on stenosis, which limits comprehensive coronary assessment. DeepCORO-CLIP was introduced to address this gap by operating at the study level across multiple diagnostic videos and by supporting both classification and regression objectives within a shared representational framework (Harrabi et al., 18 Mar 2026).

The paper positions the model as a foundation model for coronary angiography rather than a narrow task-specific detector. In the authors’ summary, it is identified as the first multi-view, video-text foundation model for coronary angiography and is reported to achieve state-of-the-art performance across diagnostic tasks including stenosis, calcification, thrombus, and chronic total occlusion. A plausible implication is that the term “foundation model” is being used here to denote a reusable representation learned from large-scale video-text alignment and then adapted through downstream fine-tuning, rather than a model restricted to a single endpoint.

A common misconception would be to interpret DeepCORO-CLIP as a single-lesion stenosis classifier. The reported experiments indicate a broader scope: study-level aggregation across up to 10 diagnostic videos, transfer learning for one-year major adverse cardiovascular events, left ventricular ejection fraction estimation, and embedding-based analysis of serial examinations. This suggests that the central object of modeling is the angiographic study as a multi-view clinical entity rather than an isolated frame.

2. Architecture and multi-view representation

The video encoder is a Multiscale Vision Transformer v2 (MViT-v2) with 12 transformer blocks and multiscale pooling, pretrained on Kinetics-400. Input clips are 16-frame sub-sequences sampled at 7.5–15 fps, spatially resized to $224 \times 224$ and patch-embedded with patch size $16 \times 16$ and stride 2. The model uses 3D positional encodings for spatiotemporal modeling. During contrastive pretraining, 80–94% of encoder layers are frozen, with the best single-view configuration freezing approximately 80%, a design intended to preserve low-level representations while adapting higher layers to angiography semantics (Harrabi et al., 18 Mar 2026).

The text encoder is BioMedBERT-base, with 12 layers and hidden size 768, pretrained on PubMed abstracts. The freezing ratio explored 60–90%, with best performance at approximately 75%. Token embeddings are mean-pooled to produce a 512-dimensional vector. Video and text are therefore aligned in a shared 512-dimensional space.

Multi-view integration is a defining element of the system. Up to 10 diagnostic videos per study are processed independently to yield 512-dimensional per-video vectors. These are fused with attention-based pooling, specifically gated attention with a learnable CLS token, into a single 512-dimensional study representation. Two-stage aggregation was also explored, first within-video patches via a CLS token and then across videos. The ablation result reported in Figure 2B shows that “multi-video attention” outperformed mean or max pooling by $\Delta \mathrm{AUROC} +0.060$ for stenosis detection. Task-specific MLP heads, implemented as a single linear layer with dropout $p=0.2$, map the CLS-pooled study embedding to either binary logits for thrombus, CTO, calcification, and stenosis $\geq 70\%$, or to a scalar regression for percent stenosis.

The architectural significance lies in the explicit handling of projection multiplicity. The model does not collapse the study into a naive average of views; instead, it learns a view-weighted aggregation. This suggests that the representation is intended to capture complementary anatomical and pathological information distributed across standard angiographic projections.

3. Training protocol and objective functions

Training proceeded in two phases. Phase 1 was contrastive video-text pretraining using an InfoNCE objective that aligns video embeddings $v_i$ and text embeddings $t_i$ in a shared 512-dimensional space. For a batch of $N$ pairs, the paper states the objective as

$$\nabla L = - \frac{1}{N} \sum_i \log \left[ \frac{\exp(\mathrm{sim}(v_i, t_i)/\tau)}{\sum_{j=1}^{N}\exp(\mathrm{sim}(v_i, t_j)/\tau)} \right]$$

where $\mathrm{sim}(\cdot,\cdot)$ is cosine similarity and the temperature $16 \times 16$0 was optimized in $16 \times 16$1, with best performance at approximately $16 \times 16$2 (Harrabi et al., 18 Mar 2026).

For the best single-view contrastive run, the reported hyperparameters were batch size 32, learning rate $16 \times 16$3, weight decay $16 \times 16$4 for video and $16 \times 16$5 for text, AdamW optimization, a linear-warmup scheduler, stride $16 \times 16$6, and 30 epochs. Freeze ratios were 80% for video and 74.6% for text. On a held-out test set of 4,851 reports, video-to-text Recall@10 reached 16.7%, with median rank 156, which the paper interprets as demonstrating semantic alignment.

Phase 2 was task-specific fine-tuning. Three encoder-weight configurations were tested: fully frozen, partially unfrozen with the top 20% of layers trainable, and a configuration augmenting the CLS token with view-position embeddings. The partially unfrozen configuration performed best. Input consisted of up to 10 videos per study, each 16 frames, with randomized augmentations via RandAugment and mixed-precision training. Binary tasks used binary cross-entropy with logits, learning rate $16 \times 16$7, and weight decay $16 \times 16$8. Stenosis regression used Huber loss, learning rate $16 \times 16$9, weight decay $\Delta \mathrm{AUROC} +0.060$0, and loss weight multiplied by 2 relative to classifiers. Fine-tuning used 50 epochs, AdamW with warm restarts, and seed 42.

A plausible implication of this two-phase design is that video-text pretraining supplies a clinically structured initialization, while partial unfreezing preserves the pretraining geometry and still permits adaptation to angiography-specific supervisory signals.

4. Data resources and external validation

The internal cohort came from the Montreal Heart Institute and comprised 28,117 patients, 32,473 studies, and 203,808 diagnostic videos, split 70% train, 15% validation, and 15% test by patient. Videos were classified into 12 standard projections using a fine-tuned X3D-M model with AUROC 0.94, then filtered for diagnostic phase and contrast presence. Reports annotated by interventional cardiologists supplied segment-level labels for stenosis, calcification, thrombus, and CTO (Harrabi et al., 18 Mar 2026).

The external validation set consisted of 4,249 studies from UCSF collected between 2013 and 2019. Stenosis labels were automatically extracted with an on-premise LLM, DeepSeek-V2, with 99% spot-check accuracy. Quantitative coronary angiography adjudication was performed on 662 randomly sampled studies for head-to-head comparison.

The internal and external data design is notable for combining large-scale routine clinical video archives with out-of-institution evaluation. The paper’s framing emphasizes that external validation was not limited to a narrow benchmark subset but used an institutional cohort and a QCA-adjudicated subset, which supports assessment of cross-site generalization. This suggests that the reported performance is intended to reflect both routine deployment conditions and more stringent lesion-level adjudication.

5. Diagnostic and quantitative performance

On internal testing for continuous stenosis quantification, spanning 30,268 videos and 86,544 segment labels, the reported mean absolute error was 9.37% with 95% confidence interval 8.99–9.90, and Pearson $\Delta \mathrm{AUROC} +0.060$1 with confidence interval 0.516–0.533. Performance was strongest in proximal segments, with proximal RCA reaching $\Delta \mathrm{AUROC} +0.060$2. For stenosis $\Delta \mathrm{AUROC} +0.060$3 detection on the internal test set, AUROC was 0.888, AUPRC 0.517, sensitivity 84.7%, and specificity 70.2%. On external UCSF validation, AUROC was 0.89, mean absolute error 12.1%, and Pearson $\Delta \mathrm{AUROC} +0.060$4, with per-segment AUROC ranging from 0.80 to 0.95 (Harrabi et al., 18 Mar 2026).

Performance extended beyond stenosis. For calcification, AUROC was 0.903 for LCA and 0.907 for RCA internally, with per-segment AUROCs up to 0.977 in distal LCX. For thrombus, AUROC was 0.898, AUPRC 0.226, sensitivity 73.7%, and specificity 91.8%, with prevalence only 0.25%. For chronic total occlusion, AUROC was 0.923 globally, 0.898 for LCA, and 0.946 for RCA, with sensitivity up to 91%.

The model was benchmarked against a supervised baseline consisting of MViT pretrained on Kinetics-400 with linear probing. For stenosis $\Delta \mathrm{AUROC} +0.060$5, AUROC improved from 0.769 to 0.888, mean absolute error from 10.4% to 9.37%, and AUPRC from 0.286 to 0.517. Gains were larger for rare lesions, with thrombus AUPRC increasing from 0.007 to 0.226. In the core laboratory adjudication subset of 662 studies and 965 lesions, DeepCORO-CLIP achieved study-level mean absolute error versus QCA of 13.56% compared with 18.97% for clinical reports, and Pearson $\Delta \mathrm{AUROC} +0.060$6 compared with 0.56 for reports.

These results bear directly on the interpretation of “comprehensive” in the model’s title. The reported endpoints span binary lesion detection, continuous stenosis estimation, rare pathology detection, and comparison against both external data and QCA. A plausible implication is that the model’s utility is linked not only to average discriminative performance but also to coverage of multiple clinically salient lesion types within a unified representation.

6. Transfer learning, embeddings, and deployment

DeepCORO-CLIP was also evaluated as a transfer-learning backbone. For one-year major adverse cardiovascular events, excluding dropouts in the first 7 days, the dataset included 1,047 training cases, 349 validation cases, and 350 test cases, stratified by composite MACE. Endpoints were urgent revascularization, non-fatal myocardial infarction, and coronary occlusion. Composite MACE prevalence was 31.7%, and the model achieved AUROC 0.790, AUPRC 0.648, sensitivity 84%, and specificity 76%. Kaplan–Meier analysis showed a high-risk versus low-risk hazard ratio of 18.6 (Harrabi et al., 18 Mar 2026).

For left ventricular ejection fraction estimation from angiograms, using echocardiography within 30 days, the reported split was 4,382 training pairs, 674 validation pairs, and 2,530 test pairs. Regression performance was mean absolute error 7.30% compared with 8.78% for the CathEF baseline, and Pearson $\Delta \mathrm{AUROC} +0.060$7 compared with 0.50. In classification mode, AUROC for $\Delta \mathrm{AUROC} +0.060$8 was 0.861 versus 0.828, and for $\Delta \mathrm{AUROC} +0.060$9 it was 0.815 versus 0.763.

Embedding analysis further characterized the learned representation. A t-SNE of 512-dimensional video embeddings showed clear clustering by coronary side, stenosis status, and lesion location. For serial examinations, study-level embedding distance defined as $p=0.2$0 was measured in 401 study pairs: stable anatomy had mean distance 0.260, improved cases 0.352 with $p=0.2$1 versus stable, and progressed cases 0.353 with $p=0.2$2 versus stable. The authors interpret this as indicating that the embeddings capture both progression and regression of disease, offering a continuous metric for longitudinal monitoring.

Deployment details underscore the system’s intended point-of-care use. The paper reports PACS-AI integration on-premises, with automatic ingestion of DICOM from the cath lab, diagnostic series selection, preprocessing, and inference on a single NVIDIA RTX A6000 GPU. End-to-end latency was approximately $p=0.2$3 seconds per study across 24 measurements, including DICOM decoding, frame sampling, model inference, and report templating. Deployment was containerized with Docker and used versioned logging for traceability. Code, model weights, and deployment infrastructure were publicly released, along with the controlled-access DeepCORO-mini dataset comprising 1,000 cases and approximately 7,000 videos.

Taken together, these experiments indicate that DeepCORO-CLIP is not limited to static lesion annotation. The reported transfer results and embedding analyses suggest a reusable angiographic representation that supports downstream prognostic modeling, functional estimation, and longitudinal disease characterization within a single multi-view video-text framework.