---
title: Disease-Level Vision Contrastive Learning
url: https://www.emergentmind.com/topics/disease-level-vision-contrastive-learning
type: topic
---

# Disease-Level Vision Contrastive Learning

Searching arXiv for recent papers on disease-level vision contrastive learning and closely related medical vision-language contrastive learning.
Recent work uses the term disease-level vision contrastive learning for a family of representation-learning strategies in which the contrastive objective is organized around clinically meaningful disease semantics rather than only image identity, coarse image–report pairing, or generic augmentation invariance. In the strongest form, disease concepts are embedded directly in a shared vision–text space and the model is optimized so that scans are closer to positive disease statements than to matched negative statements, while preserving broader image–text alignment for retrieval and transfer [2603.02026]. In adjacent forms, disease-level structure is imposed through supervised disease pairs, clinical proxy labels, soft semantic targets derived from report labels, lesion-centered sampling, or multi-granular medical language. Across these variants, the common aim is to make the latent space more discriminative for pathology, comorbidity, severity, or clinically relevant findings than conventional global contrastive pretraining.

## 1. Definition and conceptual scope

Disease-level vision contrastive learning departs from standard instance discrimination and from coarse global image–report alignment by treating disease semantics as an explicit organizing principle of the embedding space. In 3D CT vision–language pretraining, the contrast between this approach and vanilla report–volume alignment is especially clear: aligning an entire CT volume with an entire radiology report provides broad semantic supervision for retrieval, but remains coarse because one report may contain findings, negations, anatomy, incidental observations, history, and comparison statements, all compressed into a single global target [2603.02026]. The resulting representation may therefore be effective for scan–report matching while remaining weak at isolating specific diseases.

The same general critique appears in related medical contrastive literature, though operationalized differently. Relation-enhanced medical vision-language contrastive learning argues that local region–word alignment remains insufficient if it ignores relations among local matchings such as disease words and their contextual qualifiers [2401.10501]. Multi-granular language learning argues that CLIP-style one-image–one-text supervision is too restrictive for medical images that legitimately correspond to multiple disease concepts and multiple textual granularities [2511.15943]. Lesion-based contrastive learning for diabetic retinopathy makes the point in purely visual terms: whole-image instance discrimination is poorly matched to a disease whose grading depends on localized lesion evidence [2107.08274].

A useful synthesis is that disease-level contrastive learning is not one method but a design principle. It can be instantiated as text-conditioned metric learning in a shared vision–text space, supervised contrastive grouping by disease class, weakly supervised grouping by clinical state variables, lesion-aware sampling, or soft-target alignment over related disease concepts. This suggests a spectrum from direct disease supervision to clinically guided proxy supervision.

## 2. Supervision regimes and sources of disease semantics

One major axis of variation is how disease semantics are obtained. In “Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT” [2603.02026], disease supervision is derived from reports rather than from manual radiologist disease labels. The core private dataset, RefCT, contains **97,760 report–volume pairs from 50,474 patients**, with **78k pairs in the training split**, and joint pretraining uses up to **159k paired report–volume examples** across RefCT, CT-RATE, Merlin, and INSPECT. Structured binary findings are extracted from reports using the **RATE protocol from Pillar-0**, which applies **LLM-based question answering** to reports and yields **93 chest** and **226 abdomen** binary findings across 30 organ categories [2603.02026]. Disease awareness is therefore induced by report-derived labels and prompt supervision rather than by a separate hand-annotated disease benchmark.

Other papers use more explicit or more indirect supervision. Forward-Forward Contrastive Learning is directly supervised by class labels for **pneumonia versus normal classification from chest X-rays**, so positives are same-class pairs and negatives are different-class pairs [2305.02927]. RECLF uses free-text reports plus soft semantic targets derived from **CheXpert** report labels, so disease information enters at the pair level while local disease semantics are learned from token–region interactions and their relations [2401.10501]. MGLL uses structured multi-label supervision and disease/explanation text at multiple granularities, including disease category and clinical explanation in fundus imaging and disease labels plus reports in MIMIC-CXR supplementary experiments [2511.15943].

A distinct but related line uses clinical metadata as surrogate disease structure. “Clinical Contrastive Learning for Biomarker Detection” and “Clinically Labeled Contrastive Learning for OCT Biomarker Classification” define positives and negatives using routine clinical variables such as **BCVA**, **CST**, and **Eye ID**, under the premise that these clinical labels approximate the latent disease distribution relevant to biomarker detection [2211.05092; 2305.15154]. LEARNER similarly organizes positives and negatives using scalar health-score proximity, such as **S/F** in lung ultrasound or **MMSE** in ADNI MRI, so the embedding space is shaped by disease-severity geometry rather than class identity [2411.01144]. This suggests a broader interpretation in which disease-level supervision need not be exact diagnostic labeling if clinically meaningful surrogate variables organize pathology-relevant similarity.

## 3. Core mechanisms for disease-aware contrastive learning

A central mechanism is to embed disease semantics in the same shared space used for image–text alignment. In 3D CT disease-aware pretraining, the image encoder is initialized from **SPECTRE** and uses a frozen **ViT-Large local backbone** plus a trainable **4-layer global feature combiner**, while the text encoder is **Qwen3-Embedding (0.6B)** with **LoRA adapters**. Both sides are projected with **SigLIP projection heads** into a **512-dimensional shared space** [2603.02026]. Disease supervision is injected as text prompts representing positive and negative disease statements rather than as a separate classifier head.

For each binary finding \(q\), the model builds **three positive and three negative prompt variants**. With \(\mathbf z\) the L2-normalized global image embedding, \(\mathbf p_q^+\) the positive prompt embedding, and \(\mathbf p_q^-\) the negative prompt embedding, the disease logit is defined as
\[
x_q = (\mathbf z^\top \mathbf p_q^{+}-\mathbf z^\top \mathbf p_q^{-}) / \tau .
\]
The loss is a weighted BCE over valid findings,
\[
\mathcal{L}_{\text{prompt} = \frac{1}{|\mathcal{M}|}\sum_{q\in\mathcal{M} w_q\Big( -\alpha_q\, y_{q}\,\log\sigma(x_{q}) -(1- y_{q})\log\!\bigl(1-\sigma(x_{q})\bigr) \Big)
\]
with \(\alpha_q=\min(n^+_q / n^-_q, 20)\), and the total objective is
\[
\mathcal{L}_{\text{total} = \mathcal{L}_{\text{global} + \lambda \, \mathcal{L}_{\text{prompt} + \beta \, \mathcal{L}_{\text{loc}
\]
with \(\lambda = 8\) and \(\beta = 1\) [2603.02026]. Conceptually, this is a multi-label, per-disease logistic supervision in the shared contrastive embedding space, rather than a classic batchwise CLIP loss over disease prompts.

Other mechanisms operate at different granularities. RECLF first constructs local image–text matchings by cross-attention from report words to image patches, then applies a **semantic-relation reasoning module (SRM)** and an **importance-relation reasoning module (IRM)** so that disease semantics emerge from relations among local matchings rather than from independent region–word pairs [2401.10501]. MGLL replaces one-hot CLIP targets with soft targets over multiple disease texts and granularities, and supplements this with point-wise BCE and smooth KL consistency across granularities [2511.15943]. CT-DiagVLM goes further toward explicit disease-centric modeling in 3D CT by extracting organ embeddings on the visual side and disease-conditioned report embeddings on the text side using learnable condition queries, with a disease-level contrastive loss defined over organ-condition pools [2606.25546].

Lesion-centered methods alter the contrastive unit itself. Lesion-based diabetic retinopathy pretraining contrasts **\(128 \times 128\)** lesion patches rather than whole fundus images [2107.08274]. SWCL creates patch-level abnormality labels from CAMs and combines them with patch position and patient identity in a multi-label contrastive objective [2108.02122]. FocusContrast modifies positive-pair generation by rejecting crops or cutouts that remove radiologist-attended regions, thereby making positive views more pathology-preserving [2305.08826]. These methods do not all use language, but they instantiate the same idea that contrastive positives and negatives should respect disease-bearing structure.

## 4. Representative paradigms across imaging settings

Several distinct paradigms now coexist.

| Paradigm | Representative method | Disease signal source |
|---|---|---|
| Shared-space disease prompts | RadFinder [2603.02026] | RATE-derived report findings with positive/negative prompts |
| Supervised disease-pair contrast | FFCL [2305.02927] | Same-class vs different-class labels |
| Relation-aware report supervision | RECLF [2401.10501] | Report words plus CheXpert-derived semantic targets |
| Multi-granular disease language | MGLL [2511.15943] | Disease labels, explanations, and multi-granular texts |
| Clinical proxy contrast | Clinical/OCT methods [2211.05092; 2305.15154] | BCVA, CST, Eye ID |
| Severity-guided contrast | LEARNER [2411.01144] | Scalar health scores such as S/F or MMSE |
| Lesion-centered contrast | DR and retinal methods [2107.08274; 2108.02122] | Lesion patches or pseudo-localized abnormality labels |

In 3D CT, RadFinder is the clearest example of disease-level prompt supervision inside a VLM training loop. It retains large-scale report–volume pretraining but adds explicit disease supervision in the same shared space [2603.02026]. CT-DiagVLM is closely related in spirit but uses organ-level visual embeddings and disease-conditioned report embeddings extracted by learnable condition queries, with positives pooled across patients sharing a disease within an organ [2606.25546].

In chest radiography, FFCL is disease-aware in a more direct supervised sense. It performs local contrastive learning at each block, then global contrastive learning, using class labels to define positive and negative pairs, so the learned representation is explicitly disease-discriminative for pneumonia detection [2305.02927]. RECLF instead remains in the medical vision-language regime, modeling relations among local token–patch matchings to better encode disease and context jointly [2401.10501].

In ophthalmic imaging, clinically guided OCT methods are notable because they operationalize disease relevance without explicit disease-class pretraining labels. The learned representation is organized by clinical-state or severity proxies such as CST or BCVA, then transferred to biomarker classification [2211.05092; 2305.15154]. SWCL and lesion-based DR contrastive learning make pathology locality central by pretraining on weakly supervised or detector-mined lesion patches [2108.02122; 2107.08274]. PoCo adds a different form of disease-aware inductive bias by using polar transformation to better capture retina-centered structure and rotation-related textures relevant for ophthalmic disease diagnosis [2403.19124].

A plausible implication is that disease-level contrastive learning is most effective when the supervision matches the clinical granularity of the downstream task: global disease prompts for scan-level diagnosis, lesion-centered units for lesion-driven grading, and clinical-state proxies when biomarker or severity labels are scarce.

## 5. Empirical evidence and what it shows

The clearest quantitative demonstration that disease supervision changes the embedding space comes from RadFinder’s ablations. With **global loss only**, the model obtains **CT-RATE R@10 = 29.4**, **CT-RATE AUC = 56.9**, and **Rad-ChestCT AUC = 62.8**. With **prompt loss only**, it obtains **CT-RATE R@10 = 5.6**, **CT-RATE AUC = 84.8**, and **Rad-ChestCT AUC = 78.6**. The full combined model reaches **CT-RATE R@10 = 31.5**, **CT-RATE AUC = 83.8**, and **Rad-ChestCT AUC = 77.0** [2603.02026]. This directly shows that global report–volume contrast gives strong retrieval but poor disease discrimination, prompt supervision gives strong disease classification but poor retrieval, and the combined objective yields the best tradeoff.

That paper also shows transfer from internally derived disease supervision: training on **RefCT alone** gives **CT-RATE AUC = 80.4** and **Rad-ChestCT AUC = 77.7** [2603.02026]. This is especially relevant because the supervision is report-derived and not manually labeled for disease classification. In CT-DiagVLM, adding disease-level contrastive learning on top of a hybrid encoder and diagnosis-aware prompts improves CT-RATE from **82.4 AUC** to **84.4 AUC**, and the full model reports **75.4 AUC** on Rad-ChestCT with a **+5.4% AUC** improvement, with even larger gains on a **60-disease benchmark** [2606.25546].

Outside CT, supervised disease-pair contrast also yields measurable gains. FFCL with **ResNet-18** and **Local \(\rightarrow\) Global + random** reaches **Accuracy 80.45**, **F1 75.69**, **Precision 87.70**, **Recall 74.02**, and **AUC 93.33**, improving accuracy by **3.69 percentage points** over **RBP + ImageNet** on pediatric pneumonia classification [2305.02927]. RECLF reports **AUROC 0.88**, **Accuracy 0.67**, **Precision 0.67**, and **F1 0.67** for zero-shot classification on MIMIC-5x200, improving over MGCA-ViT’s **AUROC 0.84**, **Accuracy 0.60**, **Precision 0.62**, and **F1 0.59** [2401.10501]. MGLL shows especially large gains on multi-label retinal disease tasks such as RFMiD, where full fine-tuning reaches **92.83 AUC / 64.99 mAP** versus CLIP’s **65.10 AUC / 17.31 mAP** [2511.15943].

Lesion-centered contrastive methods also support the same conclusion from a different angle. Lesion-based DR pretraining reaches **66.88** quadratic weighted kappa in linear evaluation on full EyePACS with threshold **0.9**, compared with **55.32** for the stronger whole-image SimCLR baseline, and delivers **68.37** kappa at **1%** labeled transfer versus **53.36** for supervised ImageNet initialization [2107.08274]. SWCL surpasses prior self-supervised retinal methods and standard cross-entropy training across several classification and segmentation benchmarks while narrowing the gap to ImageNet pretraining [2108.02122]. These results suggest that disease-aware contrastive structure improves not only zero-shot or prompt-based classification, but also the transfer quality of the visual encoder itself.

## 6. Limitations, caveats, and unresolved questions

A recurrent limitation is dependence on noisy or indirect supervision. RadFinder’s disease supervision depends on the **RATE LLM-based label extraction**, and the paper does not quantify disease-label extraction accuracy there [2603.02026]. CT-DiagVLM’s extended disease taxonomy relies on LLM-generated pseudo-labels and external report parsing [2606.25546]. RECLF depends on CheXpert-derived soft semantic targets [2401.10501]. Clinical proxy methods assume that variables such as BCVA, CST, or MMSE are sufficiently aligned with latent pathology, which is plausible in the reported settings but not guaranteed in general [2211.05092; 2305.15154; 2411.01144].

Another limitation is granularity mismatch. Some methods are disease-aware only at the scan level, not at the lesion or region level. RadFinder’s prompt objective acts on the **global scan embedding** and is explicitly “not region-level disease alignment” [2603.02026]. CT-DiagVLM aligns disease-conditioned text to organ-level visual embeddings rather than to disease-localized 3D tokens [2606.25546]. This suggests that improved pathology discrimination does not automatically imply explicit localization or disentanglement of coexisting findings.

A further caveat is that “disease-level” can denote materially different regimes. FFCL is supervised disease-contrastive pretraining, not self-supervised disease discovery [2305.02927]. Clinically labeled OCT methods are better described as clinical-state-aware or severity-aware contrastive learning than as direct disease-label contrastive learning [2211.05092; 2305.15154]. Metadata-enhanced retinal OCT pretraining is disease-aware through patient/eye/time structure rather than through disease labels themselves [2208.02529]. An encyclopedia treatment therefore has to distinguish direct disease-semantic supervision from proxy-supervised and metadata-guided variants.

Finally, several papers leave reproducibility gaps. LEARNER does not fully specify the weighted contrastive loss, exact pair fusion, or some ADNI details [2411.01144]. VisTA does not fully specify optimization or the exact classification loss, and its embedding is organized more around abnormality types than around disease labels [2502.01535]. RECLF omits some graph-attention details [2401.10501]. This suggests that the field has converged more clearly on the principle of disease-aware metric structuring than on a single standardized implementation template.

Taken together, the literature indicates that disease-level vision contrastive learning is best viewed as a family of medically structured contrastive objectives that replace or augment generic instance-level similarity with disease-conditioned, pathology-aware, severity-aware, or clinically contextual similarity. The strongest current evidence comes from settings where disease supervision is embedded directly into a shared vision–text space and jointly optimized with broader multimodal alignment, rather than relegated to a downstream classifier [2603.02026].

Source: https://www.emergentmind.com/topics/disease-level-vision-contrastive-learning