Disease-Level Vision Contrastive Learning
- Disease-level vision contrastive learning organizes embeddings around clinical disease semantics instead of generic image features.
- It employs methods such as report-derived prompts, supervised disease pairs, and lesion-aware sampling to improve diagnostic and transfer performance.
- Empirical results demonstrate that integrating disease supervision enhances both retrieval metrics and pathology discrimination in medical imaging.
Searching arXiv for papers on disease-level vision contrastive learning and closely related medical vision-language contrastive learning. Recent work uses the term disease-level vision contrastive learning for a family of representation-learning strategies in which the contrastive objective is organized around clinically meaningful disease semantics rather than only image identity, coarse image–report pairing, or generic augmentation invariance. In the strongest form, disease concepts are embedded directly in a shared vision–text space and the model is optimized so that scans are closer to positive disease statements than to matched negative statements, while preserving broader image–text alignment for retrieval and transfer (Ging et al., 2 Mar 2026). In adjacent forms, disease-level structure is imposed through supervised disease pairs, clinical proxy labels, soft semantic targets derived from report labels, lesion-centered sampling, or multi-granular medical language. Across these variants, the common aim is to make the latent space more discriminative for pathology, comorbidity, severity, or clinically relevant findings than conventional global contrastive pretraining.
1. Definition and conceptual scope
Disease-level vision contrastive learning departs from standard instance discrimination and from coarse global image–report alignment by treating disease semantics as an explicit organizing principle of the embedding space. In 3D CT vision–language pretraining, the contrast between this approach and vanilla report–volume alignment is especially clear: aligning an entire CT volume with an entire radiology report provides broad semantic supervision for retrieval, but remains coarse because one report may contain findings, negations, anatomy, incidental observations, history, and comparison statements, all compressed into a single global target (Ging et al., 2 Mar 2026). The resulting representation may therefore be effective for scan–report matching while remaining weak at isolating specific diseases.
The same general critique appears in related medical contrastive literature, though operationalized differently. Relation-enhanced medical vision-language contrastive learning argues that local region–word alignment remains insufficient if it ignores relations among local matchings such as disease words and their contextual qualifiers (Li et al., 2024). Multi-granular language learning argues that CLIP-style one-image–one-text supervision is too restrictive for medical images that legitimately correspond to multiple disease concepts and multiple textual granularities (Li et al., 20 Nov 2025). Lesion-based contrastive learning for diabetic retinopathy makes the point in purely visual terms: whole-image instance discrimination is poorly matched to a disease whose grading depends on localized lesion evidence (Huang et al., 2021).
A useful synthesis is that disease-level contrastive learning is not one method but a design principle. It can be instantiated as text-conditioned metric learning in a shared vision–text space, supervised contrastive grouping by disease class, weakly supervised grouping by clinical state variables, lesion-aware sampling, or soft-target alignment over related disease concepts. This suggests a spectrum from direct disease supervision to clinically guided proxy supervision.
2. Supervision regimes and sources of disease semantics
One major axis of variation is how disease semantics are obtained. In “Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT” (Ging et al., 2 Mar 2026), disease supervision is derived from reports rather than from manual radiologist disease labels. The core private dataset, RefCT, contains 97,760 report–volume pairs from 50,474 patients, with 78k pairs in the training split, and joint pretraining uses up to 159k paired report–volume examples across RefCT, CT-RATE, Merlin, and INSPECT. Structured binary findings are extracted from reports using the RATE protocol from Pillar-0, which applies LLM-based question answering to reports and yields 93 chest and 226 abdomen binary findings across 30 organ categories (Ging et al., 2 Mar 2026). Disease awareness is therefore induced by report-derived labels and prompt supervision rather than by a separate hand-annotated disease benchmark.
Other papers use more explicit or more indirect supervision. Forward-Forward Contrastive Learning is directly supervised by class labels for pneumonia versus normal classification from chest X-rays, so positives are same-class pairs and negatives are different-class pairs (Ahamed et al., 2023). RECLF uses free-text reports plus soft semantic targets derived from CheXpert report labels, so disease information enters at the pair level while local disease semantics are learned from token–region interactions and their relations (Li et al., 2024). MGLL uses structured multi-label supervision and disease/explanation text at multiple granularities, including disease category and clinical explanation in fundus imaging and disease labels plus reports in MIMIC-CXR supplementary experiments (Li et al., 20 Nov 2025).
A distinct but related line uses clinical metadata as surrogate disease structure. “Clinical Contrastive Learning for Biomarker Detection” and “Clinically Labeled Contrastive Learning for OCT Biomarker Classification” define positives and negatives using routine clinical variables such as BCVA, CST, and Eye ID, under the premise that these clinical labels approximate the latent disease distribution relevant to biomarker detection (Kokilepersaud et al., 2022, Kokilepersaud et al., 2023). LEARNER similarly organizes positives and negatives using scalar health-score proximity, such as S/F in lung ultrasound or MMSE in ADNI MRI, so the embedding space is shaped by disease-severity geometry rather than class identity (Gare et al., 2024). This suggests a broader interpretation in which disease-level supervision need not be exact diagnostic labeling if clinically meaningful surrogate variables organize pathology-relevant similarity.
3. Core mechanisms for disease-aware contrastive learning
A central mechanism is to embed disease semantics in the same shared space used for image–text alignment. In 3D CT disease-aware pretraining, the image encoder is initialized from SPECTRE and uses a frozen ViT-Large local backbone plus a trainable 4-layer global feature combiner, while the text encoder is Qwen3-Embedding (0.6B) with LoRA adapters. Both sides are projected with SigLIP projection heads into a 512-dimensional shared space (Ging et al., 2 Mar 2026). Disease supervision is injected as text prompts representing positive and negative disease statements rather than as a separate classifier head.
For each binary finding , the model builds three positive and three negative prompt variants. With the L2-normalized global image embedding, the positive prompt embedding, and the negative prompt embedding, the disease logit is defined as
The loss is a weighted BCE over valid findings,
$\mathcal{L}_{\text{prompt} = \frac{1}{|\mathcal{M}|}\sum_{q\in\mathcal{M} w_q\Big( -\alpha_q\, y_{q}\,\log\sigma(x_{q}) -(1- y_{q})\log\!\bigl(1-\sigma(x_{q})\bigr) \Big)$
with , and the total objective is
$\mathcal{L}_{\text{total} = \mathcal{L}_{\text{global} + \lambda \, \mathcal{L}_{\text{prompt} + \beta \, \mathcal{L}_{\text{loc}$
with and (Ging et al., 2 Mar 2026). Conceptually, this is a multi-label, per-disease logistic supervision in the shared contrastive embedding space, rather than a classic batchwise CLIP loss over disease prompts.
Other mechanisms operate at different granularities. RECLF first constructs local image–text matchings by cross-attention from report words to image patches, then applies a semantic-relation reasoning module (SRM) and an importance-relation reasoning module (IRM) so that disease semantics emerge from relations among local matchings rather than from independent region–word pairs (Li et al., 2024). MGLL replaces one-hot CLIP targets with soft targets over multiple disease texts and granularities, and supplements this with point-wise BCE and smooth KL consistency across granularities (Li et al., 20 Nov 2025). CT-DiagVLM goes further toward explicit disease-centric modeling in 3D CT by extracting organ embeddings on the visual side and disease-conditioned report embeddings on the text side using learnable condition queries, with a disease-level contrastive loss defined over organ-condition pools (Shi et al., 24 Jun 2026).
Lesion-centered methods alter the contrastive unit itself. Lesion-based diabetic retinopathy pretraining contrasts 0 lesion patches rather than whole fundus images (Huang et al., 2021). SWCL creates patch-level abnormality labels from CAMs and combines them with patch position and patient identity in a multi-label contrastive objective (Yap et al., 2021). FocusContrast modifies positive-pair generation by rejecting crops or cutouts that remove radiologist-attended regions, thereby making positive views more pathology-preserving (Wang et al., 2023). These methods do not all use language, but they instantiate the same idea that contrastive positives and negatives should respect disease-bearing structure.
4. Representative paradigms across imaging settings
Several distinct paradigms now coexist.
| Paradigm | Representative method | Disease signal source |
|---|---|---|
| Shared-space disease prompts | RadFinder (Ging et al., 2 Mar 2026) | RATE-derived report findings with positive/negative prompts |
| Supervised disease-pair contrast | FFCL (Ahamed et al., 2023) | Same-class vs different-class labels |
| Relation-aware report supervision | RECLF (Li et al., 2024) | Report words plus CheXpert-derived semantic targets |
| Multi-granular disease language | MGLL (Li et al., 20 Nov 2025) | Disease labels, explanations, and multi-granular texts |
| Clinical proxy contrast | Clinical/OCT methods (Kokilepersaud et al., 2022, Kokilepersaud et al., 2023) | BCVA, CST, Eye ID |
| Severity-guided contrast | LEARNER (Gare et al., 2024) | Scalar health scores such as S/F or MMSE |
| Lesion-centered contrast | DR and retinal methods (Huang et al., 2021, Yap et al., 2021) | Lesion patches or pseudo-localized abnormality labels |
In 3D CT, RadFinder is the clearest example of disease-level prompt supervision inside a VLM training loop. It retains large-scale report–volume pretraining but adds explicit disease supervision in the same shared space (Ging et al., 2 Mar 2026). CT-DiagVLM is closely related in spirit but uses organ-level visual embeddings and disease-conditioned report embeddings extracted by learnable condition queries, with positives pooled across patients sharing a disease within an organ (Shi et al., 24 Jun 2026).
In chest radiography, FFCL is disease-aware in a more direct supervised sense. It performs local contrastive learning at each block, then global contrastive learning, using class labels to define positive and negative pairs, so the learned representation is explicitly disease-discriminative for pneumonia detection (Ahamed et al., 2023). RECLF instead remains in the medical vision-language regime, modeling relations among local token–patch matchings to better encode disease and context jointly (Li et al., 2024).
In ophthalmic imaging, clinically guided OCT methods are notable because they operationalize disease relevance without explicit disease-class pretraining labels. The learned representation is organized by clinical-state or severity proxies such as CST or BCVA, then transferred to biomarker classification (Kokilepersaud et al., 2022, Kokilepersaud et al., 2023). SWCL and lesion-based DR contrastive learning make pathology locality central by pretraining on weakly supervised or detector-mined lesion patches (Yap et al., 2021, Huang et al., 2021). PoCo adds a different form of disease-aware inductive bias by using polar transformation to better capture retina-centered structure and rotation-related textures relevant for ophthalmic disease diagnosis (Wang et al., 2024).
A plausible implication is that disease-level contrastive learning is most effective when the supervision matches the clinical granularity of the downstream task: global disease prompts for scan-level diagnosis, lesion-centered units for lesion-driven grading, and clinical-state proxies when biomarker or severity labels are scarce.
5. Empirical evidence and what it shows
The clearest quantitative demonstration that disease supervision changes the embedding space comes from RadFinder’s ablations. With global loss only, the model obtains CT-RATE R@10 = 29.4, CT-RATE AUC = 56.9, and Rad-ChestCT AUC = 62.8. With prompt loss only, it obtains CT-RATE R@10 = 5.6, CT-RATE AUC = 84.8, and Rad-ChestCT AUC = 78.6. The full combined model reaches CT-RATE R@10 = 31.5, CT-RATE AUC = 83.8, and Rad-ChestCT AUC = 77.0 (Ging et al., 2 Mar 2026). This directly shows that global report–volume contrast gives strong retrieval but poor disease discrimination, prompt supervision gives strong disease classification but poor retrieval, and the combined objective yields the best tradeoff.
That paper also shows transfer from internally derived disease supervision: training on RefCT alone gives CT-RATE AUC = 80.4 and Rad-ChestCT AUC = 77.7 (Ging et al., 2 Mar 2026). This is especially relevant because the supervision is report-derived and not manually labeled for disease classification. In CT-DiagVLM, adding disease-level contrastive learning on top of a hybrid encoder and diagnosis-aware prompts improves CT-RATE from 82.4 AUC to 84.4 AUC, and the full model reports 75.4 AUC on Rad-ChestCT with a +5.4% AUC improvement, with even larger gains on a 60-disease benchmark (Shi et al., 24 Jun 2026).
Outside CT, supervised disease-pair contrast also yields measurable gains. FFCL with ResNet-18 and Local 1 Global + random reaches Accuracy 80.45, F1 75.69, Precision 87.70, Recall 74.02, and AUC 93.33, improving accuracy by 3.69 percentage points over RBP + ImageNet on pediatric pneumonia classification (Ahamed et al., 2023). RECLF reports AUROC 0.88, Accuracy 0.67, Precision 0.67, and F1 0.67 for zero-shot classification on MIMIC-5x200, improving over MGCA-ViT’s AUROC 0.84, Accuracy 0.60, Precision 0.62, and F1 0.59 (Li et al., 2024). MGLL shows especially large gains on multi-label retinal disease tasks such as RFMiD, where full fine-tuning reaches 92.83 AUC / 64.99 mAP versus CLIP’s 65.10 AUC / 17.31 mAP (Li et al., 20 Nov 2025).
Lesion-centered contrastive methods also support the same conclusion from a different angle. Lesion-based DR pretraining reaches 66.88 quadratic weighted kappa in linear evaluation on full EyePACS with threshold 0.9, compared with 55.32 for the stronger whole-image SimCLR baseline, and delivers 68.37 kappa at 1% labeled transfer versus 53.36 for supervised ImageNet initialization (Huang et al., 2021). SWCL surpasses prior self-supervised retinal methods and standard cross-entropy training across several classification and segmentation benchmarks while narrowing the gap to ImageNet pretraining (Yap et al., 2021). These results suggest that disease-aware contrastive structure improves not only zero-shot or prompt-based classification, but also the transfer quality of the visual encoder itself.
6. Limitations, caveats, and unresolved questions
A recurrent limitation is dependence on noisy or indirect supervision. RadFinder’s disease supervision depends on the RATE LLM-based label extraction, and the paper does not quantify disease-label extraction accuracy there (Ging et al., 2 Mar 2026). CT-DiagVLM’s extended disease taxonomy relies on LLM-generated pseudo-labels and external report parsing (Shi et al., 24 Jun 2026). RECLF depends on CheXpert-derived soft semantic targets (Li et al., 2024). Clinical proxy methods assume that variables such as BCVA, CST, or MMSE are sufficiently aligned with latent pathology, which is plausible in the reported settings but not guaranteed in general (Kokilepersaud et al., 2022, Kokilepersaud et al., 2023, Gare et al., 2024).
Another limitation is granularity mismatch. Some methods are disease-aware only at the scan level, not at the lesion or region level. RadFinder’s prompt objective acts on the global scan embedding and is explicitly “not region-level disease alignment” (Ging et al., 2 Mar 2026). CT-DiagVLM aligns disease-conditioned text to organ-level visual embeddings rather than to disease-localized 3D tokens (Shi et al., 24 Jun 2026). This suggests that improved pathology discrimination does not automatically imply explicit localization or disentanglement of coexisting findings.
A further caveat is that “disease-level” can denote materially different regimes. FFCL is supervised disease-contrastive pretraining, not self-supervised disease discovery (Ahamed et al., 2023). Clinically labeled OCT methods are better described as clinical-state-aware or severity-aware contrastive learning than as direct disease-label contrastive learning (Kokilepersaud et al., 2022, Kokilepersaud et al., 2023). Metadata-enhanced retinal OCT pretraining is disease-aware through patient/eye/time structure rather than through disease labels themselves (Holland et al., 2022). An encyclopedia treatment therefore has to distinguish direct disease-semantic supervision from proxy-supervised and metadata-guided variants.
Finally, several papers leave reproducibility gaps. LEARNER does not fully specify the weighted contrastive loss, exact pair fusion, or some ADNI details (Gare et al., 2024). VisTA does not fully specify optimization or the exact classification loss, and its embedding is organized more around abnormality types than around disease labels (Can et al., 3 Feb 2025). RECLF omits some graph-attention details (Li et al., 2024). This suggests that the field has converged more clearly on the principle of disease-aware metric structuring than on a single standardized implementation template.
Taken together, the literature indicates that disease-level vision contrastive learning is best viewed as a family of medically structured contrastive objectives that replace or augment generic instance-level similarity with disease-conditioned, pathology-aware, severity-aware, or clinically contextual similarity. The strongest current evidence comes from settings where disease supervision is embedded directly into a shared vision–text space and jointly optimized with broader multimodal alignment, rather than relegated to a downstream classifier (Ging et al., 2 Mar 2026).