Mammo-CLIP: Vision–Language Models for Mammography

Updated 3 July 2026

Mammo-CLIP is a family of vision–language models tailored for mammography, combining multi-view high-resolution images with clinical textual reports to enhance breast cancer screening.
It employs contrastive pre-training with both image–text and multi-view image–image objectives that align visual features with clinical narratives for robust semantic representation.
The framework utilizes parameter-efficient fine-tuning and extensive augmentation techniques to improve interpretability and achieve state-of-the-art performance in computer-aided diagnosis.

Mammo-CLIP is a family of vision–LLMs (VLMs) and associated frameworks for mammography, built around contrastive language-image pre-training (CLIP) paradigms and extended specifically for the complex requirements of breast cancer screening and diagnosis. These models integrate high-resolution, multi-view mammographic images and associated textual findings—whether radiology reports, structured attributes, or atlas captions—to learn data-efficient, robust, and semantically grounded representations. Mammo-CLIP systems have advanced both the state of the art in multi-modal breast cancer computer-aided diagnosis (CAD) and the interpretability, efficiency, and adaptability of medical VLMs.

1. Core Concepts and Architectures

Mammo-CLIP extends the original CLIP framework by addressing the unique aspects of mammography: multi-view protocols, small regions of interest, scarcity of free-text annotations, and high-resolution input requirements. The backbone typically includes:

Vision encoder: Deep CNNs such as EfficientNet-B5 or ConvNeXt, or transformers (e.g., ViT-B/16, ViT-L/14) adapted to high-resolution (up to 1520×912) multi-view input. Multi-view fusion is realized either by image-wise aggregation, local-to-global token mixing (as in (Chen et al., 2024)), or explicit multi-view contrastive objectives (Zheng et al., 30 Oct 2025).
Text encoder: Biomedical transformers pre-trained on radiology or clinical corpora (e.g., BioClinicalBERT, PubMedBERT, BioMedLM). For data efficiency and scalable fine-tuning, plug-and-play adapters (bottleneck modules in BERT or LoRA adapters in LLMs) are commonly inserted, with only adapter parameters being tuned during downstream adaptation (Chen et al., 2024, Du et al., 2024).
Joint embedding space: Matched image–text pairs are projected into a shared, normalized Euclidean space using linear projections and alignment losses, facilitating both classification and cross-modal retrieval.

Notably, early feature fusion—where view-shared local blocks process each mammogram view, followed by global transformer blocks for concatenated tokens—outperforms middle- and late-fusion schemes, especially for subtle findings requiring simultaneous view reasoning (Chen et al., 2024).

2. Contrastive Learning and Multi-View Supervision

Contrastive pre-training is central to Mammo-CLIP. The core CLIP-style InfoNCE loss aligns matched image–text pairs while pushing apart mismatches. In multi-view extensions, the framework further introduces:

Multi-view image–image contrastive objectives: Enforce that anatomically corresponding CC and MLO views from the same laterality have aligned embeddings, aiding view-invariant feature learning (Zheng et al., 30 Oct 2025, Du et al., 2024).
Symmetric image–text objectives: For each study, embeddings for both images and (real or synthetic) reports are jointly aligned in both directions (Du et al., 2024, Ghosh et al., 2024).
Fine-grained patch–sentence alignment: Local contrastive losses and modules such as symmetric local alignment (SLA) align specific image regions with sentences or attributes within the radiology report (Du et al., 2024, Ghosh et al., 2024). This supports spatially precise semantic grounding of findings.

For domains with limited free-text, structured or synthetic reports—generated from metadata using LLM prompting—substitute for actual radiology narratives (Zheng et al., 30 Oct 2025). When only atlas captions are available, as in MAM-CLIP, contrastive learning with a small number of image-caption pairs can still substantially improve downstream performance (Gulluk et al., 19 May 2026).

3. Training Protocols, Parameter Efficiency, and Data Regimes

Training Mammo-CLIP models involves substantial adaptations to maximize efficiency and transferability:

Adapters and PEFT: Parameter-efficient fine-tuning updates only the adapters inserted in each encoder layer, comprising as little as ≈1%–3% of total model parameters, thus enabling robust learning from limited data (e.g., 4.1M/363.8M in ViT-L/14, (Chen et al., 2024); LoRA adapters in LLMs, (Du et al., 2024)).
Data regimes: Studies consistently use internal datasets (hundreds–thousands of cases for fine-tuning) and hold-out/external datasets for zero-shot and generalization assessment (Chen et al., 2024, Ghosh et al., 2024, Zheng et al., 30 Oct 2025).
Augmentation: Extensive image augmentations (affine, elastic, crop, normalization) are standard. Textual augmentations (sentence order swaps, back-translation) enrich the diversity of textual supervision (Ghosh et al., 2024).

Synthetic supervision—using LLM-generated pseudo-reports from exam metadata—enables models to leverage millions of unlabeled images, mitigating the scarcity of annotated mammogram–report pairs (Zheng et al., 30 Oct 2025).

4. Benchmark Performance and Robustness

Mammo-CLIP models have established new baselines in several evaluation settings:

Setting	Model / Dataset	Metric	Performance
Internal few-shot	Mammo-CLIP (ViT-B/16), Dataset 1 (Chen et al., 2024)	AUC	0.841 ± 0.017
Internal few-shot	Cross-view transformer (Chen et al., 2024)	AUC	0.817 ± 0.012
External zero-shot	Mammo-CLIP (ViT-L/14), Dataset 2 (Chen et al., 2024)	AUC	0.837 ± 0.034
Zero-shot bACC	MaMA, EMBED (Du et al., 2024)	bACC	31.04%
Zero-shot bACC	Best baseline (MM-MIL), EMBED (Du et al., 2024)	bACC	26.79%
Macro-F1 (1k data)	MAM-CLIP, TEKNOFEST (Gulluk et al., 19 May 2026)	3-class macro-F1	0.60
Macro-F1 (1k data)	ImageNet baseline, TEKNOFEST (Gulluk et al., 19 May 2026)	3-class macro-F1	0.46
Linear probe 1%	MV-MLM vs. CLIP/Supervised (Zheng et al., 30 Oct 2025)	Mass AUC, VinDr	0.7536 vs. 0.6802/0.6319

Mammo-CLIP models demonstrate superior data efficiency (outperforming supervised baselines with only a fraction of labeled data), high zero-shot transfer, improved out-of-domain robustness, and finer localization ability compared to conventional CNNs and previous CLIP adaptations.

5. Interpretability and Concept Analysis

Interpretability is substantially advanced by Mammo-CLIP Dissect (Salahuddin et al., 25 Sep 2025), which introduces the first concept-based explainability framework for VLMs in mammography:

Neuron–concept alignment: Each neuron in the visual encoder is mapped to a textual concept (e.g., “spiculated mass”, “fatty tissue”) from a mammography-centric taxonomy using a Soft Weighted PMI metric.
Coverage and specialisation metrics: Domain-specific (mammography) pretraining increases the number and ratio of neurons aligned with clinically meaningful concepts. Fine-tuning for specific tasks (e.g., mass or calcification) increases specialization but may reduce coverage for more general or orthogonal concepts (e.g., density).
Gaps and limitations: Under-representation is observed for multi-view, architectural distortion, and temporal concepts—often due to lack of explicit multi-view/sequence supervision or training data scarcity.
Clinical audit: This enables the auditing of a CNN’s knowledge at a concept level, not just pixel-level saliency, and can guide dataset curation and model refinement.

Additionally, spatial attribution heads such as Mammo-FActOR provide sentence-level heatmaps for semantic localization, facilitating human-grounded model explanations (Ghosh et al., 2024).

The Mammo-CLIP framework has inspired or co-evolved with related VLMs in mammography:

MaMA (Du et al., 2024): Incorporates multi-view and local alignment, leveraging LoRA-augmented BioMedLM for fine-tuning and achieving state-of-the-art performance with only 52% model complexity relative to full CLIP baselines.
MV-MLM (Zheng et al., 30 Oct 2025): Trains on massive synthetic report–image datasets, extends to risk prediction and subtype classification, and utilizes multi-view contrastive losses for robust cross-institution generalization.
MAM-CLIP (Gulluk et al., 19 May 2026): Demonstrates that a small number of high-quality image-caption pairs from atlas texts can outperform thousands of labeled images in BI-RADS classification, evidencing the potent “label-multiplier” effect of textual supervision.

Consistently, methods that use multi-view supervision, parameter-efficient adaptation, and sentence-level or local contrastive losses outperform plain CLIP or image-only supersvised strategies across classification, risk prediction, and weakly supervised localization.

7. Limitations and Future Directions

Several limitations and avenues for further research have been identified:

Resolution constraints: Some variants, e.g., (Chen et al., 2024), downsample images to 224×224, potentially losing fine microcalcification detail. Recent work increasingly addresses this with native high-resolution inputs.
Limited report diversity: Many datasets rely on structured or synthetic reports; greater linguistic variability and richer context could further improve cross-domain robustness (Du et al., 2024, Zheng et al., 30 Oct 2025).
Underrepresented findings: Concepts requiring multi-view or temporal reasoning, such as subtle architectural distortion or change over time, remain a challenge for all current networks (Salahuddin et al., 25 Sep 2025).
Future work: Priorities include integrating vision transformers for better long-range modeling (Chen et al., 2024, Ghosh et al., 2024), multi-institution/longitudinal extensions, improving fairness, and modularity for real-time clinical deployment (Du et al., 2024, Gulluk et al., 19 May 2026).

References

(Chen et al., 2024) "Mammo-CLIP: Leveraging Contrastive Language-Image Pre-training (CLIP) for Enhanced Breast Cancer Diagnosis with Multi-view Mammography"
(Ghosh et al., 2024) "Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammography"
(Salahuddin et al., 25 Sep 2025) "Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-LLMs"
(Du et al., 2024) "Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammography"
(Zheng et al., 30 Oct 2025) "MV-MLM: Bridging Multi-View Mammography and Language for Breast Cancer Diagnosis and Risk Prediction"
(Gulluk et al., 19 May 2026) "MAM-CLIP: Vision-Language Pretraining on Mammography Atlases for BI-RADS Classification"