Mammogram VQA: Advances in Imaging AI

• Mammogram VQA is an interdisciplinary approach combining automated breast imaging analysis with natural language processing to address clinical queries in cancer screening.
• Advanced models, from zero-shot GPT variants to domain-specific fine-tuned architectures, leverage diverse mammogram datasets to improve accuracy in malignancy detection and BI-RADS classification.
• Techniques such as contrastive self-supervised pretraining, multi-view processing, and saliency analysis are employed to optimize VQA performance and guide future clinical integration.

Mammogram Visual Question Answering (VQA) is an interdisciplinary task that merges the automated interpretation of full-field mammographic images with natural language question-answering capabilities. It encompasses closed and open-ended clinical queries pertaining to breast cancer screening, abnormality detection, BI-RADS classification, and malignancy assessment. VQA models integrate vision-language reasoning to support, and potentially automate, components of radiological workflows.

1. Datasets and Annotation Protocols

Four primary public datasets serve as the foundation for mammogram VQA research: EMBED, InBreast, CMMD, and CBIS-DDSM (Li et al., 15 Aug 2025).

EMBED is a large-scale collection with approximately 680,000 2D images sampled from 110,000 patients. Annotations include BI-RADS density (A–D), abnormality categories (mass, calcification, distortion), lesion descriptors, and pathology severity across a benign-to-malignant axis.

InBreast comprises 410 full-field digital mammograms (FFDM) from 115 cases. Lesions are labeled by type (mass, calcification, asymmetry, distortion), with BI-RADS and biopsy-verified pathology annotations.

CMMD (Chinese Mammographic Mass Dataset) includes 1,775 patients. Labels designate benign or malignant status at the breast-level, with accompanying demographic and imaging metadata.

CBIS-DDSM presents 2,620 digitized film screening studies, annotated with normal/benign/malignant classes, ROI masks for masses and calcifications, and BI-RADS scores where available.

All of these datasets have been adapted for VQA by converting panel annotations into multiple-choice or open-ended items. Standard preprocessing includes intensity normalization, view/laterality harmonization, DICOM-to-PNG conversion, ROI extraction, and upsampling to standard resolutions (typically 800-1024 pixels for input to high-capacity transformers).

A balanced sampling strategy is used to equalize the representation of each answer choice, controlling for class imbalance and enabling precise quantitative comparisons.

2. Model Architectures and Adaptation Strategies

Mammogram VQA employs a range of architectures, from proprietary large multimodal LLMs to domain-adapted, parameter-efficient vision-language transformers.

GPT-5 and GPT-4o Families

Out-of-the-box GPT-5 and GPT-4o models, unadapted to radiology, have been evaluated in a strict zero-shot chain-of-thought prompting regime. The system is presented with a structured prompt (medical assistant persona; explicit answer choice formatting), receiving the image and question simultaneously, and returning a stepwise rationale followed by a single answer letter (Li et al., 15 Aug 2025).

No model fine-tuning or prompt optimization is performed. This approach provides a baseline for general-purpose LLM-VLM capabilities on diagnostic tasks.

Domain-Specific Fine-Tuned Models

The Mammo-CLIP variants and specialized convolutional or transformer architectures (e.g., MRSN, GGP, PHYSnet, ResNet18-S896) employ direct parameter optimization on curated mammography datasets, achieving considerably higher accuracy and calibration on both closed- and open-ended tasks. These models leverage supervised pretraining on high-resolution breast imaging data, often using multi-view and longitudinal case context.

Lightweight Vision-LLMs

Work by Shourya et al. (Shourya et al., 17 Jun 2025) demonstrates that lightweight models (~3B parameters) can achieve strong mammogram VQA performance when subjected to a careful two-stage fine-tuning pipeline:

• Stage 1: Projection-head alignment to anatomical radiological labels (SLAKE) without full-model tuning.
• Stage 2: LoRA-parameter efficient adapters for further fine-tuning on a composite corpus including ROCO v2.0, MedPix v2.0, PMC-VQA, and mammogram-specific Q-A derived from DDSM/InBreast captions.

Contrastive self-supervised pretraining and instruction annealing are used to maximize transfer from generic medical visual reasoning to breast imaging–specific queries.

Self-Supervised Multimodal Pretraining (M2I2)

M2I2 (Li et al., 2022) integrates four self-supervised tasks: masked image modeling (MIM), masked language modeling (MLM), image-text matching (ITM), and image-text contrastive alignment (ITC). This paradigm uses a ViT-B/16 backbone for images and BERT-based encoders for text and multimodal fusion. Pretraining leverages medical image–caption corpora, including mammograms, providing robust representations for subsequent mammogram VQA fine-tuning with limited annotation.

3. Training, Prompting, and Evaluation Protocols

Models are typically evaluated in either zero-shot (no parameter updates on downstream data) or fine-tuned (domain-adapted) modes.

Prompt structure for zero-shot multimodal LLMs (e.g., GPT-5) generally follows a multi-turn exchange:

• System role: set as a “helpful medical assistant.”
• User role (Turn 1): presents the query with explicit answer choices and the image.
• Assistant role: provides rationale (“Let’s think step by step.”).
• User (Turn 2): asks for the answer to be chosen from the alternatives.
• Assistant: outputs the answer letter.

Metrics include closed-ended accuracy, open-ended BLEU/ROUGE scores, and, for malignancy classification, sensitivity, specificity, and overall accuracy:

$\mathrm{ACC} = \frac{TP + TN}{TP + TN + FP + FN}$

$\mathrm{Sensitivity} = \frac{TP}{TP + FN}$

$\mathrm{Specificity} = \frac{TN}{TN + FP}$

Cross-entropy loss is standard for both open and closed-format outputs, optionally supplemented with parameter regularization (e.g., LoRA $L_2$ penalties).

Human-in-the-loop evaluation and attention-based saliency analyses (raw attention, attention rollout, Grad-CAM overlays) are employed to interpret model focus and expose ill-conditioned failure modes (e.g., attention on irrelevant tissue).

4. Quantitative Results and Comparative Performance

A summary of model performance on benchmark datasets is presented below, focusing on GPT-5 variants and strong domain-adapted baselines (Li et al., 15 Aug 2025):

Dataset	Task	GPT-5 (%)	Best Fine-Tuned Model (%)
EMBED	Malignancy	52.8	Mammo-CLIP ViT-L/14: 82.3
InBreast	BI-RADS Classification	36.9	MRSN: 90.6
InBreast	Malignancy	35.0	GGP: 88.5
CMMD	Malignancy	55.0	HybMNet: 79.7
CBIS-DDSM	Malignancy	58.2	PHYSnet: 82.0

In closed-ended mammogram QA, lightweight domain-adapted models reach ~72% accuracy, approaching larger models such as LLaVA-Med (~86% on SLAKE closed, 56% ROCO open) (Shourya et al., 17 Jun 2025).

On the CBIS-DDSM malignancy task, GPT-5 lags behind human readers by 23.4 percentage points in sensitivity and 36.6 percentage points in specificity.

5. Qualitative Insights and Failure Mode Analysis

GPT-5 in zero-shot chain-of-thought:

• Correctly recognizes canonical findings (e.g., spiculated masses, dense tissue) and generates plausible medical rationales.
• Tends to confound adjacent BI-RADS density categories and is susceptible to false positives on architectural distortions without supporting multi-view evidence.
• Frequently misclassifies borderline density (e.g., D→C) and overcalls benign findings as malignant in the absence of corroborative features.

Attention-based saliency and Grad-CAM overlays confirm that failure modes often correspond to model attention drifting to irrelevant regions, especially when handling subtle findings or ambiguous view compositions (Shourya et al., 17 Jun 2025). These insights guide targeted data augmentation and prompt engineering strategies.

6. Limitations, Domain Adaptation, and Open Challenges

Zero-shot LLM-VLMs consistently underperform both domain-specific fine-tuned models and human experts (Li et al., 15 Aug 2025). Major limitations include:

• Absence of fine-tuning on high-resolution grayscale breast images.
• Lack of structured clinical context (age, family history, prior studies).
• Single-view analysis, ignoring critical multi-view or longitudinal trends.
• Balanced, rather than clinically distributed, test sets that may not reflect operational prevalence.

For small expert-annotated mammogram QA datasets (1K–5K pairs), instruction annealing (fine-tuning first on general radiology, then on mammogram Q-A) is recommended to mitigate overfitting (Shourya et al., 17 Jun 2025).

A plausible implication is that robust transfer to clinical mammogram VQA demands multi-view processing, structured patient metadata integration, and uncertainty quantification to flag low-confidence outputs for human review.

7. Recommendations and Future Directions

The following strategies are prioritized to bridge the performance gap and move toward clinical viability:

• Fine-tuning GPT-5 and other LLM-VLMs on large, curated mammography corpora, such as the full EMBED dataset.
• Prompt engineering to incorporate clinical vignettes, few-shot exemplars, and multi-view chain-of-thought sequences.
• Integrating structured metadata (age, risk factors), rare findings (asymmetries, subtle distortions), and harmonized multi-view images at the input stage.
• Employing parameter-efficient fine-tuning (e.g., LoRA) to enable rapid adaptation, even on limited compute.
• Systematic use of saliency-based diagnostics for both development-phase debugging and post-deployment monitoring.
• Human-in-the-loop implementation for triage, particularly in cases of high output variance or low answer confidence.

Prospective clinical validation is required to quantify the effect on recall and cancer detection rates and to establish trust through model transparency (chain-of-thought rationales, uncertainty calibration).

In sum, while general LLMs such as GPT-5 have demonstrated significant zero-shot improvements over previous generations, substantial adaptation and workflow integration are necessary before deployment in high-stakes mammography screening applications (Li et al., 15 Aug 2025, Shourya et al., 17 Jun 2025, Li et al., 2022).