RegionMed-CLIP: Region-Aware Medical Pretraining

• The paper introduces a region-aware contrastive pretraining framework that integrates ROI-specific processing with global context to improve medical vision-language alignment.
• It employs dual-branch transformer encoders and a multi-stage progressive curriculum to harness both local and holistic features for tasks like zero-shot classification, visual question answering, and image-text retrieval.
• RegionMed-CLIP achieves state-of-the-art performance on the MedRegion-500k dataset, enhancing clinical interpretability and diagnostic sensitivity through precise ROI modeling.

RegionMed-CLIP is a region-aware multimodal contrastive pre-training framework specifically designed for medical image understanding. It addresses two principal impediments in automated medical vision-language analysis: the scarcity of comprehensive regional annotations in large-scale datasets and the historical dominance of global image representations that overlook subtle but clinically significant pathological findings. RegionMed-CLIP introduces an explicit region-of-interest (ROI) processor to integrate localized features with holistic context, employing a multi-stage progressive curriculum to optimize both global and regional vision-language alignment. It is pre-trained on MedRegion-500k, a large-scale, densely annotated image-text corpus, and establishes new state-of-the-art results on zero-shot classification, visual question answering, and image-text retrieval across a battery of medical tasks (Fang et al., 7 Aug 2025).

1. Architectural Components

RegionMed-CLIP employs dual-branch transformer encoders: a Vision Transformer (ViT-B/16) image encoder and a PubMedBERT text encoder. The image encoder processes both full-field views ($x_{\mathrm{global}}$) and a set of $m$ automatically derived ROI crops $\{x_{\mathrm{roi}}^{(i)}\}_{i=1}^m$. For each input image, the architecture produces a global embedding $$z_{\mathrm{global}} = f_{\mathrm{img}}(x_{\mathrm{global}})$$ and multiple region embeddings $$z_{\mathrm{roi}}^{(i)} = f_{\mathrm{img}}(x_{\mathrm{roi}}^{(i)})$$, all projected to a shared 512-dimensional unit sphere. The PubMedBERT encoder generates joint textual embeddings from summary captions, full clinical reports, region descriptions, and hard negatives ($h_j = f_{\mathrm{txt}}(t_j)$), similarly normalized.

At the heart of RegionMed-CLIP is the ROI processor. For every region embedding $z_{\rm roi}^{(i)}$ and region-text embedding $h_{\rm region}^{(j)}$, the model computes a pairwise cosine similarity matrix $$S_{i,j} = \tilde z_{\rm roi}^{(i)\top} \tilde h_{\rm region}^{(j)}$$, facilitating finer vision-language alignment. The model further utilizes multi-level attention interactions within the transformer, such that local features (ROI tokens) attend to global context and vice versa.

2. Training Objectives and Optimization

RegionMed-CLIP optimizes three contrastive objectives, unified under a single loss:

1. Global Image–Text Contrast: InfoNCE-based loss aligns global image embeddings $z_{\mathrm{global}}$ with report-level captions $m$0 using a learnable temperature $m$1,

$m$2

2. ROI–Text Contrast: Region-aware contrastive loss links each ROI embedding to its paired region caption,

$m$3

3. Hard Negative Mining: Penalizes spurious similarity between ROI features and negative captions (generated by specialist prompting and clinical term perturbation),

$m$4

where $m$5 and $m$6 is the sigmoid.

The total loss is

$m$7

with all terms equally weighted.

A progressive training curriculum stabilizes optimization. It proceeds through four phases: warmup alignment (2 epochs, global only), global processing (8 epochs), ROI refinement (8 epochs, region losses active), and end-to-end fine-tuning (8 epochs, all losses active). Learning rates and batch sizes shift across phases, prioritizing global-to-local semantic alignment (Fang et al., 7 Aug 2025).

3. MedRegion-500k Dataset

MedRegion-500k is constructed to support fine-grained regional pretraining. The dataset comprises 500,000 image–text pairs (400,000 for training, 50,000 validation, 50,000 test), spanning 12 medical imaging categories (including chest, abdominal, and pathology slides) and 30 disease classes. Each case is annotated at multiple text levels: a summary caption, detailed clinical report, approximately three region captions per image (with positionally grounded, structured descriptions), and five hard negative captions per ROI.

ROIs are annotated using a hybrid pipeline that begins with a small set of manually labeled seeds and scales via Med-SAM (a foundation medical segmenter) and Grounding DINO detection, with cross-consistency scoring. Text annotations are rendered radiologist-style by prompting Qwen-2.5VL-72B. The average image includes three region crops; five negative captions are associated with each ROI. Summary texts are succinct (10–15 words), while region captions follow a (adjective|lesion|position) pattern (Fang et al., 7 Aug 2025).

4. Empirical Performance

RegionMed-CLIP establishes state-of-the-art metrics on diverse medical benchmarks:

• Zero-Shot Classification: Across ten medical datasets, RegionMed-CLIP attains an average AUC of 77.09%, outstripping BiomedCLIP by 3.7 percentage points (pc), and excels on datasets such as NLM-TB (92.92% vs. 88.60%) and PCam200 (92.43% vs. 83.16%).
• Visual Question Answering: On VQA-RAD and SLAKE, RegionMed-CLIP reaches 83.9% accuracy overall, and 90.8% on SLAKE closed questions, outperforming SigLIP-400M.
• Image–Text Retrieval: For text-to-image Recall@1, RegionMed-CLIP achieves 49.7% (vs. 6.4% for SigLIP-400M); the ROI module alone provides a substantial performance boost (R@1 baseline 10.4% rising to 45.8% with ROI processing) (Fang et al., 7 Aug 2025).

Ablation studies reveal that the region-level processor is the dominant factor in downstream improvements: sequences involving only warmup and global training reach 28.3% R@1, while addition of ROI processing lifts this to 45.8%, and subsequent joint fine-tuning yields 49.7%.

5. Clinical Implications and Model Limitations

RegionMed-CLIP enhances clinical interpretability and diagnostic sensitivity by explicitly supporting region-aware feature learning. This is evident in attention map visualizations that demonstrate precise localization of subtle pathological findings, such as small nodules, and reinforce trust in downstream diagnostic support. Its progressive curriculum and unified loss support transferability across image-text retrieval, VQA, and zero-shot tasks.

However, MedRegion-500k, while highly annotated, remains smaller than latest million-scale datasets, posing potential limitations on scaling. The current ROI processor is fixed in ROI count; adaptive or dynamic proposals may improve efficiency. Additional advancement could be realized by integrating human-in-the-loop feedback during downstream fine-tuning, especially for clinical deployment (Fang et al., 7 Aug 2025).

6. Relation to Broader Region-Aware CLIP Approaches

RegionMed-CLIP exemplifies the trend toward explicit regional modeling in medical vision-LLMs. In contrast, MedP-CLIP (Peng et al., 13 Apr 2026) extends CLIP with a feature-level prompt integration mechanism supporting not only automated crops but also anatomical prompts (points, boxes, masks) supplied by human annotators or perception models, maintaining global context via prompt dropout and attention-based fusion. MedP-CLIP operates at larger scale (6.4M images; 97.3M region-text pairs) and further demonstrates robust performance in interactive segmentation and region-level VQA, utilizing a prompt encoder and specialized attention blocks. Both frameworks underscore the necessity of native region/operator integration for fine-grained clinical semantic modeling and outperform global-only CLIP derivatives on a suite of downstream clinical imaging tasks.