Med-GLIP-5M: Medical Grounding Dataset

• Med-GLIP-5M is a comprehensive medical grounding dataset providing 5.3M region–phrase pairs across 7 imaging modalities.
• It features multi-granularity annotations from organ-level to fine lesions, enabling precise spatial and semantic alignment.
• The dataset underpins a modality-aware framework that boosts diagnostic tasks such as visual question answering and report generation.

Searching arXiv for Med-GLIP and related grounding papers to support the article. Med-GLIP-5M is a large-scale, modality-diverse medical grounding dataset introduced to address the scarcity of region–phrase supervision in medical imaging and to enable spatial-semantic alignment for phrase grounding, referring segmentation, medical visual question answering, and medical report generation. It comprises over 5.3 million region-level annotations across seven imaging modalities, with hierarchical region labels spanning organ-level boundaries, sub-structures, and fine-grained lesions or pathologies. The dataset underpins Med-GLIP, a modality-aware grounding framework that learns region–token alignment without explicitly designed expert modules, and the paper describes the resource as, to its knowledge, the largest medical grounding dataset to date (Deng et al., 14 Aug 2025).

1. Motivation and problem setting

Medical image grounding aims to align natural-language phrases with specific regions in medical images, and the paper positions this as a foundational task for intelligent diagnosis, visual question answering, and automated report generation (Deng et al., 14 Aug 2025). The central motivation is that existing medical vision-language datasets either emphasize classification or question answering without spatial grounding, or they cover only one or a few modalities with coarse labels. Med-GLIP-5M is presented as a response to that gap by providing dense, region-level supervision with natural-language phrases across modalities and across anatomical and pathological granularity.

The scope of the dataset is explicitly multi-modality and multi-granularity. It spans Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Ultrasound, X-ray, Dermoscopy, Endoscopy, and Fundus Photography. The summary description emphasizes more than 30 anatomical regions, while the normalized taxonomy reorganizes 198 fine-grained labels into 38 broader anatomical categories across 6 major body regions. This structure is intended to support both coarse localization and fine-grained spatial semantics, such as distinguishing an organ from a lesion within that organ.

A common misunderstanding would be to read Med-GLIP primarily as a segmentation benchmark. The paper instead defines the core contribution around grounding: masks are the primary annotation source, bounding boxes are computed from masks, and natural-language phrases are attached to each region so that the same corpus supports segmentation and region–phrase alignment. This suggests that the dataset is designed less as a single-task benchmark than as a shared substrate for grounded medical vision-language pre-training.

2. Dataset composition, scale, and modality coverage

Med-GLIP-5M contains over 5.3M region-level image–text pairs and 2,720,979 curated images across modalities, with 4.13 masks per image on average, reflecting an explicit preference for multi-instance supervision (Deng et al., 14 Aug 2025). The curation draws from 129 public medical datasets and hospital annotations, including sources from GitHub, Grand Challenge, OpenNeuro, BraTS, SYNAPSE, CodaLab, TCIA, Kaggle, and others. Example source datasets named in the paper include SegRap2023 for CT, AMOS for MRI, CETUS for Ultrasound, ClinicDB for Endoscopy, ISIC for Dermoscopy, and REFUGE/DRISHTI for Fundus.

The modality distribution is highly uneven, with CT and MRI dominating the corpus. The paper also reports organ or system concentration, with abdomen, bone, head, and chest accounting for substantial fractions of the annotations, and notes that brain, spine, and abdomen have the highest annotation volumes.

Modality	Share of images
CT	61.84%
MRI	26.71%
Ultrasound	1.44%
X-ray	0.22%
Dermoscopy	0.21%
Endoscopy	0.09%
Fundus	0.09%

Organ/system example	Share
abdomen	32.8%
bone	23.9%
head	17.3%
chest	7.6%

Each region is associated with a standardized label name and a natural-language phrase such as “left lung,” “hepatic tumor,” “optic disc,” or “polyp.” Instance-level segmentation masks are the primary source, and bounding boxes are computed from masks. The unified representation is COCO-style and stores masks, boxes, label texts, modality tags, and integrity checks. Because the dataset favors images with multiple spatial annotations per image, it is explicitly configured for multi-label and multi-instance grounding rather than single-object localization.

3. Annotation pipeline, normalization, and hierarchical labels

The preprocessing pipeline converts DICOM and NIfTI images using dcm2niix v1.0 with -check-mosaic, slices 3D volumes to 2D along axes to expand training data, standardizes formats to PNG while preserving resolution, harmonizes metadata, matches masks to images with format-specific scripts, and computes bounding boxes from masks (Deng et al., 14 Aug 2025). Semantic label names are standardized by mapping many synonyms into a unified taxonomy.

Quality control is organized as a three-tier process. It removes unreadable or broken files, enforces mask–image integrity, filters malformed labels or ambiguous semantics, excludes negligible masks defined empirically as less than 1.5% area following PASCAL VOC’s minimum IoU guidance, removes duplicates, and prefers multi-mask images. The ontology mapping is described as LLM-assisted and verified by human experts. The paper is explicit that it does not claim direct mapping to RadLex or SNOMED CT; the objective is internal consistency and anatomical coherence rather than alignment to an external clinical ontology.

The hierarchical label structure is central. A total of 198 fine-grained labels are normalized and reorganized into 38 broader anatomical categories across 6 major body regions. The hierarchy is defined over organ or system, sub-structure, and lesion or pathology. The example given is “lung” as organ, “upper lobe” as sub-structure, and “pneumonia consolidation” as lesion. The paper states that figure-level visualizations show multi-level boxes for hierarchical semantics.

This hierarchical organization should not be confused with an explicit hierarchy-constrained loss. The paper notes that it does not introduce an explicit hierarchical consistency loss beyond the multi-granularity supervision; hierarchical grounding emerges implicitly through diverse labels and prompts. A plausible implication is that the hierarchy is encoded primarily in the data distribution and prompt vocabulary rather than in a specialized structural regularizer.

4. Med-GLIP framework and grounding objective

Med-GLIP is described as a modality-aware hierarchical expert framework in which each modality has a dedicated image encoder $\mathrm{Enc}_{I,\mathrm{modality}}$, while a shared language encoder $\mathrm{Enc}_L$ such as BERT encodes prompts (Deng et al., 14 Aug 2025). Cross-modal fusion is performed by computing region–token alignment via inner products followed by sigmoid normalization. The architecture retains a detector-style localization head for box regression but replaces the usual classification logits with region–token grounding scores.

Prompt construction is explicit. The paper provides the example:

$$\text{Prompt} = \text{“Detect: pneumonia, nodule, fracture”}.$$

The core alignment equations are reported as:

$$\mathbf{F} = \mathrm{Enc}_I(\mathrm{Img}), \quad \mathbf{T} = \mathrm{Enc}_L(\mathrm{Prompt}), \quad S_{\mathrm{ground}} = \sigma(\mathbf{F}\mathbf{T}^{\top}),$$

where $\mathbf{F}\in\mathbb{R}^{N\times d}$ are region features, $\mathbf{T}\in\mathbb{R}^{M\times d}$ are token features, $\sigma(\cdot)$ is sigmoid, and $N$ and $M$ denote the numbers of regions and tokens, respectively.

The detector objective is adapted as

$$L_{\mathrm{cal}} = L_{\mathrm{cal}}^{\mathrm{cls}} + L_{\mathrm{cal}}^{\mathrm{loc}}.$$

In the baseline detector,

$\mathrm{Enc}_L$0

and $\mathrm{Enc}_L$1 is a box regression loss. In Med-GLIP, $\mathrm{Enc}_L$2 is replaced by $\mathrm{Enc}_L$3, and $\mathrm{Enc}_L$4 are phrase targets. For grounding classification, the paper gives

$\mathrm{Enc}_L$5

where $\mathrm{Enc}_L$6 is a token-expanded target matrix.

Token expansion is an implementation detail with conceptual importance. The phrase target $\mathrm{Enc}_L$7 with $\mathrm{Enc}_L$8 concepts is expanded to $\mathrm{Enc}_L$9 with $$\text{Prompt} = \text{“Detect: pneumonia, nodule, fracture”}.$$0 tokens, marking all subword tokens corresponding to a positive phrase as 1 and punctuation or padding as 0. At inference time, constituent token probabilities are averaged to yield a phrase score. This formulation establishes what the paper calls a modality-specific equivalence between detection and phrase grounding.

Training is performed by fine-tuning GLIP and Co-DETR baselines on modality-specific subsets from Med-GLIP-5M. The practical guidance section specifies 30 epochs, Adam, learning rate $$\text{Prompt} = \text{“Detect: pneumonia, nodule, fracture”}.$$1, batch size 5, and hardware of 8× NVIDIA RTX 3090 (24 GB) for grounding. Prompts are modality-specific and clinically coherent; the X-ray example vocabulary is “lung, heart, rib, clavicle, pneumothorax, effusion, nodule, consolidation.”

5. Tasks, evaluation protocol, and empirical results

The dataset supports phrase grounding and referring segmentation. For grounding, the reported metrics are Average Precision (AP) and AP@50, where AP is the area under the precision–recall curve and AP@50 requires $$\text{Prompt} = \text{“Detect: pneumonia, nodule, fracture”}.$$2 (Deng et al., 14 Aug 2025). For segmentation, the paper notes Dice and IoU as typical supported metrics:

$$\text{Prompt} = \text{“Detect: pneumonia, nodule, fracture”}.$$3

where $$\text{Prompt} = \text{“Detect: pneumonia, nodule, fracture”}.$$4 and $$\text{Prompt} = \text{“Detect: pneumonia, nodule, fracture”}.$$5 are predicted and ground-truth masks, although segmentation is not the focus of the reported experiments.

For grounding benchmarks, the evaluation uses six datasets across five modalities, while the description also names Dermoscopy/Rimonedl and Fundus/isic2017. Standardized train/test splits from IMed-361M are used for most datasets with a 90%/10% division, and CVC-ClinicDB uses all images because there is no official split. The paper states that zero-shot GLIP and CO-DETR baselines fail, with AP approximately equal to 0.

Benchmark	Med-GLIP 10% AP/AP@50	Med-GLIP 100% AP/AP@50	CO-DETR 100% AP/AP@50
CT (SegRap)	41.2/80.7	82.7/99.0	31.8/38.9
MRI (AMOS2022)	16.0/43.9	28.0/59.2	7.6/11.3
X-ray (sz_cxr)	4.9/38.4	86.9/100.0	69.8/84.2
Ultrasound (CETUS2014)	9.5/23.5	59.2/91.9	1.0/7.0
Endoscopy (ClinicDB)	10.3/33.4	33.4/62.5	60.4/79.7
Dermoscopy (Rimonedl)	36.0/75.9	61.7/90.1	43.0/79.5
Fundus (isic2017)	14.7/71.3	58.8/78.3	71.0/87.3

The reported takeaways are specific. Zero-shot grounding from natural-image models is ineffective in medical contexts. Scaling from 10% to 100% of training data consistently boosts AP and AP@50. Med-GLIP outperforms CO-DETR in CT, MRI, X-ray, Ultrasound, and Dermoscopy, whereas CO-DETR modestly leads in Endoscopy and Fundus. The paper interprets these results as evidence that domain adaptation with modality-aware experts is essential and that dataset scale strongly impacts performance.

6. Downstream use, limitations, and release status

The paper reports downstream integration into Med-VQA and medical report generation. For medical VQA, the datasets are VQA-RAD, SLAKE, and PathVQA, described as containing 50k+ QA pairs and being multilingual (Deng et al., 14 Aug 2025). Med-GLIP’s spatial outputs are used in two ways: to verify spatial alignment for closed-ended multiple-choice reasoning, and to provide grounded priors for open-ended answer generation with LLaVA-Med. Closed-end accuracy improves across all datasets, with gains up to +2.52% on VQA-RAD, and open-ended Rouge-L increases consistently, with a maximum of +1.68 on VQA-RAD.

For medical report generation, the datasets are MIMIC-CXR and IU-Xray, and the models are R2Gen and MLRG. The integration pipeline uses bounding-box-derived visual features from Med-GLIP as spatial priors within multi-view contrastive learning and alignment in MLRG, improving factual consistency measured by CheXpert and RadGraph F1. R2Gen is reported to show higher BLEU and METEOR when enhanced with spatial priors, with BLEU4 and METEOR notably improved on MIMIC-CXR, although CE metrics were limited for R2Gen because of dataset structuring. The paper’s stated conclusion is that explicit spatial grounding improves multimodal reasoning and clinical fidelity in both question answering and report generation.

The limitations are also explicit. Modality imbalance is substantial: CT and MRI account for approximately 88.55% of images, while X-ray, Fundus, and Endoscopy have small shares. The paper notes that performance on Endoscopy and Fundus lags behind CO-DETR in the reported table. The conversion of 3D data to 2D slices may lose volumetric context for some tasks, and small lesions with indistinct boundaries remain challenging. The labels are standardized internally, but no explicit linkage to RadLex or SNOMED CT is claimed. The dataset composition may favor anatomical systems with mature segmentation benchmarks, such as abdomen and spine, which could bias training toward these regions.

Ethical and release considerations are framed conservatively. Public datasets are de-identified, and the quality-control and standardization pipeline is described as designed to adhere to data usage regulations and ethical norms. The dataset is intended for research use in unified COCO-style format, and the release statement is simply that it “will be released soon.” The paper further notes that original dataset licenses and institutional policies remain applicable and that restrictions may apply per source dataset. Clinical deployment is not implied by benchmark performance alone; the stated requirement is further validation, calibration across sites, and robust handling of rare findings.