MedGround: Medical VLM Grounding Framework

• MedGround is a data-centric framework that fills the evidence gap by synthesizing image-query-localization triplets to bind clinical language with precise visual cues.
• It employs an automated multi-stage pipeline—comprising box extraction, mask-guided attribute extraction, and query synthesis—with rigorous quality assurance.
• Fine-tuning medical VLMs on the MedGround-35K dataset significantly boosts grounding performance, improving diagnostic interpretability and safety.

MedGround refers to a data-centric framework developed to rigorously address the evidence gap in medical vision–LLMs (VLMs) by generating and validating high-quality medical vision–language grounding data. Its core function is to synthesize large-scale triplets of images, spatial localizations (via expert masks or bounding boxes), and clinically grounded queries, thereby enabling VLMs to bind natural medical language to explicit, verifiable visual evidence. This approach directly targets the persistent failure mode in medical VLMs: the inability to reliably localize or “ground” clinical statements in image pixels, a limitation with serious implications for interpretability and diagnostic safety. The MedGround methodology centers on automated, verifiable construction of grounding datasets from existing segmentation resources, with a strong emphasis on multi-stage quality control and transferability across modalities, diseases, and settings (Zhang et al., 11 Jan 2026).

1. Motivation and Theoretical Foundations

MedGround was motivated by the observation that medical VLMs, despite being trained on large corpora of image–report pairs, typically lack fine-grained spatial supervision. Conventional report–image supervision imparts only global semantic alignment and is blind to the challenge of specifying where, within an image, the referenced clinical entity actually appears. At the opposite extreme, segmentation datasets provide detailed pixel-wise localization but without corresponding natural language or contextual queries. MedGround identifies and fills this “evidence gap” by algorithmically generating large corpora of image–query–localization triplets, thereby teaching VLMs to connect morphology-bearing and location-sensitive phrases to spatial anchors. This approach is designed to bridge the cognitive–perceptual divide and mitigate the risk of “right-for-the-wrong-reason” errors in clinical AI (Zhang et al., 11 Jan 2026).

2. Automated Data Generation Pipeline

The MedGround pipeline consists of three main, fully automated stages, optionally augmented by human auditing:

1. Box Extraction:
   • Starting from expert-annotated segmentation masks $M$, all connected components are identified, and each mask yields a tight axis-aligned bounding box $b = [x_{\min}, y_{\min}, x_{\max}, y_{\max}]$
   • Boxes are normalized to a fixed image grid (e.g., $1000\times1000$) for resolution-independence.
   • All boxes per image comprise the candidate pool $\mathcal{B}$.
2. Mask-Guided Attribute Extraction:
   • For each box, geometric and spatial features are extracted:
     • Area ratio $$A_i = \operatorname{area}(M_i)/\operatorname{area}(I)$$
     • Width, height, aspect ratio $\alpha_i$, perimeter-based compactness $C_i$
     • Centroid $(c_x, c_y)$ quantized to left/center/right and top/middle/bottom
     • Modalities and coarse category from dataset source.
3. Clinically-Grounded Query Synthesis:
   • Using a strong LLM/VLM, a controlled prompt incorporating regional attributes is used to generate a clinically precise referring query $Q$
     • Queries must describe observable morphology, specify laterality/location for disambiguation, and use correct terminology without over-interpreting unobservable pathology.
   • The system collects the output as JSON $(I, Q, b)$ or $(I, Q, \{b_1,\dots,b_k\})$ for multi-object settings.

3. Multi-Stage Verification and Quality Assurance

To minimize grounding hallucinations and ambiguity, MedGround enforces a strict, multi-stage QA process:

• Stage I: Format and Schema Check:

Reject samples failing the JSON schema or containing invalid/missing values.

• Stage II: Rule-based Geometry and Medical Priors:
  • Enforce alignment between geometric descriptors in $Q$ (e.g., “large,” “medial”) and actual region attributes (e.g., area bucket, centroid bin).
  • Apply medical allow/deny lists to block anatomical/term mismatches across modalities.
• Stage III: VLM-Based Visual Judging:
  • Use a pretrained VLM to semantically verify $(I, b, Q)$. If the model cannot restate key query attributes from the highlighted region, discard the triplet.
• Optional Human Audit:
  • For the test split, three professional annotators independently review each triplet. ≥2/3 agreement yields a 78% visual faithfulness accept rate.

4. MedGround-35K: Dataset Design and Composition

MedGround-35K is the first large-scale, verifiably grounded multimodal dataset for medical VLMs built using the above pipeline:

Statistic	Value	Notes (Train/Test)
Total triplets	35,480	25,420/10,060
Modalities (train)	US 60.2%, Nuclei 20.5%	Dermoscopy 9.9%, CT 8.2%, Bac 1.3%
Sources	8 public segmentation sets	e.g., ISIC2016, MosMedPlus
Query length	avg 12 words/query	2.77M tokens train, 1.12M test
Linguistic richness	High UMLS density	Broad morphology/spatial terms

Queries combine clinical entity names, explicit spatial descriptors, and are anchored to precise visual regions. The dataset is designed for both referring grounding (given $(I,Q)$, predict $b$) and multi-object semantic disambiguation.

5. Impact on Vision-LLM Grounding

Extensive ablations on eight test sets confirm that VLMs fine-tuned on MedGround-35K—using LoRA for 3 epochs—consistently outperform zero-shot general or medical VLMs and those only exposed to category-name supervision:

• Referring Grounding:
  • On ISIC2016, Qwen2.5-VL-7B IoU increases from 9.9 (zero-shot) to 83.0 after MedGround SFT, a +73.1 point boost.
  • Across datasets, fine-tuning yields +7 to +77 IoU gains.
• Multi-Object Disambiguation:
  • Semantic Sensitivity (correct location for two separate queries) on FHPsAOP rises from 21.5%→53.5% (Qwen3-VL-8B); MosMedPlus 5.9%→18.0%.
• Zero-Shot Generalization:
  • Models trained on MedGround transfer robustly to unseen chest X-ray referring grounding tasks (e.g., QaTa-COV19), with substantial IoU gains.

These results demonstrate that grounding logic—morphology-language-spatial alignment—generalizes to new diseases, modalities, and anatomical regions when taught with high-quality, rigorously verified supervision.

6. Scalability, Clinical Relevance, and Future Directions

MedGround is inherently scalable: given expert masks from any dataset, its deterministic, rule-driven conversion and validation pipeline supports new modalities, organ systems, and wider disease spectra with minimal human overhead. As a training resource, MedGround-35K provides a rich signal not only for spatial grounding but also for evidence-based reasoning in medical VLMs, promoting models that must justify clinical claims with explicit pixel-level support.

Possible future directions include:

• Extension to volumetric (3D) and multi-timepoint medical images.
• Integration of complex spatial/relational language for richer compositional queries.
• Expansion to multilingual settings and joint benchmarking with datasets such as PadChest-GR (Castro et al., 2024).
• Use as foundational supervision for generalized medical phrase grounding architectures (e.g., MedGrounder (Zhang et al., 30 Nov 2025), Med-GLIP (Deng et al., 14 Aug 2025)).

A plausible implication is that MedGround-instructed VLMs will play a crucial role in safe clinical AI deployment by providing interpretable, evidence-linked outputs, helping to close the trust gap between vision-language AI models and real-world, visually verifiable clinical expertise.

---

References

• "MedGround: Bridging the Evidence Gap in Medical Vision-LLMs with Verified Grounding Data" (Zhang et al., 11 Jan 2026)
• "PadChest-GR: A Bilingual Chest X-ray Dataset for Grounded Radiology Report Generation" (Castro et al., 2024)
• "Generalized Medical Phrase Grounding" (Zhang et al., 30 Nov 2025)
• "Med-GLIP: Advancing Medical Language-Image Pre-training with Large-scale Grounded Dataset" (Deng et al., 14 Aug 2025)