---
title: U-MRG-14K Medical Vision-Language Dataset
url: https://www.emergentmind.com/topics/u-mrg-14k
type: topic
---

# U-MRG-14K Medical Vision-Language Dataset

U-MRG-14K is a **14K-sample** medical vision-language grounding dataset introduced to operationalize **Unified Medical Reasoning Grounding (UMRG)**, a task in which a model receives a **medical image plus an implicit clinical query**, infers the latent clinical referent, and grounds that referent at **pixel level**. In the formulation given for UMRG, a model maps image-query input to an optional reasoning trace, a bounding box, two semantic key points, and a segmentation mask, making U-MRG-14K structurally distinct from conventional medical segmentation datasets, explicit referring-expression benchmarks, and image-level medical VQA resources [2508.08177].

## 1. Task definition and conceptual scope

U-MRG-14K is defined around the UMRG task, formalized as
\[
\{\mathcal{T},\mathcal{B},\mathcal{P}_1,\mathcal{P}_2,\mathcal{M}\}=\mathbf{G}(\mathcal{I},\mathcal{Q}).
\]
Here, given a medical image \(\mathcal{I}\) and an implicit clinical query \(\mathcal{Q}\), the model \(\mathbf{G}\) outputs an optional reasoning trace \(\mathcal{T}\), a bounding box \(\mathcal{B}\), two semantic key points \(\mathcal{P}_1,\mathcal{P}_2\), and a segmentation mask \(\mathcal{M}\). The defining property is that the query is intentionally under-specified: rather than naming the target explicitly, it uses clinically indirect language such as “What can be inferred from the irregular shadow?”, requiring latent referent inference before localization or segmentation [2508.08177].

This places U-MRG-14K at the intersection of multimodal reasoning and dense medical grounding. Standard referring image segmentation and phrase grounding typically assume that the text already identifies the target object. Ordinary medical segmentation datasets provide masks but omit language annotations. Medical VQA datasets assess semantic interpretation with question-answer pairs but generally lack spatial supervision. U-MRG-14K couples these dimensions by requiring clinically realistic ambiguous language, explicit reasoning traces, and geometrically actionable outputs in one benchmark [2508.08177].

The dataset is therefore best understood not as a segmentation corpus with auxiliary captions, nor as a VQA dataset with optional masks, but as a reasoning-grounding benchmark. This suggests that its central difficulty lies in the alignment between implicit clinical language and pixel-level structure, rather than in mask prediction alone.

## 2. Dataset composition and annotation structure

At dataset level, U-MRG-14K is described as a **14K-sample** resource spanning **10 imaging modalities**, **15 super-categories**, and **108 fine-grained categories**. The paper states that it contains “pixel-level masks alongside implicit clinical queries and reasoning traces,” and the appendix further characterizes it as combining “implicit clinical questions” with “pixel annotations (bounding boxes, interior key points, and masks)” across diverse studies [2508.08177].

The dataset coverage includes modalities such as **CT, MRI, ultrasound, histology**, and others. The super-categories include frequent anatomical regions and pathology-oriented groups, including **abdomen, lung, brain, eye, heart, vessel, neck, skeletal, oral**, as well as **neoplasm, non-neoplasm, infection, fluid abnormality**, plus categories such as **histology structure** and **surgical instrument**. Fine-grained labels include both anatomy and pathology; the paper gives examples such as **left lung** and **right lung** as distinct classes within a lung super-category [2508.08177].

Each sample includes richer supervision than a conventional segmentation benchmark. At minimum, the dataset provides a **medical image**, a **pixel-level segmentation mask**, an **implicit question/query**, and a **chain-of-thought reasoning trace**. The appendix specifies a case format with **question, think, and answer**, where the answer contains “accurate spatial grounding derived from the annotated mask, including a bounding box and two key points.” Additional metadata includes **imaging modality**, **subject health status** (patient or healthy), **super-category**, **category**, and two textual descriptions per image-mask pair: a **short description** emphasizing intuitive visual cues and a **long description** integrating medical or imaging knowledge while remaining image-grounded [2508.08177].

A concise view of the benchmark schema is as follows.

| Aspect | Specification |
|---|---|
| Scale | 14K samples |
| Coverage | 10 modalities, 15 super-categories, 108 fine-grained categories |
| Per-sample outputs | reasoning trace, bounding box, two key points, mask |

This annotation design matters because it supports both intermediate supervision and modular inference. A plausible implication is that U-MRG-14K can evaluate not only end-task segmentation quality but also whether a system produces geometrically useful prompts and interpretable reasoning traces.

## 3. Construction pipeline

U-MRG-14K was built from **three open-source datasets**: **SA-Med2D-20M**, **BiomedParse**, and **IMIS-Bench**. In **Stage 1**, the authors collected **14K image-mask pairs** and manually standardized the taxonomy, stating that they “standardize and complete the super-category labels and category labels … producing a consistent and reliable taxonomy.” The appendix adds that each image was manually annotated with attributes including **modality**, **subject type**, **category**, and **super-category** [2508.08177].

In **Stage 2**, **GPT-4o** was used to create image descriptions and category-specific QA schemas. For each image, two complementary descriptions were generated: a **short visual description** in plain language and a **long medically precise description**. In parallel, for each super-category, GPT-4o generated approximately **20 QA formats** on average, with the exact number manually adjusted “for class diversity.” The QA formats were category-conditioned: pathology categories emphasized abnormal findings, lesion extent, or uncertainty, whereas anatomical categories emphasized structure, spatial relations, and function. The generation schema required GPT-4o to return exactly \(N\) question-answer formats, default \(N=20\), in a JSON object containing the super-category and a list of `{id, question, answer}` entries [2508.08177].

In **Stage 3**, those descriptions and category-level QA formats were combined to synthesize instance-level QA pairs. The appendix describes a prompt framework in which GPT-4o is assigned the role of a professional radiologist, given the image, segmentation mask, metadata, bbox, key points, and descriptions, and then asked to revise templates into vague, indirect clinical questions plus explicit step-by-step answers. A key construction rule is the **“Pretend the Mask is Unavailable” Principle**, intended to prevent trivial template filling and force reasoning that mimics human visual search. The generated questions were required to be vague and indirect, grounded in visual attributes without naming the category; the answers were required to be step-by-step, image-grounded, non-diagnostic, and auditable. All generated QA pairs were then **manually reviewed** to remove duplicates, factual inconsistencies, and misaligned reasoning [2508.08177].

This hybrid pipeline combines synthetic text generation with manual verification. That does not make the benchmark purely synthetic; rather, it indicates that the language layer was programmatically generated on top of public mask data and then curated. This suggests that the benchmark’s implicitness is engineered rather than collected from prospective workflow logs.

## 4. Evaluation protocol and benchmark settings

The main experimental protocol uses a simple split: **2.5K samples** are randomly held out as a test set, and the remaining data are used for training. No separate validation split is explicitly described in the main text. The appendix also defines **U-MRG-6K**, a generalization setting built from **6K images from the 5 most frequent categories**, with remaining images from the other categories used as an out-of-distribution test set [2508.08177].

Evaluation uses three primary metrics: **IoU**, **pDice**, and **Dice**. IoU is used for **bounding box localization**, pDice for **keypoint-pair semantic alignment**, and Dice for the **final mask quality** after the segmentation module consumes predicted prompts. The box overlap metric is defined as
\[
\mathbb{R}_{\mathrm{iou}}=\frac{\mathrm{Area}(\mathcal{B}_p \cap \mathcal{B}_g)}{\mathrm{Area}(\mathcal{B}_p \cup \mathcal{B}_g)}.
\]
For keypoints, pDice is defined by converting a point pair into a circle whose diameter is the segment between the points:
\[
\mathbb{R}_{\mathrm{pdice}}=\frac{2 \cdot \mathrm{Area}(O_p \cap O_g)}{\mathrm{Area}(O_p) + \mathrm{Area}(O_g)}.
\]
The paper does not introduce a separate text-generation metric for reasoning quality itself; reasoning is instead assessed indirectly through grounding quality and through format rewards that enforce valid structured outputs [2508.08177].

A further benchmark variable is **\# Ref.**, the number of refusals, which measures failure to answer in the required grounding format. This is important because implicit clinical prompts can trigger generic or text-only responses in vanilla multimodal language models. The benchmark therefore evaluates not only geometric precision but also whether a model can consistently produce actionable spatial outputs under under-specified clinical language [2508.08177].

From an evaluation-design perspective, this makes U-MRG-14K a whole-pipeline benchmark. A system must infer the referent, produce geometrically useful prompts, and support downstream segmentation; success cannot be reduced to mask prediction in isolation.

## 5. Baselines, MedReasoner, and reported performance

The paper evaluates a broad range of models under a common protocol in which each candidate multimodal model acts as a reasoning module and **MedSAM2** is held fixed as the segmentation backend. The baselines include general multimodal models (**GPT-4o, Gemini-2.5-flash, Qwen2.5-VL-7B/72B, InternVL3-8B/78B**), medical multimodal models (**MedR1-2B, MiniInternVL-4B, MedGamma-4B, HuatuoGPT-7B, Lingshu-7B, Chiron-o1-8B**), and grounding-oriented systems (**VLMR1-REC-3B, SegZero-7B, SAM4MLLM-8B**) [2508.08177].

On the U-MRG-14K test set, **MedReasoner-7B** reports **IoU 32.42**, **pDice 26.55**, and **Dice 37.78**. The strongest non-MedReasoner model in overall IoU is **Qwen2.5-VL-72B** with **18.32 IoU**, **12.39 pDice**, and **29.71 Dice**. Among grounding-specific baselines, **SegZero-7B** reaches **16.14 IoU** and **26.05 Dice**, while **VLMR1-REC-3B** reports **13.96 IoU** and **22.19 Dice**. **GPT-4o** performs poorly in grounding despite fluent reasoning text, with **2.65 IoU**, **1.12 pDice**, and **4.72 Dice** [2508.08177].

The benchmark also supports category-level comparison. MedReasoner leads most reported super-categories, including **Lung 50.75 IoU**, **Eye 51.50**, **Heart 34.72**, **Abdomen 30.27**, **Neoplasm 33.58**, and **Infection 30.48**. **Histology** remains difficult for all methods; MedReasoner reports **11.66 IoU** there, still the best in that column [2508.08177].

The MedReasoner architecture is modular. Its **Clinical Reasoning Module (CRM)**—default **Lingshu-7B**—takes \((\mathcal{I}, \mathcal{Q})\) and emits structured reasoning plus prompts in the form `<think>...</think> <answer>{bbox, points_1, points_2}</answer>`. Its **Anatomical Segmentation Module (ASM)**—default **MedSAM2**—consumes the box and two points and returns the mask. Only the CRM is trained with **Group Relative Policy Optimization (GRPO)**; the ASM remains frozen [2508.08177].

The paper’s ablations indicate that U-MRG-14K strongly differentiates reinforcement learning from supervised fine-tuning for this task. Starting from **Lingshu** alone, performance is **8.19 IoU / 3.73 pDice / 16.51 Dice**, with **2 refusals**. **Lingshu w/ SFT** raises IoU only to **9.15** and worsens pDice and Dice, still with **2 refusals**. **Lingshu w/ RL(Base)** improves to **15.85 / 8.29 / 28.79**. **RL(Hard)** reaches **31.69 / 24.36 / 33.51**, and **RL(Soft)** reaches the full **32.42 / 26.55 / 37.78**, with **0 refusals** [2508.08177].

This suggests that U-MRG-14K is not merely sensitive to model scale; it is particularly sensitive to whether the system can convert implicit clinical language into geometrically valid prompts under a format-constrained objective.

## 6. Generalization, limitations, and disambiguation

The benchmark’s generalization setting, **U-MRG-6K**, is designed to test robustness beyond the standard split. In this setting, the CRM is trained on a biased subset from only 5 frequent categories and evaluated on remaining categories. The out-of-distribution results show severe phrase overfitting for supervised fine-tuning: **SFT without reasoning** reports **0.32 IoU / 0.08 pDice / 0.55 Dice** with **1081 refusals**, while **SFT with reasoning** reaches **6.32 / 1.56 / 13.45**. By contrast, **RL-Soft** reports **17.27 / 12.02 / 25.2** without reasoning and **17.67 / 11.33 / 26.72** with reasoning, both with **0 refusals** [2508.08177].

The paper also implies several limitations. First, the dataset uses **GPT-4o-generated descriptions, QA formats, and reasoning traces**, although these are manually filtered; the reasoning annotations are therefore useful supervision but not guaranteed faithful models of clinician cognition. Second, the appendix notes class imbalance: the four largest super-categories—**abdomen, heart, neoplasm, and non-neoplasm**—account for **59%** of samples. Third, some domains remain substantially harder than others, especially **histology**, indicating that the dataset aggregates heterogeneous visual regimes under one benchmark. Fourth, the benchmark is based on public-source masks and synthetic query generation rather than prospectively collected clinical dialogue, so its implicitness is simulated rather than observed in routine workflow [2508.08177].

U-MRG-14K should also be distinguished from similarly named but unrelated entities. A chest X-ray report-generation paper, **“M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation,”** explicitly does **not** mention U-MRG-14K and instead uses a MIMIC-CXR-derived report-generation setup [2408.16213]. The dataset should likewise not be confused with **ResearchMath-14k**, a research-level mathematics dataset of **14,056** problems whose paper states that it does **not** mention “U-MRG-14K” and provides no alias mapping [2605.28003]. Nor does the U-MRG-14K name appear in **“API Security Based on Automatic OpenAPI Mapping,”** whose **MRG** refers to **Map Reduce Graph**, an unsupervised REST-API modeling method rather than a medical grounding benchmark [2604.19471].

These distinctions matter because the string “MRG” is overloaded across medical imaging, report generation, mathematics, and API security. In current arXiv usage, U-MRG-14K refers specifically to the medical grounding dataset introduced with UMRG and MedReasoner [2508.08177].

Source: https://www.emergentmind.com/topics/u-mrg-14k