U-MRG-14K Medical Vision-Language Dataset

• U-MRG-14K is a 14K-sample benchmark that combines implicit clinical queries with pixel-level annotations for precise medical vision-language grounding.
• It integrates diverse modalities and categories by providing images with reasoning traces, bounding boxes, key points, and segmentation masks.
• The dataset supports evaluation of multimodal models' ability to infer under-specified clinical language into actionable visual outputs.

U-MRG-14K is a 14K-sample medical vision-language grounding dataset introduced to operationalize Unified Medical Reasoning Grounding (UMRG), a task in which a model receives a medical image plus an implicit clinical query, infers the latent clinical referent, and grounds that referent at pixel level. In the formulation given for UMRG, a model maps image-query input to an optional reasoning trace, a bounding box, two semantic key points, and a segmentation mask, making U-MRG-14K structurally distinct from conventional medical segmentation datasets, explicit referring-expression benchmarks, and image-level medical VQA resources (Yan et al., 11 Aug 2025).

1. Task definition and conceptual scope

U-MRG-14K is defined around the UMRG task, formalized as

$$\{\mathcal{T},\mathcal{B},\mathcal{P}_1,\mathcal{P}_2,\mathcal{M}\}=\mathbf{G}(\mathcal{I},\mathcal{Q}).$$

Here, given a medical image $\mathcal{I}$ and an implicit clinical query $\mathcal{Q}$, the model $\mathbf{G}$ outputs an optional reasoning trace $\mathcal{T}$, a bounding box $\mathcal{B}$, two semantic key points $\mathcal{P}_1,\mathcal{P}_2$, and a segmentation mask $\mathcal{M}$. The defining property is that the query is intentionally under-specified: rather than naming the target explicitly, it uses clinically indirect language such as “What can be inferred from the irregular shadow?”, requiring latent referent inference before localization or segmentation (Yan et al., 11 Aug 2025).

This places U-MRG-14K at the intersection of multimodal reasoning and dense medical grounding. Standard referring image segmentation and phrase grounding typically assume that the text already identifies the target object. Ordinary medical segmentation datasets provide masks but omit language annotations. Medical VQA datasets assess semantic interpretation with question-answer pairs but generally lack spatial supervision. U-MRG-14K couples these dimensions by requiring clinically realistic ambiguous language, explicit reasoning traces, and geometrically actionable outputs in one benchmark (Yan et al., 11 Aug 2025).

The dataset is therefore best understood not as a segmentation corpus with auxiliary captions, nor as a VQA dataset with optional masks, but as a reasoning-grounding benchmark. This suggests that its central difficulty lies in the alignment between implicit clinical language and pixel-level structure, rather than in mask prediction alone.

2. Dataset composition and annotation structure

At dataset level, U-MRG-14K is described as a 14K-sample resource spanning 10 imaging modalities, 15 super-categories, and 108 fine-grained categories. The paper states that it contains “pixel-level masks alongside implicit clinical queries and reasoning traces,” and the appendix further characterizes it as combining “implicit clinical questions” with “pixel annotations (bounding boxes, interior key points, and masks)” across diverse studies (Yan et al., 11 Aug 2025).

The dataset coverage includes modalities such as CT, MRI, ultrasound, histology, and others. The super-categories include frequent anatomical regions and pathology-oriented groups, including abdomen, lung, brain, eye, heart, vessel, neck, skeletal, oral, as well as neoplasm, non-neoplasm, infection, fluid abnormality, plus categories such as histology structure and surgical instrument. Fine-grained labels include both anatomy and pathology; the paper gives examples such as left lung and right lung as distinct classes within a lung super-category (Yan et al., 11 Aug 2025).

Each sample includes richer supervision than a conventional segmentation benchmark. At minimum, the dataset provides a medical image, a pixel-level segmentation mask, an implicit question/query, and a chain-of-thought reasoning trace. The appendix specifies a case format with question, think, and answer, where the answer contains “accurate spatial grounding derived from the annotated mask, including a bounding box and two key points.” Additional metadata includes imaging modality, subject health status (patient or healthy), super-category, category, and two textual descriptions per image-mask pair: a short description emphasizing intuitive visual cues and a long description integrating medical or imaging knowledge while remaining image-grounded (Yan et al., 11 Aug 2025).

A concise view of the benchmark schema is as follows.

Aspect	Specification
Scale	14K samples
Coverage	10 modalities, 15 super-categories, 108 fine-grained categories
Per-sample outputs	reasoning trace, bounding box, two key points, mask

This annotation design matters because it supports both intermediate supervision and modular inference. A plausible implication is that U-MRG-14K can evaluate not only end-task segmentation quality but also whether a system produces geometrically useful prompts and interpretable reasoning traces.

3. Construction pipeline

U-MRG-14K was built from three open-source datasets: SA-Med2D-20M, BiomedParse, and IMIS-Bench. In Stage 1, the authors collected 14K image-mask pairs and manually standardized the taxonomy, stating that they “standardize and complete the super-category labels and category labels … producing a consistent and reliable taxonomy.” The appendix adds that each image was manually annotated with attributes including modality, subject type, category, and super-category (Yan et al., 11 Aug 2025).

In Stage 2, GPT-4o was used to create image descriptions and category-specific QA schemas. For each image, two complementary descriptions were generated: a short visual description in plain language and a long medically precise description. In parallel, for each super-category, GPT-4o generated approximately 20 QA formats on average, with the exact number manually adjusted “for class diversity.” The QA formats were category-conditioned: pathology categories emphasized abnormal findings, lesion extent, or uncertainty, whereas anatomical categories emphasized structure, spatial relations, and function. The generation schema required GPT-4o to return exactly $N$ question-answer formats, default $N=20$, in a JSON object containing the super-category and a list of {id, question, answer} entries (Yan et al., 11 Aug 2025).

In Stage 3, those descriptions and category-level QA formats were combined to synthesize instance-level QA pairs. The appendix describes a prompt framework in which GPT-4o is assigned the role of a professional radiologist, given the image, segmentation mask, metadata, bbox, key points, and descriptions, and then asked to revise templates into vague, indirect clinical questions plus explicit step-by-step answers. A key construction rule is the “Pretend the Mask is Unavailable” Principle, intended to prevent trivial template filling and force reasoning that mimics human visual search. The generated questions were required to be vague and indirect, grounded in visual attributes without naming the category; the answers were required to be step-by-step, image-grounded, non-diagnostic, and auditable. All generated QA pairs were then manually reviewed to remove duplicates, factual inconsistencies, and misaligned reasoning (Yan et al., 11 Aug 2025).

This hybrid pipeline combines synthetic text generation with manual verification. That does not make the benchmark purely synthetic; rather, it indicates that the language layer was programmatically generated on top of public mask data and then curated. This suggests that the benchmark’s implicitness is engineered rather than collected from prospective workflow logs.

4. Evaluation protocol and benchmark settings

The main experimental protocol uses a simple split: 2.5K samples are randomly held out as a test set, and the remaining data are used for training. No separate validation split is explicitly described in the main text. The appendix also defines U-MRG-6K, a generalization setting built from 6K images from the 5 most frequent categories, with remaining images from the other categories used as an out-of-distribution test set (Yan et al., 11 Aug 2025).

Evaluation uses three primary metrics: IoU, pDice, and Dice. IoU is used for bounding box localization, pDice for keypoint-pair semantic alignment, and Dice for the final mask quality after the segmentation module consumes predicted prompts. The box overlap metric is defined as

$\mathcal{I}$0

For keypoints, pDice is defined by converting a point pair into a circle whose diameter is the segment between the points: $\mathcal{I}$1 The paper does not introduce a separate text-generation metric for reasoning quality itself; reasoning is instead assessed indirectly through grounding quality and through format rewards that enforce valid structured outputs (Yan et al., 11 Aug 2025).

A further benchmark variable is # Ref., the number of refusals, which measures failure to answer in the required grounding format. This is important because implicit clinical prompts can trigger generic or text-only responses in vanilla multimodal LLMs. The benchmark therefore evaluates not only geometric precision but also whether a model can consistently produce actionable spatial outputs under under-specified clinical language (Yan et al., 11 Aug 2025).

From an evaluation-design perspective, this makes U-MRG-14K a whole-pipeline benchmark. A system must infer the referent, produce geometrically useful prompts, and support downstream segmentation; success cannot be reduced to mask prediction in isolation.

5. Baselines, MedReasoner, and reported performance

The paper evaluates a broad range of models under a common protocol in which each candidate multimodal model acts as a reasoning module and MedSAM2 is held fixed as the segmentation backend. The baselines include general multimodal models (GPT-4o, Gemini-2.5-flash, Qwen2.5-VL-7B/72B, InternVL3-8B/78B), medical multimodal models (MedR1-2B, MiniInternVL-4B, MedGamma-4B, HuatuoGPT-7B, Lingshu-7B, Chiron-o1-8B), and grounding-oriented systems (VLMR1-REC-3B, SegZero-7B, SAM4MLLM-8B) (Yan et al., 11 Aug 2025).

On the U-MRG-14K test set, MedReasoner-7B reports IoU 32.42, pDice 26.55, and Dice 37.78. The strongest non-MedReasoner model in overall IoU is Qwen2.5-VL-72B with 18.32 IoU, 12.39 pDice, and 29.71 Dice. Among grounding-specific baselines, SegZero-7B reaches 16.14 IoU and 26.05 Dice, while VLMR1-REC-3B reports 13.96 IoU and 22.19 Dice. GPT-4o performs poorly in grounding despite fluent reasoning text, with 2.65 IoU, 1.12 pDice, and 4.72 Dice (Yan et al., 11 Aug 2025).

The benchmark also supports category-level comparison. MedReasoner leads most reported super-categories, including Lung 50.75 IoU, Eye 51.50, Heart 34.72, Abdomen 30.27, Neoplasm 33.58, and Infection 30.48. Histology remains difficult for all methods; MedReasoner reports 11.66 IoU there, still the best in that column (Yan et al., 11 Aug 2025).

The MedReasoner architecture is modular. Its Clinical Reasoning Module (CRM)—default Lingshu-7B—takes $\mathcal{I}$2 and emits structured reasoning plus prompts in the form > ... <answer>{bbox, points_1, points_2}</answer>. Its Anatomical Segmentation Module (ASM)—default MedSAM2—consumes the box and two points and returns the mask. Only the CRM is trained with Group Relative Policy Optimization (GRPO); the ASM remains frozen (Yan et al., 11 Aug 2025).

The paper’s ablations indicate that U-MRG-14K strongly differentiates reinforcement learning from supervised fine-tuning for this task. Starting from Lingshu alone, performance is 8.19 IoU / 3.73 pDice / 16.51 Dice, with 2 refusals. Lingshu w/ SFT raises IoU only to 9.15 and worsens pDice and Dice, still with 2 refusals. Lingshu w/ RL(Base) improves to 15.85 / 8.29 / 28.79. RL(Hard) reaches 31.69 / 24.36 / 33.51, and RL(Soft) reaches the full 32.42 / 26.55 / 37.78, with 0 refusals (Yan et al., 11 Aug 2025).

This suggests that U-MRG-14K is not merely sensitive to model scale; it is particularly sensitive to whether the system can convert implicit clinical language into geometrically valid prompts under a format-constrained objective.

6. Generalization, limitations, and disambiguation

The benchmark’s generalization setting, U-MRG-6K, is designed to test robustness beyond the standard split. In this setting, the CRM is trained on a biased subset from only 5 frequent categories and evaluated on remaining categories. The out-of-distribution results show severe phrase overfitting for supervised fine-tuning: SFT without reasoning reports 0.32 IoU / 0.08 pDice / 0.55 Dice with 1081 refusals, while SFT with reasoning reaches 6.32 / 1.56 / 13.45. By contrast, RL-Soft reports 17.27 / 12.02 / 25.2 without reasoning and 17.67 / 11.33 / 26.72 with reasoning, both with 0 refusals (Yan et al., 11 Aug 2025).

The paper also implies several limitations. First, the dataset uses GPT-4o-generated descriptions, QA formats, and reasoning traces, although these are manually filtered; the reasoning annotations are therefore useful supervision but not guaranteed faithful models of clinician cognition. Second, the appendix notes class imbalance: the four largest super-categories—abdomen, heart, neoplasm, and non-neoplasm—account for 59% of samples. Third, some domains remain substantially harder than others, especially histology, indicating that the dataset aggregates heterogeneous visual regimes under one benchmark. Fourth, the benchmark is based on public-source masks and synthetic query generation rather than prospectively collected clinical dialogue, so its implicitness is simulated rather than observed in routine workflow (Yan et al., 11 Aug 2025).

U-MRG-14K should also be distinguished from similarly named but unrelated entities. A chest X-ray report-generation paper, “M4CXR: Exploring Multi-task Potentials of Multi-modal LLMs for Chest X-ray Interpretation,” explicitly does not mention U-MRG-14K and instead uses a MIMIC-CXR-derived report-generation setup (Park et al., 2024). The dataset should likewise not be confused with ResearchMath-14k, a research-level mathematics dataset of 14,056 problems whose paper states that it does not mention “U-MRG-14K” and provides no alias mapping (Son et al., 27 May 2026). Nor does the U-MRG-14K name appear in “API Security Based on Automatic OpenAPI Mapping,” whose MRG refers to Map Reduce Graph, an unsupervised REST-API modeling method rather than a medical grounding benchmark (Levi et al., 21 Apr 2026).

These distinctions matter because the string “MRG” is overloaded across medical imaging, report generation, mathematics, and API security. In current arXiv usage, U-MRG-14K refers specifically to the medical grounding dataset introduced with UMRG and MedReasoner (Yan et al., 11 Aug 2025).