Unified Medical Reasoning Grounding

• UMRG is a research paradigm that integrates implicit clinical queries with explicit evidence localization, yielding structured outputs like masks, boxes, and key points.
• It employs methods such as prompt prediction, modular segmentation, and token bridging to unify 2D and 3D medical imaging tasks, enhancing accuracy and interpretability.
• UMRG leverages specialized datasets and reinforcement learning strategies to optimize the alignment of reasoning outputs with clinically relevant spatial evidence.

Unified Medical Reasoning Grounding (UMRG) denotes a line of research that seeks to couple medical reasoning with explicit evidence localization rather than treating inference and perception as separate stages. In its clearest task-level formulation, UMRG requires a system, given a medical image $\mathcal{I}$ and an implicit clinical query $\mathcal{Q}$, to predict $$\{\mathcal{T}, \mathcal{B}, \mathcal{P}_1, \mathcal{P}_2, \mathcal{M}\} = \mathbf{G}(\mathcal{I}, \mathcal{Q})$$, where $\mathcal{T}$ is a reasoning trace, $\mathcal{B}$ a bounding box, $\mathcal{P}_1,\mathcal{P}_2$ two key semantic points, and $\mathcal{M}$ a pixel-level mask (Yan et al., 11 Aug 2025). Across the broader literature, the same unifying ambition appears in systems that map language to boxes, masks, slice anchors, cuboids, visual tokens, or clinically structured supports, spanning 2D imaging, volumetric CT, surgical video, medical image sequences, and patient-contextual language reasoning (Chen et al., 10 Jun 2026, Wang et al., 26 May 2026, Huang et al., 13 May 2026).

1. Definition, scope, and neighboring formulations

UMRG emerged from the observation that many medical tasks are underspecified if framed only as classification, report generation, or segmentation. Clinical questions are often implicit, spatially latent, and semantically constrained by anatomy, pathology, procedure phase, or patient context. The formal definition introduced for UMRG emphasizes this gap directly: the model must interpret an implicit query, reason over visual cues and anatomical or clinical priors, and produce pixel-level grounding, optionally with interpretable intermediate reasoning (Yan et al., 11 Aug 2025).

The scope of the term is broader in later work. Some papers use UMRG to describe unified medical QA across output formats, including MCQ, numerical reasoning, and open-ended generation, with explicit > ... and <answer>...</answer> structure (Zhang et al., 14 Jun 2025). Others extend grounding beyond images alone: MediEval links factual statements to both biomedical knowledge and patient-specific EHR context (Qu et al., 23 Dec 2025), while MIND grounds psychiatric inquiry and diagnosis in criteria-aligned support notes retrieved from a Criteria-Grounded Psychiatric Reasoning Bank (Li et al., 4 Mar 2026). This suggests that “grounding” in the UMRG literature is not limited to pixel evidence; it can also denote alignment to patient context, diagnostic criteria, or external factual supports.

Adjacent formulations refine the output space rather than the reasoning objective. Generalized Medical Phrase Grounding reformulates phrase grounding so that a sentence may map to zero, one, or multiple scored regions, explicitly handling non-groundable phrases such as negations, normal anatomy, or administrative text (Zhang et al., 30 Nov 2025). That formulation is narrower than full UMRG, but it addresses a recurrent issue in grounded medical language: the world is not always single-box, and not every phrase should be localized.

2. Interface designs that connect reasoning to evidence

A central design problem in UMRG is how to expose an intermediate representation that is both semantically meaningful to the reasoner and operationally useful to the grounding module. One common pattern is explicit prompt prediction followed by modular segmentation. MedReasoner separates a clinical reasoning module from a frozen anatomical segmentation module: the reasoner produces a chain of thought plus a structured answer containing a bounding box and two key points, and a frozen MedSAM2 converts these prompts into the final mask (Yan et al., 11 Aug 2025). MedVol-R1 applies the same decoupling idea in 3D: the LVLM first predicts a key axial slice and 2D bounding boxes as verifiable evidence anchors, and a frozen MedSAM2 then propagates these into a coherent volumetric mask (Wang et al., 26 May 2026).

A second pattern uses special language tokens as semantic-visual bridges. SurgMLLM inserts a [SEG] token immediately after each detected instrument or target name in structured surgical reasoning output; these tokens are temporally aggregated and used as prompts for SAM2-based segmentation (Huang et al., 13 May 2026). MedVL-SAM2 likewise uses the hidden state of an LLM-emitted [SEG] token to prompt a SAM2-based volumetric segmentation module, unifying report generation, VQA, and semantic, referring, and interactive segmentation in 3D (Xing et al., 14 Jan 2026). Citrus-V follows the same bridging principle through a segmentation projector that maps the [SEG] token hidden state into a SAM2-inspired segmentation backbone (Wang et al., 23 Sep 2025).

A third pattern standardizes grounded reasoning syntax across modalities. UniReason-Med uses shared box syntax, region-token injection, and a common grounded reasoning policy so that the same checkpoint can process either a 2D image or a slice-serialized 3D volume at inference time (Chen et al., 10 Jun 2026). ViToS moves the interface from symbolic prompts to visual token control: a localization branch predicts grounding regions, after which a reasoning branch answers using token-sparse visual input produced by grounding-aware visual token pruning (Chen et al., 30 Jun 2026). In a related but more detection-oriented direction, MedGrounder uses set prediction to support zero, one, or multiple regions with confidence scores, making abstention part of the grounded interface rather than a failure mode (Zhang et al., 30 Nov 2025).

3. Datasets and benchmarks for unified reasoning and grounding

The rapid expansion of UMRG work is closely tied to new datasets that combine language supervision with explicit spatial or contextual evidence. Earlier medical VQA and report datasets usually lacked usable localization targets, while segmentation datasets lacked reasoning-oriented language. The newer benchmarks are explicitly designed to close that gap.

Resource	Scope	Role in UMRG
U-MRG-14K (Yan et al., 11 Aug 2025)	14,000 samples; 10 modalities; 15 super-categories; 108 specific categories	Implicit clinical queries, reasoning traces, masks, boxes, and key points
CholecT45-Scene (Huang et al., 13 May 2026)	64,299 frames with pixel-level masks for instruments and targets	Joint evaluation of phases, IVT triplets, and segmentation
MedSG-Bench / MedSG-188K (Yue et al., 17 May 2025)	76 public datasets; 10 modalities; 9,630 VQA-style QA pairs; ~188K instruction-tuning samples	Sequential grounding via Image Difference and Image Consistency Grounding
UniMed-CoT (Chen et al., 10 Jun 2026)	220K instruction-tuning samples; 170K 2D and 50K 3D	Shared grounded reasoning supervision across 2D and 3D
MedGround-35K (Zhang et al., 11 Jan 2026)	35,480 image-query-box triplets; 25,420 train / 10,060 test	Verified medical referring grounding data
MediEval (Qu et al., 23 Dec 2025)	MIMIC-IV linked to UMLS and other biomedical vocabularies	Patient-contextual and knowledge-grounded reasoning evaluation

These resources differ substantially in what counts as evidence. U-MRG-14K pairs implicit queries with masks, boxes, points, and reasoning traces. CholecT45-Scene aligns pixel masks with surgical phases and instrument-verb-target triplets. MedSG-Bench organizes eight VQA-style tasks into Image Difference Grounding and Image Consistency Grounding, thereby making temporal and cross-image dependence a first-class evaluation target. UniMed-CoT focuses on interleaved text-and-evidence supervision across 2D and 3D. MedGround-35K is built through an overview-and-verification pipeline that converts segmentation resources into referring-localization data. MediEval, by contrast, is not an image-grounding benchmark but a benchmark of whether a model can distinguish knowledge grounding from patient-context support.

This diversity is consequential. It suggests that UMRG is increasingly benchmarked along several axes simultaneously: implicitness of the query, dimensionality of the evidence, temporal structure, granularity of the spatial target, and the distinction between medically true and patient-supported statements.

4. Optimization strategies: supervised alignment, verifiable rewards, and reinforcement learning

Most UMRG systems rely on structured outputs that can be parsed, checked, and scored. Med-U1 standardizes reasoning with <think> and <answer> tags and optimizes a mixed reward composed of format reward, correctness reward, and a length matching reward under Group Relative Policy Optimization (GRPO) (Zhang et al., 14 Jun 2025). MedFact-R1 also uses a two-stage scheme—pseudo-label supervised fine-tuning followed by GRPO—but its reward is explicitly factual: accuracy, format, fact, and consistency rewards are combined to promote self-consistent medical reasoning (Li et al., 18 Sep 2025).

A second theme is domain- or guideline-aware reinforcement learning. MMedExpert-R1 builds specialty-specific LoRA experts through Domain-Specific Adaptation, then uses Guideline-Based Advantages to model distinct clinical reasoning perspectives, and finally merges them through conflict-aware capability integration with TIES-Merging (Ding et al., 16 Jan 2026). MIND uses rubric-based process rewards over symptom analysis, differential or exclusion consideration, and decision logic, together with retrieval shaping, information gain, and a value-aware trajectory rectification mechanism to manage inquiry drift in psychiatric consultation (Li et al., 4 Mar 2026).

On the visual-grounding side, the reward functions become geometric and evidence-specific. MedReasoner trains only the reasoning module, keeping the segmentation expert frozen, and scores outputs using format rewards plus spatial rewards over boxes and semantic points (Yan et al., 11 Aug 2025). MedVol-R1 uses a multi-component reward that includes format compliance, axial localization, 2D box spatial localization, and cross-slice consistency after MedSAM2 propagation (Wang et al., 26 May 2026). MedMO adds a box-level reward built from normalized L1 distance, GIoU, Hungarian matching, and false-positive or false-negative penalties, explicitly incorporating verifiable spatial supervision into RL (Deria et al., 6 Feb 2026). ViToS formulates grounding and reasoning as two branches of one policy model and uses cross-feedback sequential optimization so that localization and token-sparse reasoning can co-adapt without destabilizing one another (Chen et al., 30 Jun 2026).

A plausible implication is that UMRG is converging on a specific methodological template: generate structured intermediate evidence, define rewards on that evidence rather than on text alone, and treat segmentation or localization modules as either frozen experts or promptable subsystems whose behavior can be externally verified.

5. Representative systems and empirical results across domains

The empirical record in the current literature is strongest in domains where a model must jointly predict semantics and evidence. In surgical scene understanding, SurgMLLM improves the primary triplet recognition metric $AP_{IVT}$ from $40.7\%$ to $46.0\%$, raises phase recognition to Accuracy $\mathcal{Q}$0, Recall $\mathcal{Q}$1, and Jaccard $\mathcal{Q}$2, and increases entity segmentation from $\mathcal{Q}$3 to $\mathcal{Q}$4 mIoU (Huang et al., 13 May 2026). These gains are reported in a setting where phases, IVT triplets, and triplet-entity segmentation are modeled jointly rather than in isolation.

In sequential grounding, MedSG-Bench shows that even advanced MLLMs perform poorly on image sequences, especially on Image Difference Grounding. The best cited open baseline, Migician, reaches $\mathcal{Q}$5 average IoU and $\mathcal{Q}$6 [email protected], whereas MedSeq-Grounder reaches $\mathcal{Q}$7 mean IoU and $\mathcal{Q}$8 [email protected] after instruction tuning on MedSG-188K (Yue et al., 17 May 2025). This is a particularly direct result for UMRG because the benchmark is designed around cross-image semantic alignment, not single-image recognition.

For unified 3D reasoning and segmentation, MedVL-SAM2 reports VQA Accuracy $\mathcal{Q}$9, report-generation BERTScore $$\{\mathcal{T}, \mathcal{B}, \mathcal{P}_1, \mathcal{P}_2, \mathcal{M}\} = \mathbf{G}(\mathcal{I}, \mathcal{Q})$$0, referring segmentation Dice of $$\{\mathcal{T}, \mathcal{B}, \mathcal{P}_1, \mathcal{P}_2, \mathcal{M}\} = \mathbf{G}(\mathcal{I}, \mathcal{Q})$$1 on CTOrg, $$\{\mathcal{T}, \mathcal{B}, \mathcal{P}_1, \mathcal{P}_2, \mathcal{M}\} = \mathbf{G}(\mathcal{I}, \mathcal{Q})$$2 on ACT-1K, and $$\{\mathcal{T}, \mathcal{B}, \mathcal{P}_1, \mathcal{P}_2, \mathcal{M}\} = \mathbf{G}(\mathcal{I}, \mathcal{Q})$$3 on TotalSegmentator, and interactive segmentation Dice of $$\{\mathcal{T}, \mathcal{B}, \mathcal{P}_1, \mathcal{P}_2, \mathcal{M}\} = \mathbf{G}(\mathcal{I}, \mathcal{Q})$$4 on TotalSegmentator with box prompts (Xing et al., 14 Jan 2026). UniReason-Med addresses a different question—whether 2D grounded reasoning supervision transfers to 3D—and reports that joint 2D+3D training raises M3D-VQA mean accuracy from $$\{\mathcal{T}, \mathcal{B}, \mathcal{P}_1, \mathcal{P}_2, \mathcal{M}\} = \mathbf{G}(\mathcal{I}, \mathcal{Q})$$5 to $$\{\mathcal{T}, \mathcal{B}, \mathcal{P}_1, \mathcal{P}_2, \mathcal{M}\} = \mathbf{G}(\mathcal{I}, \mathcal{Q})$$6; it also reports a Dice improvement on MSD from $$\{\mathcal{T}, \mathcal{B}, \mathcal{P}_1, \mathcal{P}_2, \mathcal{M}\} = \mathbf{G}(\mathcal{I}, \mathcal{Q})$$7 after SFT to $$\{\mathcal{T}, \mathcal{B}, \mathcal{P}_1, \mathcal{P}_2, \mathcal{M}\} = \mathbf{G}(\mathcal{I}, \mathcal{Q})$$8 after SFT+RL and a positive correlation between correct answer and correct spatial grounding of $$\{\mathcal{T}, \mathcal{B}, \mathcal{P}_1, \mathcal{P}_2, \mathcal{M}\} = \mathbf{G}(\mathcal{I}, \mathcal{Q})$$9 with $\mathcal{T}$0 (Chen et al., 10 Jun 2026).

General-purpose medical MLLMs also show substantial gains when grounding is built into training. MedMO reports an average VQA improvement of $\mathcal{T}$1 over its baseline, $\mathcal{T}$2 on text-based QA, and an IoU improvement of $\mathcal{T}$3 over the baseline and $\mathcal{T}$4 over Fleming-VL (Deria et al., 6 Feb 2026). MedReasoner reports the best overall performance on U-MRG-14K with IoU $\mathcal{T}$5, pDice $\mathcal{T}$6, and Dice $\mathcal{T}$7, compared with $\mathcal{T}$8, $\mathcal{T}$9, and $\mathcal{B}$0 for the next-best baseline named in the comparison (Yan et al., 11 Aug 2025). In volumetric reasoning segmentation, MedVol-R1 reaches $\mathcal{B}$1 DSC and $\mathcal{B}$2 IoU on AbdomenCT-1K, $\mathcal{B}$3 DSC and $\mathcal{B}$4 IoU on CT-ORG, and $\mathcal{B}$5 DSC and $\mathcal{B}$6 IoU on KiTS23, while explicitly grounding free-form clinical queries to key slices and 2D boxes (Wang et al., 26 May 2026).

These results do not establish a single dominant architecture. They do indicate, however, that explicit intermediate grounding—whether as [SEG] tokens, boxes, points, cuboids, or slice anchors—can improve both interpretability and task performance when the supervision and reward design make those intermediates consequential.

6. Evaluation beyond accuracy: grounding fidelity, hallucination, and safety

A recurring controversy in this area is whether benchmark accuracy actually measures grounded medical reasoning. Counterfactual evaluation suggests that it often does not. On four medical VQA benchmarks, text-only RLVR achieves a negative Visual Reliance Score on PathVQA of $\mathcal{B}$7, image-text RLVR reduces Image Sensitivity to $\mathcal{B}$8 overall despite improving accuracy, and models generate visual claims in $\mathcal{B}$9–$\mathcal{P}_1,\mathcal{P}_2$0 of responses while $\mathcal{P}_1,\mathcal{P}_2$1–$\mathcal{P}_1,\mathcal{P}_2$2 are ungrounded according to the Hallucinated Visual Reasoning Rate (Zafar et al., 3 Mar 2026). This directly challenges the assumption that better answer accuracy implies stronger visual grounding.

MediEval extends this critique from image dependence to patient-context dependence. Its 4-quadrant framework distinguishes True-Supported, True-Unsupported, False-Supported, and False-Unsupported statements, thereby exposing failure modes such as hallucinated support and truth inversion (Qu et al., 23 Dec 2025). The proposed Counterfactual Risk-Aware Fine-tuning (CoRFu) improves macro-F1 by $\mathcal{P}_1,\mathcal{P}_2$3 over the base model and eliminates truth inversion errors in the regime targeting this error, showing that safety-relevant confusions can be isolated and penalized rather than merely averaged into overall accuracy.

Other work focuses on output discipline and evidence verification. A context-aligned reasoning framework augments a frozen VLM with radiomic statistics, explainability activations, and vocabulary-grounded semantic cues, and requires structured outputs containing supporting evidence, uncertainty estimates, limitations, and safety notes. On chest X-ray datasets it improves AUC from $\mathcal{P}_1,\mathcal{P}_2$4 to $\mathcal{P}_1,\mathcal{P}_2$5, reduces hallucinated keywords from $\mathcal{P}_1,\mathcal{P}_2$6 to $\mathcal{P}_1,\mathcal{P}_2$7, shortens explanations from $\mathcal{P}_1,\mathcal{P}_2$8 to $\mathcal{P}_1,\mathcal{P}_2$9 words, and keeps model confidence from increasing, moving from $\mathcal{M}$0 to $\mathcal{M}$1 (Khan et al., 9 Apr 2026). MedGround treats the data pipeline itself as a grounding problem: its multi-stage verification system includes strict formatting checks, geometry- and medical-prior rules, and image-based visual judging, with overall retention of approximately $\mathcal{M}$2 and VLM-based judge pass rates of $\mathcal{M}$3–$\mathcal{M}$4 depending on the modality (Zhang et al., 11 Jan 2026).

Taken together, these findings suggest that UMRG cannot be reduced to jointly training a reasoner and a segmenter. The harder requirement is causal dependence on appropriate evidence: the answer should change when the relevant evidence changes, the localized evidence should correspond to the claim being made, and the claim should remain faithful to patient context, diagnostic criteria, and domain facts. In that sense, UMRG is as much an evaluation doctrine as an architectural one.