PathMR: Multimodal Pathology Reasoning

• PathMR is a cell-level multimodal visual reasoning framework designed for pathology analysis that fuses text and pixel-level segmentation.
• It integrates advanced vision backbones like SAM ViT-H with large language models to generate synchronized diagnostic narratives and nuclei masks.
• The framework enhances clinical interpretability by aligning expert-style textual explanations with precise segmentation, improving accuracy and traceability.

Searching arXiv for PathMR and closely related multimodal pathology reasoning papers to ground the article in current literature. PathMR is a cell-level multimodal visual reasoning framework for pathological image analysis that takes an H&E-stained pathology image patch $x_{img} \in \mathbb{R}^{h \times w \times 3}$ and an associated textual query $x_{txt}$, and jointly generates an expert-level diagnostic explanation $\hat{y}_{txt}$ and cell-level segmentation and classification masks $\hat{M}$. It was proposed to address a core limitation of current deep learning systems in computational pathology: models can be highly accurate but are often black boxes, providing neither transparent decision paths nor clinically traceable rationales. PathMR couples pixel-wise nuclei localization with semantically aligned diagnostic narratives, and this suggests an explicit attempt to align machine inference with the way pathologists reason about cell morphology, cell types, and their spatial distribution (Zhang et al., 28 Aug 2025).

1. Clinical motivation and conceptual scope

PathMR is positioned against two dominant classes of prior systems in computational pathology. The first class comprises classification / MIL-based WSI analysis methods, which typically output a slide-level diagnosis and a coarse attention map indicating which patches are important. The paper identifies three limitations of this paradigm: granularity, because each patch may contain hundreds of heterogeneous cells; non-traceable rationale, because attention scores do not provide structured textual reasoning or explicit morphological descriptions; and limited clinical interpretability, because there is no direct mapping to the kind of narrative report that pathologists write (Zhang et al., 28 Aug 2025).

The second class comprises multimodal pathology models without pixel-level grounding, including systems such as WSI-VQA, WSICaption, PathGen, HistGen, Virchow, CONCH, CHIEF, GigaPath, and mSTAR. These models can generate captions or answer questions about whole-slide images and may classify and localize slide regions, but they typically do not perform pixel-level segmentation at the cell level and lack synchronous alignment between text explanations and fine-grained masks. PathMR is therefore framed as a pathology-specific extension of multimodal visual reasoning, defined here as joint reasoning over images and text in a unified model that produces both a textual answer and pixel-level segmentation masks of nuclei or cells with class labels (Zhang et al., 28 Aug 2025).

Within this formulation, PathMR is designed specifically to bring pixel-level reasoning architectures such as LISA, PixelLM, MMR, and GSVA into computational pathology, with additional mechanisms tailored to cell morphology and class stability. Its intended representational unit is not the slide or patch in isolation, but the nucleus-level visual field in which classes such as neoplastic, inflammatory, connective, and epithelial cells can be localized and described. This makes the framework especially relevant for diagnostic settings where the discriminative signal lies in the joint configuration of nuclear phenotypes and tissue microenvironment (Zhang et al., 28 Aug 2025).

2. Framework architecture and multimodal reasoning process

The PathMR architecture consists of four coupled components: a vision backbone, a vision decoder, a multimodal LLM, and a set of loss components that tie image reasoning and segmentation together. The visual perception module uses SAM ViT-H as the visual encoder. Following MMR, it aggregates a global feature $$v_g \in \mathbb{R}^{\frac{h}{16} \times \frac{w}{16} \times c}$$ and an early-stage local feature $$v_l \in \mathbb{R}^{\frac{h}{16} \times \frac{w}{16} \times c'}$$, and fuses them into a segmentation feature

$v_{seg} = v_g + \text{Conv}(v_l).$

The vision decoder then combines this visual segmentation representation with segmentation-related features from the LLM and predicts masks through

$$\hat{M} = \text{Dec}(\text{Concat}(v_{seg}, f_{seg})).$$

In the implementation described in the paper, $\text{Dec}(\cdot)$ is a SAM ViT-H decoder, and the decoder supports multi-class nuclei segmentation aligned with the reasoning process (Zhang et al., 28 Aug 2025).

The multimodal LLM is instantiated with vision-language backbones such as LLaVA-7B, LLaVA-13B, and, in PathGen experiments, Qwen. A separate CLIP-ViT-L/14-336 visual encoder produces visual tokens $f_{img} \in \mathbb{R}^{N_{img} \times d}$, while the LLM tokenizer embeds the textual query into $x_{txt}$0. These embeddings are concatenated and processed by the LLM $x_{txt}$1: $x_{txt}$2 The resulting sequence contains ordinary natural-language tokens together with special $x_{txt}$3 or $x_{txt}$4 tokens, which serve as segmentation control tokens (Zhang et al., 28 Aug 2025).

The connection between the LLM and the segmentation branch is central to PathMR’s reasoning design. From the generated sequence $x_{txt}$5, the model filters the hidden states corresponding to $x_{txt}$6 tokens to obtain $x_{txt}$7, which are then mapped to the conditioning representation $x_{txt}$8 used by the visual decoder. Informally, at each $x_{txt}$9 token, the LLM “asks” the visual module to segment the region corresponding to the class currently being described. This establishes the paper’s notion of synchrony: segmentation masks are produced in lockstep with the explanatory text, so that a phrase such as “neoplastic cells” is paired with a grounded mask for the corresponding nuclei class (Zhang et al., 28 Aug 2025).

The reasoning pipeline is therefore explicitly multimodal and autoregressive. First, the image is encoded twice, once through CLIP for language conditioning and once through SAM for segmentation features. Second, the LLM generates an answer sequence that may include one or more $\hat{y}_{txt}$0 tokens. Third, the filtered segmentation-token states are transformed into $\hat{y}_{txt}$1. Fourth, the decoder predicts masks conditioned jointly on $\hat{y}_{txt}$2 and $\hat{y}_{txt}$3. The paper notes that it does not present explicit attention equations inside the LLM beyond standard transformer operations and treats the LLM as a black box $\hat{y}_{txt}$4, while formalizing the aggregation and loss functions explicitly (Zhang et al., 28 Aug 2025).

3. Objectives, constraints, and optimization

PathMR is trained by jointly optimizing a mask loss $\hat{y}_{txt}$5, a text loss $\hat{y}_{txt}$6, and a consistency loss $\hat{y}_{txt}$7. The segmentation objective is

$\hat{y}_{txt}$8

where $\hat{y}_{txt}$9 is the predicted mask, $\hat{M}$0 is the ground-truth mask, $\hat{M}$1 is a penalty weight set to $\hat{M}$2, and the experimental settings are $\hat{M}$3 and $\hat{M}$4. The paper describes this as a classification supervision mechanism intended to stabilize per-class masks and reduce mislabelled fragments. Because the LLM predicts $\hat{M}$5 tokens sequentially and future classes are unknown at the current step, the model tends to label other categories as the current class; PathMR responds by adding penalties for such mis-assignments and explicitly supervising per-class predictions using known multi-class labels (Zhang et al., 28 Aug 2025).

The morphological consistency constraint is introduced to enforce smoothness and coherent cell shapes. Let $\hat{M}$6 denote the predicted probability at pixel $\hat{M}$7. Then

$\hat{M}$8

where $\hat{M}$9 is the total number of valid neighboring pairs. This penalizes local discontinuities in the predicted mask and is intended to reduce boundary noise and fragmented tiny blobs inconsistent with nuclei morphology (Zhang et al., 28 Aug 2025).

The language modeling term is the standard cross-entropy objective

$$v_g \in \mathbb{R}^{\frac{h}{16} \times \frac{w}{16} \times c}$$0

where $$v_g \in \mathbb{R}^{\frac{h}{16} \times \frac{w}{16} \times c}$$1 is the ground-truth answer sequence. This loss supervises both the narrative content and the correct placement of segmentation tokens. The total objective is

$$v_g \in \mathbb{R}^{\frac{h}{16} \times \frac{w}{16} \times c}$$2

with default $$v_g \in \mathbb{R}^{\frac{h}{16} \times \frac{w}{16} \times c}$$3 (Zhang et al., 28 Aug 2025).

Optimization uses AdamW with learning rate 0.0003. The LLM is fine-tuned with LoRA at rank $$v_g \in \mathbb{R}^{\frac{h}{16} \times \frac{w}{16} \times c}$$4. Training is performed on 8× A100 GPUs, with reported training times of approximately 1 day for the 7B model and 1.5 days for the 13B model. Batch sizes and steps per epoch are reported as 16 and 1200 for the 7B setting, and 6 and 3125 for the 13B setting. The paper states that no multi-stage curriculum is described beyond joint end-to-end optimization (Zhang et al., 28 Aug 2025).

4. Data resources and evaluation protocol

PathMR is evaluated on two datasets: GADVR and PathGen. GADVR, short for Gastric Adenocarcinoma Diagnosis Visual Reasoning, is described as a new dataset specifically built for pixel-level visual reasoning in pathology. It is based on PatchGastricADC22, which contains 262,777 pathological image patches with resolution $$v_g \in \mathbb{R}^{\frac{h}{16} \times \frac{w}{16} \times c}$$5, magnification $$v_g \in \mathbb{R}^{\frac{h}{16} \times \frac{w}{16} \times c}$$6, H&E staining, and 9 gastric adenocarcinoma subtypes. The final GADVR corpus contains approximately 190,000 image patches and more than 547,000 image-text pairs, with a split by whole-slide image at a ratio of 8:1:1 to avoid patient-level leakage: 149,928 training patches with 447,170 pairs, 14,814 validation patches with 44,080 pairs, and 18,754 test patches with 56,067 pairs (Zhang et al., 28 Aug 2025).

Nuclei segmentation and classification in GADVR are produced through a multi-step pipeline. First, pretrained CLIP-ViT and WSI-level diagnostic reports are used to filter out patches weakly correlated with their WSI subtype labels. Second, HoverNet pretrained on PanNuke provides initial pixel-level nuclear segmentation and nuclear classification. Third, two pathologists manually correct nuclear classifications on approximately 200 patches per category; the model is refined; and final segmentation and classification are re-checked by three independent pathologists. In quality control over 200 random samples, more than 92% of segmentation masks were rated Correct, and more than 95% of QA pairs were rated Okay or Good (Zhang et al., 28 Aug 2025).

Question–answer data in GADVR are built from an approximately 200-question pool generated by GPT-4o. For each image, 3 random questions are drawn from the pool, and GPT-4o generates answers based on the question and the segmentation masks, with prompts designed to ensure that answers explicitly reference nuclear categories and spatial patterns. The dataset supports three task types: reasoning segmentation, referring segmentation, and conversation (Zhang et al., 28 Aug 2025).

PathGen is used as a complementary benchmark for generalization. It is described as a pan-cancer patch-level image–text dataset in which PathGen-1.6M originally provides approximately 1.6M image–text pairs from more than 25 organs. In the PathMR study, the authors randomly sample 50,000 images to form an independent training set. Because PathGen does not provide cell-level segmentation annotations, PathMR generates pseudo segmentation labels using the same pipeline as GADVR, namely HoverNet + postprocessing (Zhang et al., 28 Aug 2025).

Dataset	Composition	Use in PathMR
GADVR	~190,000 image patches; >547,000 image-text pairs; split 8:1:1 by WSI	Reasoning segmentation, referring segmentation, conversation
PathGen	~1.6M original pairs from >25 organs; 50,000 images sampled in this paper	Generalization across diverse tissue types

Evaluation is conducted against both pixel-level reasoning models—Ov-Seg, LISA, PixelLM, GSVA, and MMR—and pathology-specific multimodal LLMs such as BiomedParse and LLaVA-Med. For segmentation, the study adopts gIoU and cIoU following PixelLM and MMR. For the conversation task, it reports BLEU-4 and F1. The paper explicitly notes that there is no separate explicit metric for cross-modal alignment; instead, alignment is evaluated indirectly through segmentation quality in reasoning and referring tasks and text metrics in conversation (Zhang et al., 28 Aug 2025).

5. Empirical performance and ablation findings

On reasoning segmentation in GADVR, PathMR improves over MMR at both model scales. For the 7B setting, MMR-7B obtains validation gIoU 0.587 and cIoU 0.698, with test overall gIoU 0.602 and cIoU 0.714. Under the same TFLOPs, PathMR-7B reaches validation gIoU 0.603 and cIoU 0.715, and test overall gIoU 0.629 and cIoU 0.734, corresponding to gains of about +2.7 points in gIoU and +2.0 points in cIoU. For the 13B setting, MMR-13B reports validation 0.598/0.708 and test overall 0.613/0.724, whereas PathMR-13B reaches validation 0.624/0.723 and test overall 0.633/0.740, yielding gains of +2.0 points gIoU and +1.6 points cIoU (Zhang et al., 28 Aug 2025).

On PathGen, the reported results indicate cross-organ generalization. LISA-LLaVA-13B achieves overall gIoU 0.484 and cIoU 0.519; MMR-LLaVA-13B reaches 0.545/0.626; PathMR-LLaVA-13B reaches 0.560/0.635; and PathMR-Qwen reaches 0.579/0.641. The paper summarizes this as a 1.5-point gIoU improvement of PathMR-LLaVA-13B over MMR-13B, with further gains from the Qwen-based variant (Zhang et al., 28 Aug 2025).

For referring segmentation on GADVR, PathMR again exceeds MMR. At 7B scale, MMR-7B achieves overall gIoU 0.527 and cIoU 0.634, while PathMR-7B reaches 0.539/0.657. At 13B scale, MMR-13B reaches 0.515/0.664, whereas PathMR-13B reaches 0.530/0.682. The paper states that the gains are especially prominent for rare or challenging categories, such as epithelial cells, where many baselines fail to segment accurately (Zhang et al., 28 Aug 2025).

For the conversation task on GADVR, the strongest baseline numbers reported are LISA-13B with validation BLEU-4 0.279, test 0.286, validation F1 0.586, and test 0.591; and LLaVA-Med with validation BLEU-4 0.284, test 0.283, validation F1 0.586, and test 0.591. PathMR reaches validation BLEU-4 0.301 and F1 0.609, and test BLEU-4 0.302 and F1 0.602. The reported improvements are approximately 1.8–2.3 points for BLEU-4 and 1.6–2.3 points for F1 relative to strong baselines (Zhang et al., 28 Aug 2025).

The qualitative analysis emphasizes that PathMR’s masks are less noisy, with more complete nuclei and more accurate boundaries than LISA, PixelLM, and GSVA, and that the generated text remains consistent with mask content. Ablation studies attribute these improvements to the two added constraints. In reasoning segmentation, the baseline without class supervision or consistency reaches gIoU 0.602; adding class supervision yields 0.609, adding the consistency constraint yields 0.605, and using both yields 0.629, with corresponding cIoU improvements from approximately 0.714 to approximately 0.73, 0.724, and 0.734. In referring segmentation for the 7B variant, the baseline yields overall gIoU 0.527 and cIoU 0.634; adding classification supervision yields 0.531/0.644; and adding both classification supervision and consistency yields 0.539/0.657 (Zhang et al., 28 Aug 2025).

6. Interpretability, limitations, and related terminology

The interpretability claims of PathMR rest on four linked properties. First, it provides cell-level localization through pixel-level nuclei segmentation and classification masks. Second, it generates expert-style textual explanations that explicitly reference cell morphology, spatial distribution patterns, and diagnostic reasoning for cancer subtypes. Third, it implements a tight coupling between masks and narratives, because each $$v_g \in \mathbb{R}^{\frac{h}{16} \times \frac{w}{16} \times c}$$7 token corresponds to a segmentation action and the dual-constraint mechanism produces cleaner masks aligned with the text. Fourth, this configuration is presented as being closer to pathologists’ workflow, in which tumor regions, cell shapes and density, and surrounding immune and stromal context are evaluated jointly. This suggests that PathMR is intended not only for prediction, but also for transparency, trust and validation, and educational and decision support (Zhang et al., 28 Aug 2025).

The paper also identifies several limitations. Data scope remains narrow because GADVR focuses on gastric adenocarcinoma, even though PathGen sampling extends experiments to multiple organs. Labeling and pseudo-labels remain a constraint, since GADVR labels are partly derived from HoverNet trained on PanNuke and then refined, while PathGen segmentation labels are fully automatic through the same pipeline. Clinical validation is absent, because current results are algorithmic offline benchmarks rather than clinical trials or prospective studies. In addition, the model currently focuses on nuclei segmentation, and tissue-level structures such as glands, layers, and boundaries are not explicitly modeled. Proposed future directions include incorporating tissue-level segmentation labels, expanding to broader cancer types and tissue morphologies, and moving toward integration in real clinical workflows, including whole-slide analysis and real-time decision support (Zhang et al., 28 Aug 2025).

A terminological distinction is useful. The name PathMR in histopathology should be distinguished from MR-Path, a likelihood-based mixture model for Mendelian randomization that clusters genetic instruments by latent causal effects and addresses mechanistic heterogeneity in summary-data MR (Iong et al., 2020). A separate MRI representation-learning paper, PEPSI, is also described as a prototype of the kind of “PathMR” system one might envision for brain MRI, but it is a different framework centered on pathology-enhanced, pulse-sequence-invariant representations rather than multimodal visual reasoning for pathology image patches (Liu et al., 2024).