---
title: PromptAnatomy for Pan-Tumor Segmentation
url: https://www.emergentmind.com/topics/promptanatomy
type: topic
---

# PromptAnatomy for Pan-Tumor Segmentation

Searching arXiv for the primary paper and closely related prompt-based medical segmentation work.

2503.14355

Search arXiv for: 2503.14355

Search arXiv for: "MAST-Pro Dynamic Mixture-of-Experts for Adaptive Segmentation of Pan-Tumors with Knowledge-Driven Prompts"

MAST-Pro—short for **Mixture-of-experts for Adaptive Segmentation of pan-Tumors with knowledge-driven Prompts**—is a framework for **pan-tumor segmentation**, meaning a single model that segments tumors arising in multiple organs and anatomical regions rather than being trained separately for one tumor type at a time. It is designed for the setting in which tumors vary substantially in **shape, texture, intensity, borders, and anatomical context**, while available datasets are strongly imbalanced across tumor types. The framework combines **knowledge-driven text prompts**, **anatomical prompts** derived from organ masks, a **Dynamic Mixture-of-Experts (D-MoE)** built into a SwinUNETR backbone, and **Parameter-Efficient Fine-Tuning (PEFT)** to improve segmentation accuracy without full-model adaptation [2503.14355].

## 1. Clinical and technical setting

MAST-Pro is motivated by a dual requirement. Clinically, cancer presents across many organs, and robust segmentation is needed for diagnosis, treatment planning, and monitoring. Technically, a universal tumor model must preserve features shared across tumors while also retaining cues that are highly specific to tumors in the liver, pancreas, colon, kidney, and other sites. The paper argues that generic foundation-model features alone are insufficient because they do not explicitly encode the medical priors that radiologists use implicitly, especially knowledge about **where tumors occur, what organ context they inhabit, what shapes and border characteristics are typical, and how these properties manifest under a given imaging modality** [2503.14355].

The paper positions earlier prompt-based approaches in two broad groups. **Vision-prompt-driven** systems such as Med-SAM3D and SegVol use prompts like points or boxes; the paper argues that these methods are annotation-dependent and do not inject enough anatomical or radiological prior knowledge to handle tumor heterogeneity. **Text-prompt-driven** systems such as the Universal Model and ZePT align image and text features, but often rely on predefined or relatively simple templates that, in the authors’ view, do not adequately capture “the extensive variability in tumor morphology and radiological presentation.” MAST-Pro is therefore framed as a response to three limitations: limited incorporation of medical priors, an imbalance between generic and tumor-specific features, and high computational costs for clinical adaptation [2503.14355].

A central point in this formulation is that MAST-Pro is not presented as a generic prompting layer attached to an otherwise unchanged segmentation model. Its prompting scheme is intended to encode **domain-specific priors**, while its expert-routing mechanism is intended to allocate representation capacity adaptively across tumor types. This suggests that the model treats tumor variability as both a **conditioning problem** and a **capacity-allocation problem**.

## 2. Knowledge-driven prompt construction

MAST-Pro uses two explicit prompt families: **text prompts** and **anatomical prompts**. The text prompts are structured, knowledge-driven descriptions generated with an LLM using a standardized medical template rather than free-form text. The prompt is defined as

\[
P_{txt} = \text{``This is a [C] in the [O], appearing as a [S] mass with [E] borders on [M].''}
\]

where \([C]\), \([O]\), \([S]\), \([E]\), and \([M]\) denote the **tumor category/type**, **organ or anatomical location**, **shape descriptor**, **edge/border characteristic**, and **imaging modality**, respectively [2503.14355].

This design is important because the prompt is not merely a class label. It bundles disease identity, anatomical site, morphology, boundary appearance, and modality-specific presentation into a reusable schema. The text prompt for tumor task \(t\) is encoded by a pretrained text encoder \(\mathcal{E}_{txt}\), cited to CLIP:

\[
E_{txt}^t = \mathcal{E}_{txt}(P_{txt}^t)
\]

where \(E_{txt}^t\) is the resulting embedding. The symbol \(t\) indexes the tumor type or segmentation task [2503.14355].

The second prompt family is the **anatomical prompt**. Rather than representing anatomy only as text, MAST-Pro uses **organ masks generated by TotalSegmentator**. These masks are passed through a pretrained anatomical encoder \(\mathcal{E}_a\), cited to SwinUNETR:

\[
E_{a}^t = \mathcal{E}_{a}(P_{a}^t)
\]

where \(P_a^t\) denotes the anatomical prompt and \(E_a^t\) its embedding [2503.14355].

This is a distinctive design choice. The anatomical prompt is a **spatially grounded prior** in mask form rather than a symbolic organ label. The paper describes this as more informative than a pure organ name because it provides geometric localization and shape context. The prompt sources themselves are largely **static and prior-driven**: text descriptions are generated according to a fixed schema, organ masks come from TotalSegmentator, and both encoders are pretrained. Their embeddings, however, become dynamic participants in downstream attention and expert routing.

## 3. Prompt-conditioned segmentation architecture

MAST-Pro is built on a **D-MoE-enhanced SwinUNETR**. Multi-anatomical CT images are processed by the transformer-based segmentation backbone, but prompt information is injected through **cross-attention** and residual fusion. The paper states that the extracted prompts act as queries, while image features act as keys and values:

\[
\mathcal{F}_{attn} = \text{Softmax} \left( \frac{[\mathcal{Q}_{txt}; \mathcal{Q}_{a}] \mathcal{K}^\top}{\sqrt{d}} \right) \mathcal{V},
\]

where \(\mathcal{Q}_{txt}\) and \(\mathcal{Q}_{a}\) are the text and anatomy prompt queries, \(\mathcal{K}\) and \(\mathcal{V}\) are the key and value projections of image features, and \(d\) is the scaling dimension [2503.14355].

In this formulation, the prompts do not merely concatenate with image features. They actively query the visual representation. The paper interprets this as a way for anatomical prompts to bias attention toward the relevant organ region and surrounding tissue context, while text prompts bias attention toward lesion appearance consistent with the described tumor morphology and modality. The attention-refined features are then fused with the image features and decoded through SwinUNETR:

\[
\mathcal{F}_{dec} = \mathcal{D} (\mathcal{F}_{attn} + \mathcal{F}_{img}),
\]

where \(\mathcal{F}_{img}\) denotes image features and \(\mathcal{D}\) is the SwinUNETR decoder [2503.14355].

The framework also introduces a **mask-proposal branch**. According to the paper, \(\mathcal{F}_{attn}\) undergoes global average pooling and is concatenated with text prompts, then passed to an MLP to generate an initial mask proposal \(\theta^t\), supervised by cross-entropy with tumor category information. The decoder output is then modulated by this proposal:

\[
\mathcal{F}_{final} = \mathcal{F}_{dec} \theta^t, \quad \text{where} \quad \theta^t = \text{MLP}(\text{GAP}(\mathcal{F}_{attn}); \mathcal{E}_{txt}).
\]

The notation is somewhat underspecified in the paper, but the intended behavior is explicit: prompt-conditioned features produce a task-aware proposal, and that proposal guides the final segmentation. The paper states that segmentation is refined with Dice loss, while the proposal branch is supervised with cross-entropy for tumor category prediction [2503.14355].

## 4. Dynamic mixture-of-experts and parameter-efficient adaptation

The **Dynamic Mixture-of-Experts (D-MoE)** is the other central component of MAST-Pro. Within the SwinUNETR block, the paper introduces two routing perspectives: a generalized router \(R^g\) for shared low-rank experts and a tumor-specific router \(R^t\) for selecting a combination of generic and tumor-specific experts. The adapted feature is defined as

\[
\bar{X}^t = \sum_{i=1}^{k} R^{t}_i (X^t) \cdot \left( E^t_i (X^t) ; E^g_i (X^t) \right),
\]

where \(X^t\) is the incoming feature for task \(t\), \(\bar{X}^t\) is the adapted feature, \(R_i^t(X^t)\) is the routing weight for the \(i\)-th selected expert, \(E_i^t\) is a tumor-specific expert, and \(E_i^g\) is a generic expert [2503.14355].

The routing weights are computed by sparse top-\(k\) routing:

\[
R(X) = \text{Softmax}(\text{KeepTop-}k(X^\top W, k)),
\]

with

\[
\text{KeepTop-}k(v, k)_i =
\begin{cases}
v_i, & \text{if } v_i \text{ is in top } k \text{ elements of } v,\\
-\infty, & \text{otherwise}.
\end{cases}
\]

This yields sparse expert activation, so only a few experts are used per instance or task. The paper states that the purpose is to preserve “foundational morphological and intensity patterns” while adapting to the peculiarities of each tumor type [2503.14355].

The model’s **PEFT** strategy is tied directly to D-MoE. Instead of fully fine-tuning the SwinUNETR backbone, MAST-Pro first uses large-scale medical pretraining and then fine-tunes by updating only a **small subset of low-rank experts and routers**. The paper further specifies that in D-MoE, \(k_2\) experts encode generic tumor features and \(k_1\) experts capture domain-specific variations, while routers select the top \(k\) from both groups [2503.14355].

The efficiency claims are concrete. MAST-Pro reports **21.04M trainable parameters**, compared with **244.80M** for Universal Model, **495.10M** for ZePT, **449.08M** for SegVol, and **363.68M** for MA-SAM. The paper states this corresponds to a **91.04% reduction in trainable parameters** without accuracy loss. Reported GPU memory usage is **8961.30 MB**, below Universal Model (**9710.55 MB**), SegVol (**19898.00 MB**), ZePT (**24705.40 MB**), and MA-SAM (**74544.02 MB**) [2503.14355].

## 5. Experimental program and empirical findings

The training and evaluation protocol is broad in anatomical coverage. For pretraining, the paper lists BTCV, CT-ORG, Pancreas-CT, CHAOS, 3D-IRCADb, WORD, AMOS, AbdomenCT-1K, LiTS, KiTS, and CT images from MSD. For pan-tumor training and evaluation, the authors curate **2,000+ tumor cases from eight tumor-specific datasets**: MSD-Liver (M-Li), MSD-Lung (M-Lu), MSD-Pancreas (M-Pa), MSD-HepaticVessel Tumor (M-HT), MSD-Colon (M-Co), LiTS, KiTS, and AbdomenCT-1K. All scans are resampled to \(1.5 \times 1.5 \times 1.5\ \text{mm}^3\), cropped to tumor-relevant regions, and trained with \(96 \times 96 \times 96\) patches. The evaluation metric is Dice Similarity Coefficient (DSC) [2503.14355].

Quantitatively, MAST-Pro achieves the best overall mean DSC of **68.71%**, compared with **63.51%** for ZePT, **62.79%** for Universal Model, **60.84%** for SegVol, and **61.10%** for Swin UNETR. The paper highlights a **5.20%** average DSC improvement over ZePT. Dataset-specific results include **59.34%** on M-Pa, **74.76%** on M-HT, **82.12%** on LiTS, and **72.99%** on KiTS, with top performance on six of eight datasets [2503.14355].

The ablation study is especially informative. With **no anatomical prompts, no text prompts, and no D-MoE**, the Swin UNETR baseline gives **61.10%** mean DSC. Adding only anatomical prompts raises this to **63.47%**, a **+2.27%** improvement. Adding only text prompts gives **63.42%**. Adding only D-MoE gives **63.01%**. Combining all three yields **68.71%** [2503.14355].

The paper’s interpretation is precise. Anatomical prompts improve segmentation “by incorporating structural priors,” especially on datasets such as M-Li and M-Pa. Text prompts help strongly on LiTS but are less reliable for “fine-grained boundaries,” implying that semantic descriptions alone are insufficient for exact spatial extent. D-MoE alone improves performance, but “without domain priors, its effectiveness is constrained in highly variable datasets.” The strongest gain comes from the combination of prompt priors and adaptive experts.

## 6. Position within prompt-based medical imaging and remaining limitations

MAST-Pro belongs to a broader shift in medical imaging toward prompt-based conditioning, but its formulation is more specific than many earlier systems. Related work includes click- and box-driven promptable 3D anatomy segmentation in CT-SAM3D [2403.15063], automatic prompt generation for 3D multi-organ CT segmentation in AutoProSAM [2308.14936], and single-point-prompt 3D tumor segmentation in ProMISe [2310.19721]. Within dual-prompt segmentation, CAT coordinates **anatomical prompts derived from 3D cropped images** with **textual prompts enriched by medical domain knowledge** for multi-organ and tumor segmentation [2406.07085]. A broader survey frames such developments as part of a wider prompt-engineering landscape spanning textual, visual, learnable, and multimodal prompts in medical imaging [2507.01055].

Within that landscape, MAST-Pro is distinctive because its prompts are both **knowledge-driven** and **anatomically grounded**, and because prompt conditioning is coupled to **dynamic expert selection** rather than only to a monolithic decoder. The method therefore redefines prompting for pan-tumor segmentation as the joint use of semantic tumor priors and organ-context priors inside an adaptive expert architecture [2503.14355].

The paper also leaves several limitations and ambiguities. It does not fully specify how the LLM-generated prompt contents are validated or whether prompt wording materially affects performance. It does not detail whether the anatomical masks correspond to the hosting organ only, multiple nearby organs, or full-body segmentation maps from TotalSegmentator. The full combined training loss is not formalized as a single equation. PEFT is described conceptually, but not with a complete low-rank parameterization formula. Finally, prompt generalization depends on the reliability of upstream organ masks: because anatomy prompts come from TotalSegmentator-derived masks, failures in organ prompting can propagate into tumor segmentation. The paper also indicates that text prompts alone struggle with boundary precision, which suggests that prompt semantics are insufficient without strong spatial grounding.

Source: https://www.emergentmind.com/topics/promptanatomy