PromptAnatomy for Pan-Tumor Segmentation

• PromptAnatomy is a framework that uses structured text and spatial anatomical prompts to encode medical priors for enhanced pan-tumor segmentation.
• It employs a cross-attention mechanism within a D-MoE-enhanced SwinUNETR, combining knowledge-driven cues with efficient parameter fine-tuning.
• Empirical results indicate a 5.20% improvement in Dice Similarity Coefficient and a 91% reduction in trainable parameters, emphasizing robust clinical performance.

Searching arXiv for the primary paper and closely related prompt-based medical segmentation work.

(Meng et al., 18 Mar 2025)

Search arXiv for: (Meng et al., 18 Mar 2025)

Search arXiv for: "MAST-Pro Dynamic Mixture-of-Experts for Adaptive Segmentation of Pan-Tumors with Knowledge-Driven Prompts"

MAST-Pro—short for Mixture-of-experts for Adaptive Segmentation of pan-Tumors with knowledge-driven Prompts—is a framework for pan-tumor segmentation, meaning a single model that segments tumors arising in multiple organs and anatomical regions rather than being trained separately for one tumor type at a time. It is designed for the setting in which tumors vary substantially in shape, texture, intensity, borders, and anatomical context, while available datasets are strongly imbalanced across tumor types. The framework combines knowledge-driven text prompts, anatomical prompts derived from organ masks, a Dynamic Mixture-of-Experts (D-MoE) built into a SwinUNETR backbone, and Parameter-Efficient Fine-Tuning (PEFT) to improve segmentation accuracy without full-model adaptation (Meng et al., 18 Mar 2025).

1. Clinical and technical setting

MAST-Pro is motivated by a dual requirement. Clinically, cancer presents across many organs, and robust segmentation is needed for diagnosis, treatment planning, and monitoring. Technically, a universal tumor model must preserve features shared across tumors while also retaining cues that are highly specific to tumors in the liver, pancreas, colon, kidney, and other sites. The paper argues that generic foundation-model features alone are insufficient because they do not explicitly encode the medical priors that radiologists use implicitly, especially knowledge about where tumors occur, what organ context they inhabit, what shapes and border characteristics are typical, and how these properties manifest under a given imaging modality (Meng et al., 18 Mar 2025).

The paper positions earlier prompt-based approaches in two broad groups. Vision-prompt-driven systems such as Med-SAM3D and SegVol use prompts like points or boxes; the paper argues that these methods are annotation-dependent and do not inject enough anatomical or radiological prior knowledge to handle tumor heterogeneity. Text-prompt-driven systems such as the Universal Model and ZePT align image and text features, but often rely on predefined or relatively simple templates that, in the authors’ view, do not adequately capture “the extensive variability in tumor morphology and radiological presentation.” MAST-Pro is therefore framed as a response to three limitations: limited incorporation of medical priors, an imbalance between generic and tumor-specific features, and high computational costs for clinical adaptation (Meng et al., 18 Mar 2025).

A central point in this formulation is that MAST-Pro is not presented as a generic prompting layer attached to an otherwise unchanged segmentation model. Its prompting scheme is intended to encode domain-specific priors, while its expert-routing mechanism is intended to allocate representation capacity adaptively across tumor types. This suggests that the model treats tumor variability as both a conditioning problem and a capacity-allocation problem.

2. Knowledge-driven prompt construction

MAST-Pro uses two explicit prompt families: text prompts and anatomical prompts. The text prompts are structured, knowledge-driven descriptions generated with an LLM using a standardized medical template rather than free-form text. The prompt is defined as

$$P_{txt} = \text{``This is a [C] in the [O], appearing as a [S] mass with [E] borders on [M].''}$$

where $[C]$, $[O]$, $[S]$, $[E]$, and $[M]$ denote the tumor category/type, organ or anatomical location, shape descriptor, edge/border characteristic, and imaging modality, respectively (Meng et al., 18 Mar 2025).

This design is important because the prompt is not merely a class label. It bundles disease identity, anatomical site, morphology, boundary appearance, and modality-specific presentation into a reusable schema. The text prompt for tumor task $t$ is encoded by a pretrained text encoder $\mathcal{E}_{txt}$, cited to CLIP:

$E_{txt}^t = \mathcal{E}_{txt}(P_{txt}^t)$

where $E_{txt}^t$ is the resulting embedding. The symbol $[C]$0 indexes the tumor type or segmentation task (Meng et al., 18 Mar 2025).

The second prompt family is the anatomical prompt. Rather than representing anatomy only as text, MAST-Pro uses organ masks generated by TotalSegmentator. These masks are passed through a pretrained anatomical encoder $[C]$1, cited to SwinUNETR:

$[C]$2

where $[C]$3 denotes the anatomical prompt and $[C]$4 its embedding (Meng et al., 18 Mar 2025).

This is a distinctive design choice. The anatomical prompt is a spatially grounded prior in mask form rather than a symbolic organ label. The paper describes this as more informative than a pure organ name because it provides geometric localization and shape context. The prompt sources themselves are largely static and prior-driven: text descriptions are generated according to a fixed schema, organ masks come from TotalSegmentator, and both encoders are pretrained. Their embeddings, however, become dynamic participants in downstream attention and expert routing.

3. Prompt-conditioned segmentation architecture

MAST-Pro is built on a D-MoE-enhanced SwinUNETR. Multi-anatomical CT images are processed by the transformer-based segmentation backbone, but prompt information is injected through cross-attention and residual fusion. The paper states that the extracted prompts act as queries, while image features act as keys and values:

$[C]$5

where $[C]$6 and $[C]$7 are the text and anatomy prompt queries, $[C]$8 and $[C]$9 are the key and value projections of image features, and $[O]$0 is the scaling dimension (Meng et al., 18 Mar 2025).

In this formulation, the prompts do not merely concatenate with image features. They actively query the visual representation. The paper interprets this as a way for anatomical prompts to bias attention toward the relevant organ region and surrounding tissue context, while text prompts bias attention toward lesion appearance consistent with the described tumor morphology and modality. The attention-refined features are then fused with the image features and decoded through SwinUNETR:

$[O]$1

where $[O]$2 denotes image features and $[O]$3 is the SwinUNETR decoder (Meng et al., 18 Mar 2025).

The framework also introduces a mask-proposal branch. According to the paper, $[O]$4 undergoes global average pooling and is concatenated with text prompts, then passed to an MLP to generate an initial mask proposal $[O]$5, supervised by cross-entropy with tumor category information. The decoder output is then modulated by this proposal:

$[O]$6

The notation is somewhat underspecified in the paper, but the intended behavior is explicit: prompt-conditioned features produce a task-aware proposal, and that proposal guides the final segmentation. The paper states that segmentation is refined with Dice loss, while the proposal branch is supervised with cross-entropy for tumor category prediction (Meng et al., 18 Mar 2025).

4. Dynamic mixture-of-experts and parameter-efficient adaptation

The Dynamic Mixture-of-Experts (D-MoE) is the other central component of MAST-Pro. Within the SwinUNETR block, the paper introduces two routing perspectives: a generalized router $[O]$7 for shared low-rank experts and a tumor-specific router $[O]$8 for selecting a combination of generic and tumor-specific experts. The adapted feature is defined as

$[O]$9

where $[S]$0 is the incoming feature for task $[S]$1, $[S]$2 is the adapted feature, $[S]$3 is the routing weight for the $[S]$4-th selected expert, $[S]$5 is a tumor-specific expert, and $[S]$6 is a generic expert (Meng et al., 18 Mar 2025).

The routing weights are computed by sparse top-$[S]$7 routing:

$[S]$8

with

$[S]$9

This yields sparse expert activation, so only a few experts are used per instance or task. The paper states that the purpose is to preserve “foundational morphological and intensity patterns” while adapting to the peculiarities of each tumor type (Meng et al., 18 Mar 2025).

The model’s PEFT strategy is tied directly to D-MoE. Instead of fully fine-tuning the SwinUNETR backbone, MAST-Pro first uses large-scale medical pretraining and then fine-tunes by updating only a small subset of low-rank experts and routers. The paper further specifies that in D-MoE, $[E]$0 experts encode generic tumor features and $[E]$1 experts capture domain-specific variations, while routers select the top $[E]$2 from both groups (Meng et al., 18 Mar 2025).

The efficiency claims are concrete. MAST-Pro reports 21.04M trainable parameters, compared with 244.80M for Universal Model, 495.10M for ZePT, 449.08M for SegVol, and 363.68M for MA-SAM. The paper states this corresponds to a 91.04% reduction in trainable parameters without accuracy loss. Reported GPU memory usage is 8961.30 MB, below Universal Model (9710.55 MB), SegVol (19898.00 MB), ZePT (24705.40 MB), and MA-SAM (74544.02 MB) (Meng et al., 18 Mar 2025).

5. Experimental program and empirical findings

The training and evaluation protocol is broad in anatomical coverage. For pretraining, the paper lists BTCV, CT-ORG, Pancreas-CT, CHAOS, 3D-IRCADb, WORD, AMOS, AbdomenCT-1K, LiTS, KiTS, and CT images from MSD. For pan-tumor training and evaluation, the authors curate 2,000+ tumor cases from eight tumor-specific datasets: MSD-Liver (M-Li), MSD-Lung (M-Lu), MSD-Pancreas (M-Pa), MSD-HepaticVessel Tumor (M-HT), MSD-Colon (M-Co), LiTS, KiTS, and AbdomenCT-1K. All scans are resampled to $[E]$3, cropped to tumor-relevant regions, and trained with $[E]$4 patches. The evaluation metric is Dice Similarity Coefficient (DSC) (Meng et al., 18 Mar 2025).

Quantitatively, MAST-Pro achieves the best overall mean DSC of 68.71%, compared with 63.51% for ZePT, 62.79% for Universal Model, 60.84% for SegVol, and 61.10% for Swin UNETR. The paper highlights a 5.20% average DSC improvement over ZePT. Dataset-specific results include 59.34% on M-Pa, 74.76% on M-HT, 82.12% on LiTS, and 72.99% on KiTS, with top performance on six of eight datasets (Meng et al., 18 Mar 2025).

The ablation study is especially informative. With no anatomical prompts, no text prompts, and no D-MoE, the Swin UNETR baseline gives 61.10% mean DSC. Adding only anatomical prompts raises this to 63.47%, a +2.27% improvement. Adding only text prompts gives 63.42%. Adding only D-MoE gives 63.01%. Combining all three yields 68.71% (Meng et al., 18 Mar 2025).

The paper’s interpretation is precise. Anatomical prompts improve segmentation “by incorporating structural priors,” especially on datasets such as M-Li and M-Pa. Text prompts help strongly on LiTS but are less reliable for “fine-grained boundaries,” implying that semantic descriptions alone are insufficient for exact spatial extent. D-MoE alone improves performance, but “without domain priors, its effectiveness is constrained in highly variable datasets.” The strongest gain comes from the combination of prompt priors and adaptive experts.

6. Position within prompt-based medical imaging and remaining limitations

MAST-Pro belongs to a broader shift in medical imaging toward prompt-based conditioning, but its formulation is more specific than many earlier systems. Related work includes click- and box-driven promptable 3D anatomy segmentation in CT-SAM3D (Guo et al., 2024), automatic prompt generation for 3D multi-organ CT segmentation in AutoProSAM (Li et al., 2023), and single-point-prompt 3D tumor segmentation in ProMISe (Li et al., 2023). Within dual-prompt segmentation, CAT coordinates anatomical prompts derived from 3D cropped images with textual prompts enriched by medical domain knowledge for multi-organ and tumor segmentation (Huang et al., 2024). A broader survey frames such developments as part of a wider prompt-engineering landscape spanning textual, visual, learnable, and multimodal prompts in medical imaging (Yang et al., 28 Jun 2025).

Within that landscape, MAST-Pro is distinctive because its prompts are both knowledge-driven and anatomically grounded, and because prompt conditioning is coupled to dynamic expert selection rather than only to a monolithic decoder. The method therefore redefines prompting for pan-tumor segmentation as the joint use of semantic tumor priors and organ-context priors inside an adaptive expert architecture (Meng et al., 18 Mar 2025).

The paper also leaves several limitations and ambiguities. It does not fully specify how the LLM-generated prompt contents are validated or whether prompt wording materially affects performance. It does not detail whether the anatomical masks correspond to the hosting organ only, multiple nearby organs, or full-body segmentation maps from TotalSegmentator. The full combined training loss is not formalized as a single equation. PEFT is described conceptually, but not with a complete low-rank parameterization formula. Finally, prompt generalization depends on the reliability of upstream organ masks: because anatomy prompts come from TotalSegmentator-derived masks, failures in organ prompting can propagate into tumor segmentation. The paper also indicates that text prompts alone struggle with boundary precision, which suggests that prompt semantics are insufficient without strong spatial grounding.