MP-Decoder in Co-Seg: Mutual Prompt Mask Decoder

Updated 4 July 2026

The MP-Decoder is a cross-guided transformer module that jointly decodes tissue and nuclei segmentation by leveraging mutual prompt inputs from preliminary masks.
It uses cross-attention where tissue queries attend to nuclei-enhanced features and vice versa, ensuring bidirectional guidance between the two segmentation tasks.
Empirical ablations in Co-Seg show that incorporating MP-Decoder improves segmentation metrics, evidencing its effectiveness in joint semantic and instance segmentation.

Mutual Prompt Mask Decoder (MP-Decoder) is the decoder module introduced in Co-Seg for joint tissue semantic segmentation and nuclei instance segmentation in histopathology images. It is designed for a co-segmentation setting in which tissue regions and nuclei instances are treated as inherently correlated, and it operationalizes that assumption by decoding the two tasks collaboratively rather than with fully independent heads. In Co-Seg, MP-Decoder receives a shared image embedding, two region-aware prompts derived from preliminary task masks, and two sets of task-specific queries, then uses cross-guided attention to produce final tissue and nuclei masks (Xu et al., 8 Sep 2025).

1. Origin, role, and conceptual basis

MP-Decoder appears in the Co-Seg framework, whose stated objective is collaborative tissue and nuclei segmentation in histopathology. The paper motivates the module by arguing that “accurately identifying nuclei provides valuable cues for understanding the underlying tissue structures, while tissue segmentation can aid in localizing nuclei.” It further contrasts this with pipelines that use two separate networks or a shared encoder with separated decoders, which “decouple the parameter space of semantic and instance segmentation,” weaken interdependence, and yield suboptimal predictions (Xu et al., 8 Sep 2025).

Within that framework, MP-Decoder is the mechanism that realizes what the paper calls “mutual prompt-guided collaborative learning” and “cross-guidance.” “Mutual prompt-guided” means that each task is not decoded only from its own queries or its own prompt. Instead, tissue decoding is guided by the nuclei prompt, and nuclei decoding is guided by the tissue prompt. “Cross-guidance” is implemented inside the decoder rather than as a loss-only coupling or post-hoc fusion step (Xu et al., 8 Sep 2025).

The broader probabilistic intuition is also made explicit. Co-Seg presents joint prediction as

$p(y_1, y_2 \mid x, \theta_1, \theta_2) = p(y_1 \mid x, \theta_1)\cdot p(y_2 \mid y_1, x, \theta_2) = p(y_2 \mid x, \theta_2)\cdot p(y_1 \mid y_2, x, \theta_1),$

where $y_1$ denotes tissue semantic segmentation and $y_2$ denotes nuclei instance segmentation. This formulation is used to justify bidirectional conditioning between the tasks, and MP-Decoder is presented as the architectural path through which that conditioning is implemented (Xu et al., 8 Sep 2025).

2. Placement in the Co-Seg pipeline

Co-Seg uses a shared image encoder, specifically a Hiera ViT initialized from SAM2-L, to extract an image embedding $z$ . The decoding pipeline then proceeds in two forwards. In the first forward, the decoder predicts preliminary tissue and nuclei masks without prompts. These coarse mask logits are passed to the Region-aware Prompt Encoder (RP-Encoder), which converts them into two dense prompts: a tissue prompt $g_1$ and a nuclei prompt $g_2$ . In the second forward, MP-Decoder consumes $z$ , $g_1$ , $g_2$ , and two sets of task-specific query embeddings $q_1$ and $y_1$ 0 to generate the final outputs (Xu et al., 8 Sep 2025).

The paper specifies the decoder inputs as:

the shared image embedding $y_1$ 1,
the tissue prompt $y_1$ 2,
the nuclei prompt $y_1$ 3,
tissue queries $y_1$ 4,
nuclei queries $y_1$ 5.

Its stated outputs are tissue semantic segmentation masks and nuclei instance segmentation masks. The paper does not provide a finer decomposition of MP-Decoder outputs into object centers, contours, embeddings, or auxiliary logits; it only states that the framework produces final masks for the two tasks, and that nuclei supervision follows a standard instance loss combination used in prior work (Xu et al., 8 Sep 2025).

A notable implementation detail is that the tissue and nuclei heads share parameters across the two decoding stages. The design therefore consists of a prompt-free preliminary decoding pass, prompt generation by RP-Encoder, and a prompt-conditioned collaborative decoding pass, rather than a long recurrent refinement schedule. The paper explicitly states only two forwards and does not describe additional iterative rounds (Xu et al., 8 Sep 2025).

3. Cross-guided attention mechanism

Architecturally, MP-Decoder contains two task heads, one for tissue and one for nuclei. Each head begins with task-specific learned queries, denoted $y_1$ 6 and $y_1$ 7. These queries are first refined by self-attention independently, after which each task attends not to same-task prompt-enhanced features, but to the image embedding fused with the opposite task’s prompt:

$y_1$ 8

with $y_1$ 9, $y_2$ 0, and $y_2$ 1 denoting element-wise addition (Xu et al., 8 Sep 2025).

This equation defines the core asymmetry of the module. Tissue decoding uses $y_2$ 2, so the tissue query is updated from nuclei-aware features. Nuclei decoding uses $y_2$ 3, so the nuclei query is updated from tissue-aware features. After this step, an MLP is applied “to ensure the refined query embeddings are expressive,” although the paper does not specify the depth, hidden ratio, activation, or residual design of that MLP (Xu et al., 8 Sep 2025).

MP-Decoder then performs what the paper calls reverse cross-attention:

$y_2$ 4

This reverses the roles of the feature source and the queries. In the first attention step, the task queries attend to prompt-enhanced image features; in the reverse step, the prompt-enhanced image features attend to the refined task queries. The resulting $y_2$ 5 is therefore a task-specific image embedding conditioned on the opposite task’s prompt via the updated query tokens (Xu et al., 8 Sep 2025).

The paper characterizes the collaboration as occurring at the prompt level, query level, and feature level. This is significant because the two task heads are not independent branches sharing only low-level backbone features. Their interaction is embedded in decoder attention, query refinement, and feature conditioning. The paper’s own summary of the two directions of cross-guidance is that “nuclei context prompts can enhance tissue segmentation consistency, while tissue boundary feedback provides prior constraints for nuclei discrimination” (Xu et al., 8 Sep 2025).

4. Region-aware prompts, outputs, and supervision

MP-Decoder depends on RP-Encoder for prompt construction. RP-Encoder takes task mask logits $y_2$ 6 and transforms them into dense region-aware prompts using

$y_2$ 7

$y_2$ 8

The paper specifies that $y_2$ 9 is a $z$ 0 convolution for downsampling and $z$ 1 is a $z$ 2 convolution for channel expansion, with LayerNorm and GELU in between. Functionally, $z$ 3 is derived from tissue semantic mask logits and $z$ 4 from nuclei instance mask logits, and both are used as prior constraints in the second decoding forward (Xu et al., 8 Sep 2025).

After reverse cross-attention produces $z$ 5 and $z$ 6, each head follows “the standard SAM” design and adopts a pixel decoder to upsample the refined task-specific image embedding. Final masks are then produced by taking the dot product between the upscaled image embeddings and the corresponding task queries. The paper therefore places MP-Decoder within a SAM-style mask decoding paradigm, but does not expose the internal architecture of the pixel decoder, the exact upsampling path, the number of transformer blocks, the number of attention heads, the hidden dimensionality, the number of queries, or the prompt dimensionality (Xu et al., 8 Sep 2025).

Supervision combines prompt-related prior losses and final segmentation losses. Tissue segmentation uses cross-entropy loss plus Dice loss. Nuclei segmentation uses a standard combination loss of Focal loss, Dice loss, MSE loss, and MSGE loss following HoverNet and CellViT. The balancing coefficients are $z$ 7 and $z$ 8. The paper does not introduce a separate explicit consistency loss between tissue and nuclei outputs beyond this joint formulation (Xu et al., 8 Sep 2025).

An important boundary on interpretation is that the exact target representation of the nuclei branch is not specified in the MP-Decoder subsection itself. The loss combination strongly suggests multi-map instance supervision compatible with prior nuclei instance segmentation systems, but the paper does not map each loss term to an explicit sub-head or expose whether the branch predicts center maps, contour maps, or horizontal/vertical offsets (Xu et al., 8 Sep 2025).

5. Empirical evidence and implementation scope

The principal evidence for MP-Decoder comes from Co-Seg’s ablation study on PUMA. A baseline without RP-Encoder, MP-Decoder, or the co-segmentation paradigm achieves 90.98 Dice / 84.71 mIoU / 287.19 HD for tissue and 76.35 F1 / 73.18 Precision / 82.27 Recall / 66.59 AJI for nuclei. Adding RP-Encoder alone yields 91.23 Dice and 77.01 F1. Adding both RP-Encoder and MP-Decoder raises this to 91.76 Dice / 86.13 mIoU / 246.56 HD and 78.19 F1 / 74.91 Precision / 82.54 Recall / 67.93 AJI. Enabling the full co-segmentation paradigm further improves performance to 92.51 Dice / 87.18 mIoU / 206.72 HD and 79.70 F1 / 77.05 Precision / 83.61 Recall / 69.14 AJI (Xu et al., 8 Sep 2025).

These ablations are the most direct quantitative evidence for MP-Decoder itself. Relative to RP-Encoder alone, introducing MP-Decoder produces approximately +0.53 Dice for tissue and +1.18 F1 for nuclei. The paper interprets this as evidence that decoder-level cross-guidance improves both tasks, beyond prompt construction alone (Xu et al., 8 Sep 2025).

At the full-system level, Co-Seg reports 92.51 Dice for tissue semantic segmentation, 79.70 F1 for nuclei instance segmentation, and the best panoptic metrics on both tissue and nuclei. The qualitative description states that, “benefiting from the mutual reinforcement of semantic and instance segmentation tasks, our Co-Seg can delineate precise tissue regions and segment accurate nuclei, containing fewer false positives.” While that statement refers to the overall framework rather than MP-Decoder in isolation, it is aligned with the decoder’s intended function (Xu et al., 8 Sep 2025).

The implementation details that are explicitly provided are limited but nontrivial. Co-Seg uses Hiera ViT initialized from SAM2-L, and fine-tunes via LoRA and Adapter inserted into attention and FFN layers. Training uses Adam with learning rate $z$ 9, decay factor 0.98, batch size 16, and 300 epochs. The additional learnable parameters introduced by Co-Seg are 9.1\% over the baseline. At the same time, many decoder-specific hyperparameters remain unspecified in the paper text, so faithful re-implementation requires code or supplementary material beyond the main description (Xu et al., 8 Sep 2025).

The novelty of MP-Decoder is not merely that prompts are used in segmentation. Its novelty, as described in Co-Seg, is that task-generated prompts act as reciprocal priors during decoding: tissue queries attend to nuclei-prompt-enhanced image features, and nuclei queries attend to tissue-prompt-enhanced image features. This differs from independent task decoders that predict each task from image features and same-task queries with little or no direct conditional coupling (Xu et al., 8 Sep 2025).

This distinction matters because adjacent literature uses related terms and mechanisms without implementing the same decoder form. “MP-Former: Mask-Piloted Transformer for Image Segmentation” modifies Mask2Former by feeding noised ground-truth masks and class embeddings into masked attention during training, but it is a training-only mask-piloted denoising auxiliary branch rather than an inference-time mutual prompt decoder (Zhang et al., 2023). “Prompt-Guided Mask Proposal for Two-Stage Open-Vocabulary Segmentation” injects text tokens into a mask proposal decoder via text-query cross-attention, yet the interaction is one-way—query attends to text—rather than reciprocal prompt exchange between segmentation tasks (Li et al., 2024). “Regressor-Segmenter Mutual Prompt Learning for Crowd Counting” implements bidirectional supervision between a regressor and a segmenter, but its mask-producing component is a conventional CNN head rather than a dedicated prompt-conditioned transformer decoder (Guo et al., 2023).

Later work also broadens the prompt–mask interaction space without matching MP-Decoder exactly. “PR-MaGIC: Prompt Refinement Via Mask Decoder Gradient Flow For In-Context Segmentation” uses gradients from SAM’s mask decoder to refine prompts at test time, making prompt quality depend on decoder feedback, but it does not define a jointly learned mutual prompt mask decoder (Lee et al., 13 Apr 2026). “DeCoDrift: Stabilizing Decoder Coupling in Closed-Loop Foundation Segmentation” studies iterative prompting as a closed-loop dynamical system and shows that prompt-conditioned mask decoding can drift over repeated feedback iterations, which is directly relevant whenever decoded masks become new prompts (Tabib et al., 25 May 2026).

A common misconception is to equate MP-Decoder with generic masked prompting or any prompt-conditioned mask generator. The Co-Seg module is more specific: it is a bidirectionally coupled decoder for two correlated segmentation tasks, with mutual prompt injection and reverse cross-attention embedded in the decoding graph itself (Xu et al., 8 Sep 2025). Another misconception is to treat it as a fully specified decoder family. The paper provides the main computation graph and training losses, but leaves several architectural details unreported, including decoder depth, hidden dimensions, attention head counts, and prompt dimensionality. A plausible implication is that MP-Decoder is best understood as a decoder principle—cross-task prompt-mediated collaborative decoding—rather than a completely parameterized standalone template (Xu et al., 8 Sep 2025).