Cross-Image Interaction Prompting Module

• The paper introduces cross-image interaction prompting modules that leverage learnable prompt banks, adaptive attention, and bidirectional fusion to improve image translation and segmentation tasks.
• It employs a shared encoder–decoder backbone with Transformer-based self-attention, enabling selective integration of modality-specific features while preserving structural details.
• Empirical results demonstrate significant performance gains, including higher PSNR, SSIM, and Dice scores compared to traditional concatenation or naive fusion methods.

A Cross-Image Interaction Prompting Module is a network component that enables explicit conditioning of feature transformations on information extracted from one or more reference images, with the aim of improving multi-modal synthesis or segmentation by leveraging complementary cues across images or modalities. Such modules have emerged in multi-task medical image translation and segmentation pipelines to address the limitations of conventional concatenation or naive fusion, which often fail to selectively highlight the semantically or structurally relevant content contained in distinct image streams. Current state-of-the-art approaches utilize learnable prompt banks, adaptive attention, and bidirectional fusion as means for content-aware and task-specific cross-image feature integration (Chen et al., 2023, Noh et al., 7 Sep 2025).

1. Architectural Foundations

In multi-modal medical image analysis, architectures leveraging cross-image interaction prompting generally adopt a shared encoder–decoder backbone, with Transformer blocks providing the global self-attention required for cross-region or cross-image context modeling. For example, MedPrompt (Chen et al., 2023) utilizes Self-adaptive Prompt Blocks (SPBs) between encoder and decoder, each operating on the output feature map $F \in \mathbb{R}^{C \times H \times W}$ and a bank of $N$ learnable prompt embeddings $P_c \in \mathbb{R}^{C \times H \times W}$. These blocks adaptively inject modality-specific information via a composite of prompt extraction (conditioned on the input features) and prompt fusion (via attention-enhanced integration), resulting in a modified feature map $\hat F$ that is specialized for the current translation task.

In segmentation, the Dual Interactive Fusion Module (DIFM) (Noh et al., 7 Sep 2025) combines encodings from an original image and its enhanced counterpart through staged feature merging, bidirectional cross-attention, and global spatial attention at each skip connection between encoder and decoder. This design maintains the semantic and fine-grained details unique to each stream while facilitating mutual enhancement via structured attention interactions.

2. Key Module Components

MedPrompt: Self-Adaptive Prompt Block (SPB)

The SPB consists of two cascaded submodules:

(a) Prompt Extraction Block (PEB):

Conditioned on current feature map $F$, PEB first applies global average pooling (GAP) to obtain a channel descriptor, then reduces this via a $1\times 1$ convolution and softmax to produce modality weights $w_i$. These weights are combined over the prompt bank: $$w = \mathrm{Softmax}(\mathrm{Conv}_{1\times1}(\mathrm{GAP}(F)))$$

$\widetilde{P} = \sum_{c=1}^N w_c P_c$

$P = \mathrm{Conv}_{3\times3}(\widetilde{P})$

(b) Prompt Fusion Block (PFB):

The refined prompt $N$0 is concatenated with the original feature $N$1 and passed through Restormer-style attention blocks: $N$2 where MDTA provides multi-head attention and GDFN supplies gated feed-forward transformation.

Dual Interactive Fusion Module (DIFM)

DIFM executes fusion between two semantically aligned feature streams (original and enhanced):

• Merge low- and high-level features from both streams via channel concatenation and convolution.
• Compute bidirectional cross-attention, using each stream in turn as queries and the other as keys/values: $N$3

$N$4

where $N$5.

• Apply per-branch FFN refinement with channel gating and skip connections.
• Perform global spatial attention (per-channel sigmoid after averaging) and sum the fused outputs.

3. Training Objectives and Optimization

For translation, MedPrompt employs a composite objective:

• Mean squared error ($N$6) to enforce pixel-level fidelity,
• Structural similarity ($N$7) to reward perceptual and anatomical integrity,
• Weighted total loss: $N$8 with $N$9 (Chen et al., 2023).

For segmentation, DIFM-based networks use:

• Weighted cross-entropy and Dice losses,
• An explicit boundary loss based on multi-scale gradient differences using Sobel operators,
• Total loss: $P_c \in \mathbb{R}^{C \times H \times W}$0 (Noh et al., 7 Sep 2025).

This combination ensures the network is sensitive both to overall structure and to fine boundary detail.

4. Empirical Performance and Comparative Analysis

MedPrompt demonstrates state-of-the-art results for cross-modal translation on five public datasets and four modality pairs, outperforming CycleGAN, Pix2Pix, UNIT, MUNIT, and other baselines in PSNR and SSIM across all tasks. Notably, on ADNI MRI→PET, MedPrompt achieves PSNR $P_c \in \mathbb{R}^{C \times H \times W}$1 dB and SSIM $P_c \in \mathbb{R}^{C \times H \times W}$2, compared to RIED-Net’s $P_c \in \mathbb{R}^{C \times H \times W}$3 dB and $P_c \in \mathbb{R}^{C \times H \times W}$4, respectively; on IXI T1→T2, the values are $P_c \in \mathbb{R}^{C \times H \times W}$5 dB and $P_c \in \mathbb{R}^{C \times H \times W}$6, surpassing all competitors (Chen et al., 2023).

DIFM yields consistently higher Dice scores and lower Hausdorff distances than naive concatenation or unidirectional fusion. Integrating DIFM into off-the-shelf models (SwinUnet+DIFM, FCT+DIFM) improves Dice by $P_c \in \mathbb{R}^{C \times H \times W}$7 (ACDC) and $P_c \in \mathbb{R}^{C \times H \times W}$8 (ACDC), and the full model reports ACDC Dice $P_c \in \mathbb{R}^{C \times H \times W}$9 and Synapse Dice $\hat F$0, with reduced HD95 from $\hat F$1 to $\hat F$2 (Noh et al., 7 Sep 2025).

Ablation analyses in MedPrompt indicate that removing the prompt extraction or fusion blocks causes a $\hat F$3 dB drop in PSNR and $\hat F$4 in SSIM; eliminating the Transformer itself leads to a $\hat F$5 dB PSNR loss, confirming the necessity of both dynamic prompt interaction and global feature modeling.

5. Mechanistic Advantages and Theoretical Implications

Cross-image interaction prompting modules, by adaptively extracting, weighting, and integrating modality- or stream-specific signal, address two primary limitations of conventional fusion mechanisms:

• Selective Modality Focus: The learnable prompt bank and adaptive softmax-weight extraction in MedPrompt allow a single shared model to steer its generative process towards any of $\hat F$6 target modalities, overcoming the rigidity of models that require per-pair training.
• Contextual and Boundary-Aware Fusion: DIFM’s bidirectional cross-attention and global spatial pooling ensure that salient cues—be they boundaries, edges, or deep semantic features—are communicated across streams while suppressing noise and irrelevant patterns.

This architecture not only enhances basic translation and segmentation accuracy but also supports robust generalization, as demonstrated by single-instance training across multiple modality pairs and competitive performance in both seen and missing-modality regimes.

6. Adaptability, Limitations, and Broader Applications

Cross-image interaction prompting can be straightforwardly extended to multi-modal or temporal vision problems. For example, tasks such as RGB-depth fusion, MRI-PET integration, video frame prompting, and 3D volumetric analysis can all leverage the same principles of adaptive cross-image feature exchange. In remote-sensing, pre-disaster and post-disaster images can serve as paired streams for change detection via cross-attention gating.

However, this approach typically requires spatial alignment (registration) between input streams, can be computationally and memory intensive due to dual encoders and attention blocks, and may require redesign for more than two input streams. A plausible implication is that lightweight variants or hierarchical prompt selection will be required for real-time or resource-constrained deployment scenarios.

7. Summary Table: Key Module Designs

Module	Cross-Image Inputs	Interaction Mechanism	Task Domain
SPB/MedPrompt	Bank of prompts + encoder features	Prompt extraction + fusion (Restormer attention)	Image translation (Chen et al., 2023)
DIFM	Original + enhanced encodings	Bidirectional cross-attention + spatial gating	Segmentation (Noh et al., 7 Sep 2025)

This encapsulates structural similarities and design differences in leading cross-image interaction prompting modules, highlighting their task-specific adaptations and broader cross-modal applicability.