MPEF-Net: Hierarchical Prompt Fusion

• The paper demonstrates that multi-level prompt embedding improves visual tasks by fusing low-, mid-, and high-level features through hierarchical cross-attention.
• The methodology utilizes level-specific learnable queries and prompt-conditioned fusion blocks to integrate semantic guidance into deep visual representations.
• Empirical evaluations reveal notable performance gains in metrics like SRCC, PLCC, Dice, and HD95, validating the approach's effectiveness.

Multi-Level Prompt-Embedded Fusion Network (MPEF-Net) denotes a class of architectures in which prompt-derived semantics are injected into visual processing at multiple representational levels rather than being applied only at the output stage. In the 2025 literature, the designation appears in two closely related but not identical forms: as a named network for blind text-to-image correspondence assessment based on CLIP and hierarchical query fusion, and as an architectural reading of a CLIP-guided 3D U-Net for brain tumor segmentation that fuses pixel-level, feature-level, and semantic-level information (Meng et al., 23 Jul 2025, Zhang, 14 Jul 2025). Across these usages, the central premise is that low-level detail, mid-level structure, and high-level prompt semantics should be coupled explicitly inside the model.

1. Scope and definitional usage

The most explicit usage of MPEF-Net occurs in "Hierarchical Fusion and Joint Aggregation: A Multi-Level Feature Representation Method for AIGC Image Quality Assessment" (Meng et al., 23 Jul 2025). There, MPEF-Net is the task-specific network for text-to-image correspondence assessment within a broader three-stage paradigm consisting of multi-level feature extraction, hierarchical fusion, and joint aggregation. A second usage appears in "A Brain Tumor Segmentation Method Based on CLIP and 3D U-Net with Cross-Modal Semantic Guidance and Multi-Level Feature Fusion" (Zhang, 14 Jul 2025), where the model is described as viewable as a Multi-Level Prompt-Embedded Fusion Network because CLIP-derived semantic prompts modulate a 3D U-Net at several levels.

This suggests that MPEF-Net is best understood not as a single fixed topology, but as a recurrent architectural pattern centered on hierarchical prompt-conditioned fusion.

Work	Task	Prompt/fusion formulation
(Meng et al., 23 Jul 2025)	T2I correspondence assessment	Prompt-Embedded Fusion Block at each CLIP feature level
(Zhang, 14 Jul 2025)	Brain tumor MRI segmentation	Pixel-, feature-, and semantic-level fusion with CLIP guidance
(Han et al., 2023)	Medical image segmentation	Progressive text prompts with multiscale feature fusion
(Dong et al., 2023)	RGB-D/T semantic segmentation	Multi-level multimodal prompts injected into a frozen backbone
(Liang et al., 2022)	Multimodal fusion in frozen PLMs	Prompt vectors as modular fusion interface

2. Core architectural principle

The shared logic of MPEF-Net is hierarchical prompt conditioning. Rather than relying on a single pooled visual embedding or a single late fusion layer, the architecture exposes multiple internal feature levels and conditions each level on prompt semantics. In the AIGC formulation, the levels are extracted from the 3rd, 6th, 9th, and 12th Transformer blocks of a CLIP-B/32 image encoder, and each level is processed by a Prompt-Embedded Fusion Block. In the brain-tumor formulation, the hierarchy is explicitly organized as pixel-level preprocessing, feature-level volumetric modeling, and semantic-level CLIP fusion (Meng et al., 23 Jul 2025, Zhang, 14 Jul 2025).

A compact formalization appears in the Prompt-Embedded Fusion Block. Let $Q_i$ denote level-specific learnable queries, $P$ the projected prompt-token sequence, and $G_i$ the image-token sequence at level $i$. The per-level refinement is

$Q'_i = \text{CrossAttention}(Q_i, P, P) + Q_i$

$$Q''_i = \text{CrossAttention}(Q'_i, G_i, G_i) + Q'_i$$

$\widetilde{Q}_i = \text{FFN}(Q''_i) + Q''_i$

This ordering is significant: prompt semantics are embedded into the query state before visual matching. The resulting $\widetilde{Q}_i$ is therefore not merely a visual summary, but a prompt-conditioned visual representation.

The brain-tumor variant instantiates the same idea through a different fusion operator. There, CLIP-derived vision and text semantics are mapped into a shared space and combined by a gating mechanism,

$G_{text} = \sigma(F_{text\_mapped})$

$$F_{combined} = F_{vision\_mapped} \odot G_{text} + F_{text\_mapped} \odot (1-G_{text})$$

followed by transformer-style self-attention to yield $P$0. That fused semantic code is then injected globally at the bottleneck and locally at decoder and output stages. A plausible unifying abstraction is that MPEF-Net implements a learned function $P$1 repeatedly across depth rather than once at the end.

3. MPEF-Net for text-to-image correspondence assessment

In the AIGC IQA setting, MPEF-Net targets blind evaluation of text-to-image correspondence as a scalar Mean Opinion Score regression problem. The image is processed by a CLIP-B/32 image encoder with 12 Transformer blocks, and multi-level token sequences are taken from blocks 3, 6, 9, and 12. The prompt is processed by the CLIP text encoder, but only the final-layer token sequence is used; this sequence is projected into the same embedding dimension as the image tokens so that prompt semantics remain token-resolved rather than being collapsed into a single pooled vector (Meng et al., 23 Jul 2025).

At each feature level, level-specific learnable queries are refined first by cross-attention to prompt tokens and then by cross-attention to visual tokens. The four level-wise outputs are concatenated along the token dimension, globally average pooled, and passed to an MLP regression head to predict correspondence MOS. The training configuration reported for this model uses AdamW with learning rate $P$2, weight decay $P$3, batch size 16, and 30 epochs, while CLIP components are described as pretrained and largely frozen, with projection layers and Prompt-Embedded Fusion Blocks trainable.

The ablation results make the role of prompt embedding and multi-level fusion explicit. On AGIQA-3K correspondence assessment, the full model reports SRCC 0.8410 and PLCC 0.8968, whereas removing prompt embedding reduces the scores to 0.7849 and 0.8710. Using only the final visual level yields 0.8258 and 0.8869. The query budget is also nontrivial: 8 learnable queries outperform 4, with 0.8410 SRCC and 0.8968 PLCC versus 0.8363 and 0.8902. On AIGCIQA2023 correspondence assessment, the same model reports SRCC 0.7764 and PLCC 0.7649, exceeding the listed PSCR and AMFF-Net baselines on that benchmark. A common misconception is that prompt-aware CLIP scoring is equivalent to computing a final-layer cosine similarity; this model instead uses token-level prompt cross-attention at four separate depths.

4. Prompt-embedded multi-level fusion in brain tumor segmentation

In brain tumor segmentation, the MPEF-Net interpretation arises from a three-layered architecture for automatic segmentation of BraTS 2020 MRI volumes into whole tumor, tumor core, and enhancing tumor. The model combines pixel-level preprocessing, a feature-level 3D U-Net backbone, and semantic-level CLIP guidance. The 3D U-Net encoder has four blocks with channels increasing from 32 to 512, residual blocks with 3D convolutions, CBAM attention, pooling, symmetric upsampling in the decoder, skip connections, deep supervision, and region-specific heads (Zhang, 14 Jul 2025).

The semantic pathway addresses the mismatch between 3D MRI and CLIP’s 2D visual encoder by a 3D-2D semantic bridging mechanism. Multi-modal MRI volumes are projected into axial, coronal, and sagittal planes, center slices are extracted, processed by a light 2D network, resized to $P$4, and passed through a pre-trained CLIP ViT-B/32 visual encoder. The resulting view-specific [CLS] features are averaged,

$P$5

to form a global visual semantic embedding. Text prompts describing tumor subregions, modality characteristics, and clinical attributes are encoded by the CLIP text encoder, mapped together with $P$6 into a shared semantic space, and fused through the gating mechanism above. The final semantic representation $P$7 conditions the bottleneck, decoder, and segmentation heads. At the decoder/output level, semantic attention is region-specific, for example

$P$8

with an analogous expression for tumor core.

On BraTS 2020, the full model reports Avg Dice 0.8567 and Avg HD95 3.0972, versus 0.8088 and 4.8528 for a traditional 3D U-Net. The enhancing-tumor Dice is 0.8005 versus 0.7277, and ET HD95 is 3.1638 versus 5.7379. Removing pixel fusion lowers Avg Dice to 0.8435; removing semantic fusion lowers it to 0.8433 and reduces ET Dice to 0.7815. A separate semantic-fusion ablation shows that removing semantic guidance produces only a small Dice decrease, from 0.8567 to 0.8532, but worsens Avg HD95 from 3.0972 to 4.0712 and ET HD95 from 3.1638 to 4.7239, indicating a specific contribution to boundary precision.

5. Relation to adjacent prompt-fusion research

MPEF-Net sits at the intersection of prompt learning, multimodal fusion, and hierarchical feature representation. Several prior lines of work supply architectural precedents. "Modular and Parameter-Efficient Multimodal Fusion with Prompting" uses prompt vectors inside a frozen pretrained LLM as the trainable fusion interface across modalities, with only about 15K trainable parameters in the cited VQAv2 setting (Liang et al., 2022). "Efficient Multimodal Fusion via Interactive Prompting" reports a modular multimodal fusion method with less than 3% trainable parameters and up to 66% saving of training memory usage by adding prompt vectors only on deep layers of unimodal transformers (Li et al., 2023).

On the dense-prediction side, "Multiscale Progressive Text Prompt Network for Medical Image Segmentation" combines a Prior Prompt Encoder, a Multiscale Feature Fusion Block, and UpAttention so that text prior prompts shape segmentation progressively across scales (Han et al., 2023). "Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning" introduces a Multimodal Prompt Generator and Multimodal Feature Adapter that inject multi-level multimodal prompts into a frozen SegFormer MiT backbone; only 3.88M parameters are added for backbone adaptation and 7.15M trainable parameters are used in total (Dong et al., 2023). Even non-prompt architectures such as "Multi-level feature fusion network combining attention mechanisms for polyp segmentation" reinforce the same multi-level thesis by showing that structured shallow/deep/global fusion is superior to unfiltered aggregation (Liu et al., 2023).

These works do not define MPEF-Net as a unified standard, but they delineate its enabling components: prompt-conditioned alignment, multi-level extraction, lightweight adapters or queries, and fusion mechanisms that are level-aware rather than purely late-fusion.

6. Empirical patterns, misconceptions, and open directions

Two empirical regularities recur across the reported MPEF-Net formulations. First, multi-level fusion consistently outperforms single-level or partially fused alternatives. In text-to-image correspondence assessment, the single-level last-block variant underperforms the full four-level model. In brain tumor segmentation, removing pixel or semantic fusion degrades Dice and HD95, and traditional 3D U-Net lags substantially behind the full fusion model (Meng et al., 23 Jul 2025, Zhang, 14 Jul 2025).

Second, prompt semantics appear most useful when they are operationalized through explicit gating or attention rather than naive concatenation. In the AIGC model, prompt tokens first reshape the learnable queries and only then guide visual extraction. In the segmentation model, prompt semantics modulate bottleneck channels and decoder-space attention maps, with the largest reported gains concentrated in clinically critical ET delineation. A common misconception is therefore that MPEF-Net simply means “adding text to vision”; the published formulations are more specific, requiring hierarchical insertion points and task-aware fusion operators.

The present literature also reveals that MPEF-Net is still a moving target. One branch uses CLIP token hierarchies for no-reference quality regression; another uses CLIP to inject semantic priors into 3D medical segmentation. This suggests that the term currently functions more as an architectural schema than as a single benchmark-standardized model family. The future directions proposed or implied in the cited works converge on richer prompt engineering, stronger domain-specific text encoders, explicit alignment losses, and interpretability of attention or gating trajectories. A plausible implication is that subsequent MPEF-Net variants will increasingly treat prompts as first-class structured inputs whose effect is distributed across depth, scale, and task-specific heads rather than confined to a single multimodal fusion layer.