Image-Prompt Embedding Fusion (IPEF)

• IPEF is a multimodal interface pattern that fuses image-derived representations with diverse prompt embeddings, enabling models to be both image-grounded and prompt-aware.
• It employs various fusion operators such as concatenation, attention pooling, and affine modulation to integrate visual and prompt signals, thereby boosting performance in tasks like segmentation and retrieval.
• IPEF optimization strategies involve freezing large foundation encoders and using task-specific loss functions to balance prompt guidance with image evidence, improving overall robustness.

Image-Prompt Embedding Fusion (IPEF) denotes a family of multimodal mechanisms in which image-derived representations are combined with prompt embeddings or prompt-conditioned control signals before prediction, scoring, or generation. In the cited literature, prompts are not restricted to natural language: they include bounding boxes, masks, learnable prompt tokens, prompt directional vectors, degradation descriptions, and pose-guided embeddings. The common objective is to make a model simultaneously image-grounded and prompt-aware, whether the downstream task is medical segmentation, AI-generated image quality assessment, image fusion, visual prompt tuning, retrieval, diffusion-based generation, or EEG-image alignment (Zhuo et al., 23 Jul 2025, Qu et al., 2024, Zhao et al., 2024, Xiao et al., 24 Jun 2026).

1. Definition, scope, and terminology

In the cited work, IPEF is best understood as an interface pattern rather than a single canonical block. The pattern appears whenever a model must reconcile image evidence with an external or internally generated prompt signal, and the fusion occurs at the embedding level rather than only at the decision layer. This interpretation covers dense prediction, retrieval, controllable fusion, diffusion conditioning, and parameter-efficient adaptation.

Setting	Prompt form	Representative fusion pathway
Medical segmentation	Bounding box prompts	Multiscale concatenation, SE gating, upsampling
AGI image quality assessment	Text prompt with special token	Attention pooling and cross-modality pooling
Vision-language image fusion	ChatGPT-generated descriptions	Text-to-image cross-attention
Frozen ViT adaptation	Learnable prompt tokens	Concatenation, addition, affine, or cross-attention
Controllable IR-VIS fusion	Mask prompts or degradation prompts	Cross-attention, mask gating, or FiLM-like modulation
Zero-shot composed retrieval	Prompt directional vector	Weighted fusion of composed text and image embeddings

This breadth matters because the term “prompt” is semantically overloaded. In FA-SAM, the prompt is a spatial prior for segmentation; in IP-IQA it is the textual condition whose compliance partly defines the MOS; in CtrlFuse it is a mask specifying semantic targets of interest; in ControlFusion it can be a user instruction about degradation type and severity; and in PDV it is a residual direction in a shared CLIP space (Zhuo et al., 23 Jul 2025, Qu et al., 2024, Sun et al., 12 Jan 2026, Tang et al., 30 Mar 2025, Tursun et al., 11 Feb 2025).

Nomenclature is also non-uniform. One hazy IR-VIS fusion paper explicitly states that it does not use the term “Image-Prompt Embedding Fusion,” mapping the concept instead to the combination of Prompt Generation Module, Prompt Embedding Block, and Multi-stage Prompt Embedding Fusion Module. The underlying operation, however, remains prompt-conditioned multi-stage fusion of visual features (Li et al., 2024).

2. Fusion operators and architectural motifs

A dense-prediction instantiation appears in FA-SAM. The module takes multiscale ViT-B image embeddings $E_{\mathrm{img}}^{(s)}$, spatializes prompt tokens into $P^{(s)}$, concatenates them as $Z^{(s)} = [E_{\mathrm{img}}^{(s)}; P^{(s)}]$, applies Squeeze-and-Excitation residual refinement, upsamples all scales to a common resolution, and maps the result to a high-resolution feature $F_{\mathrm{high}}$ by a final $\mathrm{Conv}_{1\times1}$ before the unchanged SAM mask decoder. The paper is explicit that no cross-attention is introduced in IPEF; robustness is instead obtained through multiscale image grounding, SE channel re-weighting, upsampling, and skip connections (Zhuo et al., 23 Jul 2025).

In IP-IQA, fusion is not spatial reconstruction but attention pooling. Visual patch embeddings provide keys and values, while either the visual global token $G_v$ or the textual $[QA]$ token $G_t$ serves as query, yielding a pure visual descriptor $z_{\mathrm{vis}}$ and a cross-modality descriptor $z_{\mathrm{xmod}}$; the fused representation is $P^{(s)}$0. FILM uses a different directionality: text-derived fused descriptions act as queries and image features act as keys and values inside stacked cross-attention blocks, so textual guidance modulates visual feature extraction rather than only the final scorer (Qu et al., 2024, Zhao et al., 2024).

Other systems implement IPEF as feature modulation rather than pooling. ControlFusion maps a prompt embedding $P^{(s)}$1 to per-channel affine parameters $P^{(s)}$2 and applies

$P^{(s)}$3

thereby letting degradation prompts alter restoration and fusion behavior directly. In frozen ViTs, layer-specific prompt fusion formalizes four operators—concatenation, addition, affine transformation, and cross-attention—and uses DARTS to choose a different fusion rule per layer at the pre-MSA injection point (Tang et al., 30 Mar 2025, Xiao et al., 24 Jun 2026).

Generative models reveal yet another architectural form. VCF trains a lightweight aligner $P^{(s)}$4 to map CLIP image tokens into the text-token manifold and then conditions Stable Diffusion on $P^{(s)}$5 or a cross-attentive alternative entirely at inference time. FPDM, by contrast, learns a global fusion embedding $P^{(s)}$6 from source image and target pose in CLIP space, then injects it into a latent diffusion model through the time-embedding path, source-token cross-attention, and additive pose conditioning (Żywot et al., 24 May 2026, Lee et al., 2024).

3. Prompt construction and semantic alignment

Prompt construction is often itself a learned subproblem. FA-SAM generates target-domain bounding boxes automatically through AGM and SUFM, using uncertainty-injected shallow features and BFS to keep the largest connected component before the boxes are sent to the frozen SAM prompt encoder. CtrlFuse similarly treats mask prompts as first-class semantic inputs: the Reference Prompt Encoder converts support/query multimodal features plus $P^{(s)}$7 into prompt embeddings $P^{(s)}$8 that drive frozen SAM masks, and the Prompt-Semantic Fusion Module then injects these embeddings back into IR-visible fusion features through cross-attention and mask gating (Zhuo et al., 23 Jul 2025, Sun et al., 12 Jan 2026).

Textual IPEF methods usually emphasize alignment between prompt and image representation spaces. IP-IQA performs an Image2Prompt incremental pretraining stage with

$P^{(s)}$9

freezes the text encoder except for a trainable $Z^{(s)} = [E_{\mathrm{img}}^{(s)}; P^{(s)}]$0 token replacing $Z^{(s)} = [E_{\mathrm{img}}^{(s)}; P^{(s)}]$1, and uses that token as a global textual query during fusion. ControlFusion learns an alternative alignment route: a spatial-frequency collaborative visual adapter produces $Z^{(s)} = [E_{\mathrm{img}}^{(s)}; P^{(s)}]$2 from degraded IR-VIS inputs, a CLIP text encoder produces $Z^{(s)} = [E_{\mathrm{img}}^{(s)}; P^{(s)}]$3 from user instructions, and Stage 1 minimizes

$Z^{(s)} = [E_{\mathrm{img}}^{(s)}; P^{(s)}]$4

so that prompting can be either user-specified or autonomous (Qu et al., 2024, Tang et al., 30 Mar 2025).

Several works use prompt embeddings as explicit directional or hierarchical control variables. PDV defines a prompt directional vector

$Z^{(s)} = [E_{\mathrm{img}}^{(s)}; P^{(s)}]$5

reuses it to form dynamic composed text and image embeddings, and then performs weighted embedding-level fusion for zero-shot composed retrieval. NeuroCLIP combines instance-level prompts produced by dynamic filtering with global visual prompt tokens $Z^{(s)} = [E_{\mathrm{img}}^{(s)}; P^{(s)}]$6 inserted before a frozen ViT, so that token-level content adaptation and dataset-level neural-aware prompting coexist in one encoder (Tursun et al., 11 Feb 2025, Wang et al., 12 Nov 2025).

FILM makes prompt construction explicitly multi-granular. BLIP2 image captions, GRIT dense captions, and SAM semantic masks are converted by ChatGPT into multi-sentence descriptions $Z^{(s)} = [E_{\mathrm{img}}^{(s)}; P^{(s)}]$7 and $Z^{(s)} = [E_{\mathrm{img}}^{(s)}; P^{(s)}]$8, encoded by a frozen BLIP2 text encoder, concatenated as $Z^{(s)} = [E_{\mathrm{img}}^{(s)}; P^{(s)}]$9, and then used to guide visual fusion across infrared-visible, medical, multi-exposure, and multi-focus image fusion (Zhao et al., 2024).

4. Optimization regimes and computational design

A recurrent engineering choice is to freeze large foundation encoders and train only the fusion interface. FA-SAM freezes the SAM ViT-B image encoder and prompt encoder and backpropagates through IPEF and the mask decoder using a combined cross-entropy plus Dice loss. IP-IQA freezes the CLIP text encoder except for the $F_{\mathrm{high}}$0 token, incrementally pretrains the image encoder with Image2Prompt, and then trains the fusion and regression heads with Adam for 100 epochs at learning rate $F_{\mathrm{high}}$1 on 80/20 splits repeated 10 times (Zhuo et al., 23 Jul 2025, Qu et al., 2024).

Loss design varies with task semantics. FILM uses unsupervised fusion objectives,

$F_{\mathrm{high}}$2

with task-specific settings such as IVF $F_{\mathrm{high}}$3 and MFF $F_{\mathrm{high}}$4. ControlFusion uses a prompt-regulated restoration-and-fusion objective

$F_{\mathrm{high}}$5

so the prompt affects both features and optimization priorities (Zhao et al., 2024, Tang et al., 30 Mar 2025).

When the fusion policy itself is a variable, optimization becomes bi-level. The layer-specific ViT framework relaxes per-layer operator selection with $F_{\mathrm{high}}$6, optimizes model parameters on $F_{\mathrm{high}}$7, architecture logits on $F_{\mathrm{high}}$8, adds entropy and cost regularizers, and anneals $F_{\mathrm{high}}$9 from $\mathrm{Conv}_{1\times1}$0 to $\mathrm{Conv}_{1\times1}$1 before discretization. In generative inference-time IPEF, VCF instead trains only a small aligner with

$\mathrm{Conv}_{1\times1}$2

where $\mathrm{Conv}_{1\times1}$3 and $\mathrm{Conv}_{1\times1}$4, then keeps the diffusion model frozen and optionally refines conditioning with Prompt-Noise Optimization (Xiao et al., 24 Jun 2026, Żywot et al., 24 May 2026).

NeuroCLIP illustrates a contrastive variant in which prompt fusion is optimized against physiological ambiguity rather than dense supervision. Its total objective,

$\mathrm{Conv}_{1\times1}$5

uses $\mathrm{Conv}_{1\times1}$6, $\mathrm{Conv}_{1\times1}$7, $\mathrm{Conv}_{1\times1}$8, and $\mathrm{Conv}_{1\times1}$9 for soft-target mixing while keeping the CLIP ViT frozen and training only EEG perturbation, dynamic filtering, CATF, visual prompt tokens, and lightweight heads (Wang et al., 12 Nov 2025).

5. Empirical behavior across application domains

In medical segmentation, IPEF was introduced primarily as a robustness mechanism against poor prompts. On prostate MRI, FA-SAM reaches average Dice 84.53 across unseen sites B–F, compared with SAMMed at 79.54; on particularly difficult sites, the reported gaps are 87.00 vs 77.29 on site B and 82.93 vs 73.98 on site C. The module-wise ablation also isolates its contribution: AGM*+SAM yields 81.31, AGM*+IPEF+SAM 83.52, and full SUFM+AGM+IPEF+SAM 84.53; with GT prompts, adding IPEF improves SAM by approximately 1.01% Dice on average (Zhuo et al., 23 Jul 2025).

In AIGC quality assessment, prompt-conditioned fusion materially changes correlation with human ratings. IP-IQA reports SRCC/PLCC/KRCC of 0.8401/0.8922/0.6635 on AGIQA-1k and 0.8634/0.9116/0.6844 on AGIQA-3k perceptual quality, while its alignment track on AGIQA-3k reaches 0.7578/0.8544/0.5734. A related multi-level formulation, MPEF-Net, reports Corr 0.8410/0.8968 on AGIQA-3K, whereas removing prompt embedding drops Corr to 0.7849/0.8710, indicating that hierarchical prompt-conditioned fusion rather than late fusion drives much of the gain (Qu et al., 2024, Meng et al., 23 Jul 2025).

In image fusion, IPEF has been used both for semantic guidance and for controllability. FILM obtains, for example on RoadScene, EN = 7.48, SD = 53.10, SF = 19.19, AG = 7.19, VIF = 0.64, and $G_v$0, and on Harvard medical fusion without fine-tuning reports EN = 4.74, SD = 69.48, SF = 25.58, AG = 6.71, VIF = 0.77, $G_v$1. CtrlFuse achieves overall mIoU 0.7963 on MSRS segmentation and an overall AP@[0.5:0.95] of 0.525 on DroneVehicle detection, while ControlFusion reports best EN, SD, VIF, and $G_v$2 on MSRS, LLVIP, RoadScene, and FMB and reaches mAP@0.5:0.95 = 0.609 on LLVIP with YOLOv8 (Zhao et al., 2024, Sun et al., 12 Jan 2026, Tang et al., 30 Mar 2025).

In parameter-efficient adaptation and multimodal alignment, the same design principle also yields measurable gains. Layer-specific fusion discovery raises VTAB-1k mean accuracy to 77.01% from 69.43% for VPT-Deep, with FGVC mean 91.60% and HTA mean 92.5. NeuroCLIP reaches 63.2% Top-1 zero-shot image retrieval on THINGS-EEG2, improving the previous best method by +12.3%, and reports 17.0% Top-1 under inter-subject conditions. In diffusion-based dual conditioning, VCF improves reference fidelity relative to text-only Stable Diffusion, moving from CLIP 0.29 and LPIPS 0.78 to CLIP 0.27 and LPIPS 0.76 under concatenation fusion, thereby making the text-alignment versus visual-correspondence trade-off explicit (Xiao et al., 24 Jun 2026, Wang et al., 12 Nov 2025, Żywot et al., 24 May 2026).

6. Limitations, misconceptions, and research directions

A common misconception is that IPEF is synonymous with cross-attention. The literature does not support that reduction. FA-SAM explicitly avoids new cross-attention and relies on concatenation, SE gating, upsampling, and skip connections, while PDV realizes prompt fusion through vector arithmetic and weighted embedding fusion rather than token-level attention (Zhuo et al., 23 Jul 2025, Tursun et al., 11 Feb 2025).

A second misconception is that prompt-aware fusion eliminates prompt sensitivity. The cited systems usually mitigate, rather than remove, this dependence. CtrlFuse reports robustness to incomplete or coarse masks, yet incorrect or overly broad masks may still emphasize wrong regions. IP-IQA likewise notes that noisy, ambiguous, or domain-shifted prompts may degrade performance, and its fusion does not explicitly model logical relations, negations, or multiple constraints (Sun et al., 12 Jan 2026, Qu et al., 2024).

Reported limitations are also methodological. Several papers omit FLOPs, parameter counts, latency, or hardware-constrained deployment studies even when fusion adds SEResBlocks, cross-attention, prompt encoders, or longer conditioning sequences. ControlFusion reports 123.898M parameters, 327.980G FLOPs, and 0.344 s average inference time, but many other works provide only qualitative statements about moderate overhead. This suggests that benchmarking IPEF modules by accuracy alone can hide important operating-cost trade-offs (Tang et al., 30 Mar 2025, Zhao et al., 2024).

Future directions in the cited work converge on richer semantics and more adaptive control. Proposed avenues include degradation-aware prompt pools and registration-robust prompt generation, as well as more general dual conditioning for diffusion models using lightweight aligners and inference-time refinement. A plausible implication is that IPEF will increasingly function as a general control interface between frozen foundation backbones and task-specific supervision, rather than as a single architectural block (Li et al., 2024, Żywot et al., 24 May 2026).