MPLM: Multimodal Prompt Learning

• MPLM is a family of parameter-efficient multimodal adaptation methods that use learned prompts to steer frozen backbones across diverse modalities.
• It employs diverse prompt mechanisms—such as modality-specific cues, low-rank factorization, and cross-modal alignment—to handle missing information and improve robustness.
• Evaluations in domains like MRI+PET diagnosis and RGB-D segmentation demonstrate competitive performance while tuning only a fraction of model parameters.

Multimodal Prompt Learning Mechanism (MPLM) denotes a family of prompt-based adaptation schemes for multimodal models in which a frozen or largely frozen backbone is steered by learned prompts, prompt-conditioned token transformations, or prompt-guided alignment modules rather than by full end-to-end fine-tuning. In the recent literature, MPLM appears in several distinct but related forms: concept- and graph-driven prompting for MRI+PET diagnosis, modality-specific prompting for missing image-text inputs, prompt-conditioned visual encoding in multimodal LLMs, and deep low-rank prompting for CLIP-style vision-LLMs (Peng et al., 2023, Jang et al., 2023, Wu et al., 2024, Ghiasvand et al., 24 Feb 2026).

1. Conceptual scope and representative formulations

The term does not denote a single canonical architecture. Some works use it to name a concrete module inside a task-specific system, whereas others explicitly derive a general design template from one instantiation. This suggests that MPLM is best understood as an umbrella category of parameter-efficient multimodal adaptation methods whose common feature is the use of prompts as the main control interface between pretrained backbones and downstream objectives (Peng et al., 2023, Xu et al., 3 Mar 2025, Chen et al., 14 Nov 2025).

System	Modalities	Prompt mechanism
MMGPL (Peng et al., 2023)	MRI + PET	concept-derived token weighting and graph prompts
MSPs (Jang et al., 2023)	image + text	modality-specific prompts with orthogonality
DPLNet (Dong et al., 2023)	RGB-D / RGB-T	MPG + MFA on a frozen RGB backbone
PIP-MM (Wu et al., 2024)	prompt text + image in MLLM	prompt-derived T-CLS replaces ViT CLS
MMLoP (Ghiasvand et al., 24 Feb 2026)	vision + text	deep low-rank prompts with shared up-projection

Across these systems, prompts may be explicit learnable tokens, graph-conditioned feature updates, prompt pools, placeholders for missing modalities, or shared prompt anchors in an output-level aligner. The family therefore includes both classical prompt tuning, where prompts are concatenated to token sequences, and broader prompt-conditioned mechanisms, where prompts alter embeddings, attention, or cross-modal geometry without necessarily appearing as literal prefix tokens (Ng et al., 18 Aug 2025, Jiang et al., 2023).

2. Architectural patterns

A recurrent architectural pattern is the use of frozen multimodal encoders with small trainable prompt modules and lightweight heads. In the modality-specific prompting framework for missing modalities, the backbone is ViLT, the prompts are attached at the input of the multimodal transformer’s MHSA, and the complete case is handled by the element-wise sum

$P_{\text{complete}} = P_{is} + P_{ts},$

so that the same image-specific and text-specific prompts are reused across image-only, text-only, and complete inputs (Jang et al., 2023).

Other systems distribute prompts across depth. MuAP introduces modality-specific prompts for text and vision, generates missing-type prompts through a residual MLP, and injects them in the first six transformer layers with either head-fusion prompting or cross-fusion prompting; the training schedule then alternates text-step and image-step tuning before a joint alignment stage (Dai et al., 2024). MMLoP keeps this deep multimodal prompting idea but factorizes each layer’s prompt matrix into low-rank components,

$$P_v^{(l)} = U^{(l)}V_v^{(l)}, \qquad P_t^{(l)} = U^{(l)}V_t^{(l)},$$

and uses a shared up-projection $U^{(l)}$ to couple vision and text prompts while retaining only $11.5\text{K}$ trainable parameters (Ghiasvand et al., 24 Feb 2026).

A distinct pattern moves prompt interaction to the encoder output rather than the input. SPANER keeps CLIP- or CLAP-style backbones frozen and lets each modality embedding query a shared prompt matrix through a lightweight cross-attention aligner, so that prompts act as modality-agnostic conceptual anchors for a unified semantic space (Ng et al., 18 Aug 2025). PIP-MM shifts prompt conditioning even earlier: it uses the frozen LLM to vectorize the natural-language prompt, maps the last hidden state through a trained MLP into a ViT-compatible vector, and replaces the image encoder’s class embedding with this prompt-derived token, yielding

$$\mathbf{z}_0 = \big[\mathbf{T}_{\text{class}};\mathbf{I}_p^1\mathbf{E};\dots;\mathbf{I}_p^N\mathbf{E}\big]+\mathbf{E}_{\text{pos}},$$

so that visual encoding becomes prompt-aware from the first layer (Wu et al., 2024).

3. Mechanisms for incomplete or missing modalities

Handling missing modalities is one of the most developed MPLM use cases. A central observation is combinatorial: missing-aware prompting that allocates a separate prompt tensor to every non-empty modality subset scales as $2^M-1$, whereas modality-specific prompting requires only $M$ prompts. The modality-specific alternative also improves robustness when train-time and test-time missingness differ, because prompts are tied to modalities rather than to missingness patterns (Jang et al., 2023). In that setting, prompt diversity is enforced with an orthogonality term

$$L_{ortho} = \frac{|f(P_{is})\cdot f(P_{ts})|}{\max(\|f(P_{is})\|_2\|f(P_{ts})\|_2,\epsilon)},$$

and the total objective is $L_{total}=L_{cls}+\lambda L_{ortho}$ (Jang et al., 2023).

Tri-modal sentiment analysis and emotion recognition extend the idea with three prompt types: generative prompts, missing-signal prompts, and missing-type prompts. The generative prompts drive a Missing Modality Generation Module that reconstructs missing feature streams, the missing-signal prompts mark whether each modality is real or generated, and the missing-type prompts encode joint missingness patterns through a projection mechanism while a pre-trained MulT backbone stays frozen (Guo et al., 2024). MuAP arrives at the same problem from a different angle: missing modalities are represented with blank-image or empty-text placeholders, and missing-type prompts are generated from the opposite modality’s complete-type prompts rather than allocated as a separate prompt bank for every pattern (Dai et al., 2024).

PROMISE pushes incomplete-modality prompting toward representation generation. It keeps CLIP ViT-L/14 frozen, stores modality-specific prompt pools, and uses a prompt-attention block to generate latent representations for missing modalities from available ones. Those generated features are then constrained by a hierarchical contrastive objective,

$$\mathcal{L}_{\text{contrast}}=\alpha\mathcal{L}_{FNCL}+(1-\alpha)\mathcal{L}_{CCCL},$$

where FNCL enforces inter-modal consistency and CCCL enforces label-aware intra-modal cohesion (Chen et al., 14 Nov 2025). DisPro develops an analogous two-stage formulation for incomplete multimodal survival prediction: UniPro first distills modality-specific knowledge distributions with unimodal prompts, and MultiPro then uses available modalities as prompts for an LLM to infer missing modalities while unimodal teachers supervise the inferred tokens through modality-specific distillation losses (Xu et al., 3 Mar 2025).

4. Semantic, structural, and cross-modal prompt design

Several MPLM variants move beyond generic learnable tokens and explicitly encode semantics, structure, or task priors. MMGPL is exemplary in this regard. It uses GPT-4 to generate disease-related concepts, aligns MRI and PET patch tokens to those concepts, re-weights tokens by concept similarity to suppress irrelevant patches, constructs a graph over tokens from their concept-similarity profiles, and applies a GCN to produce graph-prompted token embeddings before feeding them into a frozen unified transformer encoder (Peng et al., 2023). The method therefore treats prompts not as extra prefix vectors but as concept-derived token weights and graph-derived structural embeddings.

MCL-AD applies the same principle to zero-shot 3D anomaly detection. Its Object-Agnostic Decoupled Text Prompt defines normal and anomaly prompts separately for RGB and point-cloud branches,

$$P_v^{(l)} = U^{(l)}V_v^{(l)}, \qquad P_t^{(l)} = U^{(l)}V_t^{(l)},$$0

$$P_v^{(l)} = U^{(l)}V_v^{(l)}, \qquad P_t^{(l)} = U^{(l)}V_t^{(l)},$$1

and then aligns same-state prompts across modalities with a triplet-style multimodal contrastive loss (Li et al., 12 Sep 2025). Because the prompts remain category-agnostic and state-based, the framework targets unseen objects by comparing RGB and point-cloud features to shared “normal object” and “damaged object” semantics rather than to class names (Li et al., 12 Sep 2025).

A different semantic strategy appears in task-oriented multimodal mutual learning for CLIP. There, image features produce class-aware text prompts through a Class-Aware Text Prompt module, while the resulting text embeddings guide image features through Text-Guided Feature Tuning. The merged prediction combines original CLIP similarities and augmented similarities,

$$P_v^{(l)} = U^{(l)}V_v^{(l)}, \qquad P_t^{(l)} = U^{(l)}V_t^{(l)},$$2

which couples image-to-text and text-to-image prompting in a bidirectional loop (Long et al., 2023). SAMPLE adopts a lighter variant: CLIP text and image features are concatenated, scaled by a similarity-aware gate $$P_v^{(l)} = U^{(l)}V_v^{(l)}, \qquad P_t^{(l)} = U^{(l)}V_t^{(l)},$$3, and fused with a prompt-based RoBERTa masked-token representation through a soft verbalizer, so that multimodal evidence is explicitly down-weighted when cross-modal similarity is low (Jiang et al., 2023).

5. Objectives, regularization, and efficiency

Despite the diversity of prompt forms, the optimization regime is usually parameter-efficient and heavily regularized. In the missing-modality MSP formulation, only prompts, pooler, and classifier are trained while the text encoder, image encoder, and multimodal transformer remain frozen, yielding less than $$P_v^{(l)} = U^{(l)}V_v^{(l)}, \qquad P_t^{(l)} = U^{(l)}V_t^{(l)},$$4 trainable parameters (Jang et al., 2023). SAMPLE keeps both RoBERTa and CLIP frozen and trains only prompt-related components, projection heads, a fusion layer, and a soft verbalizer, resulting in $$P_v^{(l)} = U^{(l)}V_v^{(l)}, \qquad P_t^{(l)} = U^{(l)}V_t^{(l)},$$5–$$P_v^{(l)} = U^{(l)}V_v^{(l)}, \qquad P_t^{(l)} = U^{(l)}V_t^{(l)},$$6M trainable parameters instead of full RoBERTa fine-tuning at about $$P_v^{(l)} = U^{(l)}V_v^{(l)}, \qquad P_t^{(l)} = U^{(l)}V_t^{(l)},$$7M (Jiang et al., 2023).

MMLoP emphasizes that deep multimodal prompting need not abandon parameter efficiency. Besides its low-rank prompt factorization, it introduces a self-regulating consistency loss that anchors prompted image and text features to frozen zero-shot CLIP features at both feature and logit levels, and a uniform drift correction that subtracts the mean residual shared across class text embeddings before normalization. With these components, MMLoP reports a harmonic mean of $$P_v^{(l)} = U^{(l)}V_v^{(l)}, \qquad P_t^{(l)} = U^{(l)}V_t^{(l)},$$8 on base-to-novel generalization while retaining only $$P_v^{(l)} = U^{(l)}V_v^{(l)}, \qquad P_t^{(l)} = U^{(l)}V_t^{(l)},$$9 trainable parameters (Ghiasvand et al., 24 Feb 2026).

Prototype- and prompt-selection mechanisms provide a second efficiency axis. ModalPrompt, designed for continual learning of large multimodal models, freezes the LMM backbone and learns prototype prompts for each task; at inference it selects the top-$U^{(l)}$0 prompts through CLIP image-text similarities instead of evaluating all task-specific adapters. The reported result is about $U^{(l)}$1 performance gain on continual learning benchmarks with $U^{(l)}$2 inference speed and without training cost growing in proportion to the number of tasks (Zeng et al., 2024). This suggests that prompt routing, not only prompt construction, is a central MPLM design variable.

6. Domains, empirical record, limitations, and security

MPLM has been applied across markedly different multimodal domains. In neurological disorder diagnosis, MMGPL reports $U^{(l)}$3 ACC and $U^{(l)}$4 AUC on ADNI-3CLS, $U^{(l)}$5 ACC and $U^{(l)}$6 AUC on ADNI-4CLS, and $U^{(l)}$7 ACC and $U^{(l)}$8 AUC on ABIDE, while its heat maps concentrate on clinically relevant areas such as the hippocampus and parahippocampal gyrus (Peng et al., 2023). In multimodal sentiment analysis and emotion recognition with missing modalities, the tri-prompt MulT framework reports average results over six missing cases of $U^{(l)}$9 ACC/F1 on MOSI, $11.5\text{K}$0 on IEMOCAP, $11.5\text{K}$1 on CH-SIMS, and $11.5\text{K}$2 on MOSEI (Guo et al., 2024). In zero-shot 3D anomaly detection, MCL-AD reports $11.5\text{K}$3 I-AUROC, $11.5\text{K}$4 AP, $11.5\text{K}$5 P-AUROC, and $11.5\text{K}$6 AUPRO for the full MPLM with object-agnostic decoupled prompts and multimodal contrastive loss (Li et al., 12 Sep 2025).

Dense prediction is another active area. DPLNet adapts a frozen RGB segmentation model to RGB-D/T inputs through its dual-prompt mechanism and reports $11.5\text{K}$7 mIoU on NYUD-v2 with multi-scale inference, $11.5\text{K}$8 mIoU on SUN RGB-D, and $11.5\text{K}$9 mIoU on PST900, while introducing $$\mathbf{z}_0 = \big[\mathbf{T}_{\text{class}};\mathbf{I}_p^1\mathbf{E};\dots;\mathbf{I}_p^N\mathbf{E}\big]+\mathbf{E}_{\text{pos}},$$0M trainable prompt and adapter parameters plus a $$\mathbf{z}_0 = \big[\mathbf{T}_{\text{class}};\mathbf{I}_p^1\mathbf{E};\dots;\mathbf{I}_p^N\mathbf{E}\big]+\mathbf{E}_{\text{pos}},$$1M decoder (Dong et al., 2023). Prompt-conditioned MLLM adaptation shows a related effect at the generative interface: PIP-MM reports gains such as MM-Vet $$\mathbf{z}_0 = \big[\mathbf{T}_{\text{class}};\mathbf{I}_p^1\mathbf{E};\dots;\mathbf{I}_p^N\mathbf{E}\big]+\mathbf{E}_{\text{pos}},$$2, MME $$\mathbf{z}_0 = \big[\mathbf{T}_{\text{class}};\mathbf{I}_p^1\mathbf{E};\dots;\mathbf{I}_p^N\mathbf{E}\big]+\mathbf{E}_{\text{pos}},$$3, and MMMU $$\mathbf{z}_0 = \big[\mathbf{T}_{\text{class}};\mathbf{I}_p^1\mathbf{E};\dots;\mathbf{I}_p^N\mathbf{E}\big]+\mathbf{E}_{\text{pos}},$$4, and maintains excellent generation results even when half of the visual tokens are reduced (Wu et al., 2024). In multimodal fake news detection, M-SAMPLE reaches $$\mathbf{z}_0 = \big[\mathbf{T}_{\text{class}};\mathbf{I}_p^1\mathbf{E};\dots;\mathbf{I}_p^N\mathbf{E}\big]+\mathbf{E}_{\text{pos}},$$5 Macro-F1 and $$\mathbf{z}_0 = \big[\mathbf{T}_{\text{class}};\mathbf{I}_p^1\mathbf{E};\dots;\mathbf{I}_p^N\mathbf{E}\big]+\mathbf{E}_{\text{pos}},$$6 accuracy on PolitiFact in the 100-shot setting and improves over FT-RoBERTa with more than $$\mathbf{z}_0 = \big[\mathbf{T}_{\text{class}};\mathbf{I}_p^1\mathbf{E};\dots;\mathbf{I}_p^N\mathbf{E}\big]+\mathbf{E}_{\text{pos}},$$7 fewer trainable parameters (Jiang et al., 2023).

The literature also exposes systematic limitations. Several frameworks are evaluated only in the two-modality image-text setting, orthogonality is often defined only at the prompt level, and more structured uncertainty modeling is usually absent (Jang et al., 2023, Chen et al., 14 Nov 2025). Other works note that prompt mechanisms may still depend heavily on CLIP-, ViLT-, or BioBERT-style pretrained spaces, that prompt counts or stored prompt banks grow with the number of modalities or tasks, and that task boundaries remain explicit in continual-learning settings (Xu et al., 3 Mar 2025, Zeng et al., 2024). A further complication is security: BadPromptFL shows that prompt-based federated learning in CLIP-style multimodal models opens a distinct attack surface, achieving attack success rates above $$\mathbf{z}_0 = \big[\mathbf{T}_{\text{class}};\mathbf{I}_p^1\mathbf{E};\dots;\mathbf{I}_p^N\mathbf{E}\big]+\mathbf{E}_{\text{pos}},$$8 with minimal visibility and limited client participation by poisoning only aggregated prompt parameters rather than the backbone weights (Zhang et al., 11 Aug 2025).

Taken together, the available formulations depict MPLM not as a single algorithm but as a broad parameter-efficient design space. Its principal axes are prompt location, prompt semantics, modality routing under incompleteness, structural regularization, and the degree to which prompting shapes encoder internals rather than merely appending tokens at the input. This suggests that future MPLM research will continue to be defined less by one standard architecture than by how prompts are used to encode multimodal priors, compensate for missing information, and preserve generalization under constrained adaptation.