Cross-Modal Prompt Learning

• Cross-modal prompt learning is a method that optimizes learnable prompts across multiple modalities, serving as a dynamic interface to preserve semantic alignment while adapting frozen backbones.
• It leverages diverse architectural patterns, including bidirectional prompt exchange, shared-source generation, and hierarchical coupling, to enhance multimodal integration and robustness against modality gaps.
• This approach improves model performance in tasks like few-shot detection, image translation, and continual adaptation, effectively addressing issues like catastrophic forgetting and label noise.

Cross-modal prompt learning is a family of parameter-efficient adaptation methods for multimodal models in which learnable prompts are optimized in more than one modality, projected between modalities, or generated from one modality to condition another. In the literature summarized here, the approach appears in CLIP-style dual encoders, multimodal LLMs, medical image translation systems, retrieval models, and continual-learning pipelines. The common objective is to preserve or improve cross-modal alignment while adapting frozen or largely frozen backbones to downstream tasks, especially when the failure modes include modality isolation, hierarchical semantic decay, catastrophic forgetting, missing modalities, or label noise (Liu et al., 2023, Zheng et al., 20 Jul 2025).

1. From shallow prompting to cross-modal coupling

One early line of work framed cross-modal prompting as a way to combine few-shot visual evidence with text-side semantics without requiring class names at test time. In multi-modal few-shot object detection, a cross-modal soft prompt generator was learned from few-shot visual supports, and knowledge distillation was used so that the student generator could infer text-side prompts without human prior knowledge of class names (Han et al., 2022). This established a template that reappears later: prompts are not merely static context tokens, but a learnable interface between modalities.

Subsequent CLIP-based work moved from single-prompt or single-branch tuning to deeper coupling. PMPO extended the soft prompt from a single learnable prompt to multiple prompts, partitioned the visual encoder depth, and connected learnable prompts to separated visual depths so that different prompts could capture hierarchical contextual depths of visual representations (Tian et al., 2023). DCP then made the interaction explicitly bidirectional and layerwise through Cross-Modal Prompt Attention, arguing that prior methods either focused on the language branch alone or relied on shallow vision-language interaction (Liu et al., 2023). MmAP further formalized alignment during fine-tuning by generating both text and image prompts from a shared source prompt through a Kronecker Product construction, with a group-shared and task-specific prompt hierarchy for multi-task learning (Xin et al., 2023).

A plausible implication is that the field rapidly moved away from viewing prompts as isolated input-side adapters. The surveyed methods instead treat prompts as cross-modal carriers of semantic structure, routing information between visual and textual streams, across layers, tasks, and domains.

2. Architectural patterns

Several recurring architectural patterns organize the design space.

Method	Prompt interaction	Target setting
DCP	Cross-Modal Prompt Attention progressively generates visual and text prompts for the next layer	few-shot learning and domain generalization
MmAP	A shared source prompt is expanded into text and image prompts with the Kronecker Product	cross-domain multi-task learning
ChordPrompt	Bidirectional visual and textual prompt exchange via an aligner; projected prompts are injected into the Value vector	multi-domain task incremental learning
HiCroPL	Early text-to-image flow, later image-to-text flow, with a hierarchical knowledge mapper and layer-specific knowledge proxy	downstream adaptation and generalization

DCP is the clearest example of deeply coupled prompting. If $P_v^\ell$ and $P_t^\ell$ denote visual and text prompts at layer $\ell$, then the next-layer prompts are generated by cross-attention-like exchange,

$$P_{v}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{v}^{\ell} (P_{t}^{\ell})^T}{\sqrt{d_k}} \right) P_{t}^{\ell}, \qquad P_{t}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{t}^{\ell} (P_{v}^{\ell})^T}{\sqrt{d_k}} \right) P_{v}^{\ell},$$

so prompt learning is both deep and bidirectional rather than confined to a single initialization step (Liu et al., 2023).

MmAP represents a different pattern: shared-source generation. A source prompt $P_s$ is expanded to the text and image branches by

$$P_l = \mathcal{M}_l \otimes P_s, \qquad P_v = \mathcal{M}_v \otimes P_s,$$

which is explicitly intended to maintain multimodal alignment during fine-tuning while using only approximately $0.09\%$ of trainable parameters (Xin et al., 2023).

Another pattern is prompt exchange at every encoder layer. ChordPrompt introduces visual prompts and textual prompts at each layer of CLIP, projects prompts between modalities through an aligner, and injects the projected prompts into the Value vector rather than Query or Key (Wang et al., 24 Jun 2025). HiCroPL generalizes the same intuition to a hierarchical schedule: in early layers text prompts inject relatively clear semantics into visual prompts, while in later layers visual prompts encoding specific task-relevant objects flow back to refine text prompts. Its hierarchical knowledge mapper fuses representations from all previous layers, and its layer-specific knowledge proxy compresses the prompt set of a layer into a single proxy token for efficient interaction (Zheng et al., 20 Jul 2025).

A recurrent misconception is that cross-modal prompt learning is equivalent to text-to-vision prompting. The surveyed literature does not support that reduction. Some methods are symmetric and bidirectional, some are staged, and some use a shared latent prompt source rather than a single directional mapping.

3. Alignment objectives and prompt semantics

A central issue is what, precisely, prompts are supposed to align. In ALIGN, each image and class label is represented as an empirical distribution over multiple visual and textual prompts,

$$P = \sum_{m=1}^M \frac{1}{M}\delta_{v^m}, \qquad Q = \sum_{n=1}^N \frac{1}{N}\delta_{t^n},$$

and the image-label relation is formulated as a hierarchical optimal transport problem. The prompt-level cost matrix is

$$C_{mn} = 1 - \mathrm{sim}(v^m, t^n) + \beta d_{\mathrm{OT}^\lambda}(p^m, q^n; \hat{C}^{mn}),$$

so global prompt alignment and token-level alignment are coupled in a single transport framework (Wang et al., 2023). In that formulation, prior single-mode or holistic prompt-tuning methods become special cases obtained by specific hyperparameter choices.

PROMISE addresses a different semantic problem: inconsistency between complete multimodal data and incomplete-modality scenarios. It uses modality-specific prompt pools and a prompt-attention mechanism to synthesize missing-modality representations, and it couples that synthesis with hierarchical contrastive learning. The total contrastive objective is

$$\mathcal{L}_{contrast} = \alpha \mathcal{L}_{FNCL} + (1-\alpha)\mathcal{L}_{CCCL},$$

where Fusion-driven Nexus Contrastive Learning enforces cross-modal semantic consistency and Cohesion-driven Core Contrastive Learning promotes intra-modal discriminability (Chen et al., 14 Nov 2025).

Other methods ground prompt semantics in external or structured knowledge. KAnoCLIP replaces fixed anomaly prompts with Knowledge-Driven Prompt Learning, combining general knowledge from GPT-3.5 and image-specific knowledge from Llama3-VQA. Its knowledge-driven loss pushes learnable abnormal prompts toward the knowledge base and away from normal prompts,

$P_t^\ell$0

and is integrated with multi-level cross-modal interaction for zero-shot anomaly detection (Li et al., 7 Jan 2025). K-MaT likewise anchors prompt learning to automatically generated clinical descriptions and aligns high-end and low-end prompt manifolds with Fused Gromov-Wasserstein optimal transport, explicitly targeting strict zero-shot transfer to low-end modalities without low-end training images (Zeng et al., 6 Mar 2026).

The literature also contains prompt designs where semantics are injected from the image side rather than the text side. VisPrompt uses cross-modal attention so that prompt tokens act as queries over projected image patches, then applies FiLM-based conditional modulation and token-wise gating to control how much visual evidence should be injected into the prompt under label noise (Geng et al., 10 Apr 2026). PromptHash, by contrast, uses text affinity-aware prompting to preserve semantics truncated by CLIP’s 77-token context limit, then aligns images, prompt-text features, and prompts through hierarchical contrastive learning for cross-modal hashing (Zou et al., 20 Mar 2025).

These formulations show that “prompt alignment” can mean token-level transport, instance-level contrastive agreement, knowledge anchoring, manifold alignment, or robustness-oriented visual conditioning, depending on the failure mode being addressed.

4. Continual, incremental, and federated learning

Cross-modal prompt learning has become especially prominent in continual adaptation. CP-Prompt addresses cross-modal domain-incremental learning with a twin prompting architecture: common prompts for inter-domain shared knowledge are inserted before the first transformer layer, while personalized prompts for intra-domain knowledge are compositionally inserted within multi-head self-attention layers as Prefix-One prompts. On CDDB-Hard, it reports $P_t^\ell$1 and $P_t^\ell$2; on CORe50, $P_t^\ell$3; and on DomainNet, $P_t^\ell$4. The paper also reports that tuning only prompts, about $P_t^\ell$5 of parameters, yields better performance than full fine-tuning (Feng et al., 2024).

ChordPrompt adapts CLIP to multi-domain task incremental learning by orchestrating bidirectional prompt exchange between visual and textual branches, with domain-adaptive text prompts for prompt selection. On the MTIL benchmark with 11 diverse domains and over 1,200 classes, it reports stronger “Last”, “Avg”, and “Transfer” accuracy than prior baselines, uses about $P_t^\ell$6M trainable parameters, and in ablation its cross-modal prompt learning gives more than $P_t^\ell$7 improvement over previous prompt methods such as L2P, S-Prompts, and MaPLe (Wang et al., 24 Jun 2025).

In continual Visual Question Answering, MM-Prompt argues that prompt isolation exacerbates modality imbalance. It therefore introduces cross-modal prompt query and cross-modal prompt recovery, with masking and alignment losses to reduce representational drift. On VQA v2 under the Dual Increment setting, it reports $P_t^\ell$8 and $P_t^\ell$9; on NExT-QA under the same setting, $\ell$0 and $\ell$1 (Li et al., 26 May 2025).

DRAPE extends the same general direction to multimodal continual instruction tuning for MLLMs, but replaces prompt selection from a fixed pool with dynamic instance-specific soft prompt generation. It derives prompt queries from the textual instruction, cross-attends to visual patch features, prepends the synthesized prompt to a frozen LLM, and uses null-space gradient projection plus CLIP-based prototype routing to limit forgetting and enable task-label-free inference. On CoIN it reports average accuracy $\ell$2, compared with $\ell$3 for the strongest baseline ProgLoRA; on UCIT it reports $\ell$4; it also reports $\ell$5, $\ell$6, $\ell$7M trainable parameters per task, and $\ell$8h training time versus $\ell$9M and $$P_{v}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{v}^{\ell} (P_{t}^{\ell})^T}{\sqrt{d_k}} \right) P_{t}^{\ell}, \qquad P_{t}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{t}^{\ell} (P_{v}^{\ell})^T}{\sqrt{d_k}} \right) P_{v}^{\ell},$$0h for MoELoRA (Hu et al., 11 May 2026).

A related decentralization of prompt adaptation appears in FedCSAP. There, low-, mid-, and high-level CLIP visual features are combined with client-specific style indicators from batch-level statistics and textual context, and prompt-related parameters are trained by local updates with global aggregation. On the reported federated benchmark, FedCSAP achieves Local $$P_{v}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{v}^{\ell} (P_{t}^{\ell})^T}{\sqrt{d_k}} \right) P_{t}^{\ell}, \qquad P_{t}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{t}^{\ell} (P_{v}^{\ell})^T}{\sqrt{d_k}} \right) P_{v}^{\ell},$$1, Base $$P_{v}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{v}^{\ell} (P_{t}^{\ell})^T}{\sqrt{d_k}} \right) P_{t}^{\ell}, \qquad P_{t}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{t}^{\ell} (P_{v}^{\ell})^T}{\sqrt{d_k}} \right) P_{v}^{\ell},$$2, New $$P_{v}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{v}^{\ell} (P_{t}^{\ell})^T}{\sqrt{d_k}} \right) P_{t}^{\ell}, \qquad P_{t}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{t}^{\ell} (P_{v}^{\ell})^T}{\sqrt{d_k}} \right) P_{v}^{\ell},$$3, and harmonic mean $$P_{v}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{v}^{\ell} (P_{t}^{\ell})^T}{\sqrt{d_k}} \right) P_{t}^{\ell}, \qquad P_{t}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{t}^{\ell} (P_{v}^{\ell})^T}{\sqrt{d_k}} \right) P_{v}^{\ell},$$4, and improves over FedCoOp by $$P_{v}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{v}^{\ell} (P_{t}^{\ell})^T}{\sqrt{d_k}} \right) P_{t}^{\ell}, \qquad P_{t}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{t}^{\ell} (P_{v}^{\ell})^T}{\sqrt{d_k}} \right) P_{v}^{\ell},$$5 on unseen classes and $$P_{v}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{v}^{\ell} (P_{t}^{\ell})^T}{\sqrt{d_k}} \right) P_{t}^{\ell}, \qquad P_{t}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{t}^{\ell} (P_{v}^{\ell})^T}{\sqrt{d_k}} \right) P_{v}^{\ell},$$6 on unseen datasets on average (Prasad et al., 17 Aug 2025).

Taken together, these systems show that continual cross-modal prompt learning has expanded from task-level prompt pools toward instance-level generation, explicit anti-forgetting mechanisms, and domain- or client-adaptive routing.

5. Robustness under missing data, label noise, and modality shift

One important branch of the literature uses cross-modal prompt learning to make multimodal systems robust when a modality is absent. PROMISE is designed explicitly for missing modalities, using modality-specific prompt pools and prompt-attention to generate representations for missing modalities, then aligning complete and incomplete scenarios through hierarchical contrastive learning. The reported experiments emphasize robustness at very high missing rates, including $$P_{v}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{v}^{\ell} (P_{t}^{\ell})^T}{\sqrt{d_k}} \right) P_{t}^{\ell}, \qquad P_{t}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{t}^{\ell} (P_{v}^{\ell})^T}{\sqrt{d_k}} \right) P_{v}^{\ell},$$7, and ablations indicate that prompt-attention alone is roughly baseline-level, whereas combining prompt-attention with both contrastive components yields the best gains (Chen et al., 14 Nov 2025).

Label-noise robustness motivates a different design. VisPrompt starts from the observation that prompt parameters are highly susceptible to noisy supervision, while visual content is relatively reliable. It therefore reverse-injects visual instance semantics into prompt representations by cross-modal attention and regulates the injection with a lightweight FiLM modulation module. The reported parameter overhead is less than $$P_{v}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{v}^{\ell} (P_{t}^{\ell})^T}{\sqrt{d_k}} \right) P_{t}^{\ell}, \qquad P_{t}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{t}^{\ell} (P_{v}^{\ell})^T}{\sqrt{d_k}} \right) P_{v}^{\ell},$$8, the FiLM gate itself adds $$P_{v}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{v}^{\ell} (P_{t}^{\ell})^T}{\sqrt{d_k}} \right) P_{t}^{\ell}, \qquad P_{t}^{\ell+1} = \mathrm{softmax} \left( \frac{P_{t}^{\ell} (P_{v}^{\ell})^T}{\sqrt{d_k}} \right) P_{v}^{\ell},$$9, and under $P_s$0 noise VisPrompt maintains up to 15 points higher accuracy on EuroSAT than alternatives (Geng et al., 10 Apr 2026).

Cross-modal prompt learning is also used to traverse severe modality shifts in medical imaging. K-MaT is built for the case where a biomedical VLM adapted on high-end imaging such as CT must transfer to low-end modalities such as chest X-ray or ultrasound without low-end training images. It factorizes prompts, anchors them to clinical text descriptions, and aligns the low-end prompt manifold to the high-end space with Fused Gromov-Wasserstein optimal transport. Across four cross-modal benchmarks it reports average harmonic mean accuracy $P_s$1, up from BiomedCoOp’s $P_s$2, and macro-F1 $P_s$3. On the breast imaging task, CoOp drops to $P_s$4 accuracy on the low-end modality, whereas K-MaT reports $P_s$5 high-end/low-end accuracy and harmonic mean $P_s$6 (Zeng et al., 6 Mar 2026).

KAnoCLIP addresses zero-shot anomaly detection with a different form of knowledge transfer. It combines knowledge-driven prompt learning, CLIP-VV, Bi-Directional Cross-Attention for Multi-Level Cross-Modal Interaction, and Conv-Adapter. Across 12 industrial and medical datasets, the reported gains include up to $P_s$7 points in pixel-level AUC and $P_s$8 points in image-level AUC on MVTec-AD, and up to $P_s$9 and $$P_l = \mathcal{M}_l \otimes P_s, \qquad P_v = \mathcal{M}_v \otimes P_s,$$0 on medical datasets (Li et al., 7 Jan 2025).

These systems indicate that cross-modal prompt learning is not limited to clean, fully observed, in-distribution settings. It is increasingly treated as a robustness mechanism for incomplete inputs, corrupted supervision, and large domain or modality gaps.

6. Applications beyond classification

The technique has expanded well beyond CLIP classification. MedPrompt applies cross-modal prompting to multi-task medical image translation through a Self-adaptive Prompt Block composed of a Prompt Extraction Block and Prompt Fusion Block within a Transformer-based encoder-decoder. Across five datasets and four modality pairs, it reports state-of-the-art visual quality, and in the summarized results it outperforms the second-best by 1 dB in PSNR on average (Chen et al., 2023).

MGCC uses cross-modal in-context learning for multimodal generation from complex prompt sequences. Its Cross-Modal Refinement Module learns cross-modal dependencies in the LLM embedding space, and its Contextual Object Grounding Module predicts object bounding boxes for multi-object scenes before conditioning the diffusion model. On VIST, MGCC reports CLIP Similarity $$P_l = \mathcal{M}_l \otimes P_s, \qquad P_v = \mathcal{M}_v \otimes P_s,$$1 versus $$P_l = \mathcal{M}_l \otimes P_s, \qquad P_v = \mathcal{M}_v \otimes P_s,$$2 for GILL; on VisDial it reports $$P_l = \mathcal{M}_l \otimes P_s, \qquad P_v = \mathcal{M}_v \otimes P_s,$$3 versus $$P_l = \mathcal{M}_l \otimes P_s, \qquad P_v = \mathcal{M}_v \otimes P_s,$$4 (Kumar et al., 2024).

PromptHash brings prompt learning into adaptive hashing retrieval. Its Text Affinity-Aware Prompting preserves contextual information under text truncation, its Adaptive Gated State Space Fusion combines State Space Models with Transformers for feature fusion, and its Prompt Affinity Alignment performs hierarchical contrastive alignment. On NUS-WIDE it reports gains of $$P_l = \mathcal{M}_l \otimes P_s, \qquad P_v = \mathcal{M}_v \otimes P_s,$$5 and $$P_l = \mathcal{M}_l \otimes P_s, \qquad P_v = \mathcal{M}_v \otimes P_s,$$6 for image-to-text and text-to-image retrieval; on MIRFLICKR-25K, $$P_l = \mathcal{M}_l \otimes P_s, \qquad P_v = \mathcal{M}_v \otimes P_s,$$7 and $$P_l = \mathcal{M}_l \otimes P_s, \qquad P_v = \mathcal{M}_v \otimes P_s,$$8; and on MS COCO, $$P_l = \mathcal{M}_l \otimes P_s, \qquad P_v = \mathcal{M}_v \otimes P_s,$$9 and $0.09\%$0 (Zou et al., 20 Mar 2025).

Other application-specific systems broaden the notion of prompt further. In social media popularity prediction, dual-grained prompt learning combines global and local prompts with hierarchical visual and textual prototypes and cross-modal attention, reporting SRC up to $0.09\%$1 and MAE as low as $0.09\%$2 on the SMPD dataset (Zhou et al., 22 Aug 2025). In remote sensing, CLV-Net lets users provide a bounding box as a visual prompt, embeds that cue into the language prompt at a special <BOX> token, and aligns captions with masks through semantic and relationship consistency losses; on the reported benchmark table it achieves CIDEr $0.09\%$3, METEOR $0.09\%$4, AP50 $0.09\%$5, mIoU $0.09\%$6, and Recall $0.09\%$7 (Zhang et al., 12 Dec 2025).

The broadest lesson is that cross-modal prompt learning is not a single algorithmic recipe but a design principle. Prompts may be static or instance-specific, shallow or deep, symmetric or asymmetric, class-conditioned or task-conditioned, and they may target recognition, retrieval, reasoning, generation, translation, anomaly detection, or continual adaptation. The surveyed literature repeatedly identifies modality isolation, hierarchical semantic decay, catastrophic forgetting, label noise, missing modalities, and modality-specific shortcuts as the main failure modes. This suggests that future work will likely continue to emphasize bidirectional prompt exchange, multi-scale alignment, knowledge anchoring, and prompt generation mechanisms that are adaptive at the instance, task, or domain level.