---
title: 'VaMP: Variational Multi-Modal Prompt Learning'
url: https://www.emergentmind.com/topics/variational-multi-modal-prompt-learning-vamp
type: topic
---

# VaMP: Variational Multi-Modal Prompt Learning

Variational Multi-Modal Prompt Learning (VaMP) is a probabilistic prompt-tuning paradigm for vision-language models (VLMs) enabling instance-conditioned, uncertainty-aware adaptation of prompts to input content and class semantics. Building on foundational limitations of deterministic prompt learning and recent advances in variational prompt modeling, VaMP structurally unifies instance-specific prompt generation, semantic consistency through class-aware priors, and end-to-end training by variational inference. It achieves state-of-the-art results in few-shot and domain generalization settings, demonstrates compatibility with diverse CLIP-style backbones, and offers extendability to a range of multimodal tasks [2511.22664].

## 1. Motivation and Limitations of Deterministic Prompt Learning

Existing multi-modal prompt learning for VLMs, such as CLIP and its soft prompt-tuning extensions, typically uses deterministic and globally shared prompt tokens. These approaches lack the capacity to adapt the prompt representation to instance-specific visual or semantic cues and do not quantify epistemic uncertainty, which is critical for robust generalization, particularly in few-shot or out-of-distribution regimes. Additionally, standard practice assumes isotropic Gaussian priors in the prompt latent space, disregarding inter-class semantic variability [2511.22664]. This framework constrains model expressivity and impairs robust transfer.

## 2. Probabilistic Formulation: Latent Prompt Modeling and Inference

VaMP recasts prompt tuning as variational inference over a learned latent prompt variable sequence \( z = \{z_i\}_{i=J}^{J+H-1} \), where each \( z_i \) is injected as a prompt token at transformer layer \( i \) in the frozen CLIP text encoder. For each input image \( x \) and label \( y \), the framework models an input- and class-conditioned posterior over \( z \):

\[
q_\phi(z\mid x, y) = \prod_{i=J}^{J+H-1} \mathcal N(z_i; \mu_i, \mathrm{diag}(\sigma_i^2)), \quad [\mu_i, \sigma_i] = \phi_i(\bar f_x)
\]

where \( \bar f_x \) denotes the frozen CLIP image embedding and the \( \phi_i \) are small MLPs. Uncertainty and sample-specificity are captured by the distributional posterior. VaMP employs a class-aware prior, constructed from class prototypes \( o_y = \frac{1}{|D_y|}\sum_{x'\in D_y}\bar f_{x'} \) using frozen CLIP features; each layer's prior is parameterized as \( [\hat\mu_i, \hat\sigma_i] = \psi_i(o_y) \) for MLPs \( \psi_i \). The prior thus encodes semantic structure beyond mere isotropy [2511.22664].

## 3. Evidence Lower Bound Objective and Optimization

Variational training seeks to maximize the marginal log-likelihood of target labels under the prompted model, realized via the evidence lower bound (ELBO):

\[
\log p(y\mid x, t) \geq \mathbb{E}_{z \sim q_\phi(z\mid x, y)}[\log p(y\mid x, t, z)] - \mathrm{KL}(q_\phi(z\mid x, y) \| p_\theta(z\mid x, y))
\]

The first term encourages correct predictions conditioned on the sampled latent prompt sequence \( z \); the second term regularizes the instance posterior against the class-aware prior, maintaining semantic consistency. Latent sampling is performed per instance and per prompt layer via the reparameterization trick:

\[
z_i = \mu_i + \sigma_i \odot \epsilon_i, \quad \epsilon_i \sim \mathcal N(0, I)
\]

All MLPs and prompt generator components are trained end-to-end with the CLIP backbones remaining frozen. Monte Carlo estimation is used for the ELBO expectation in practice [2511.22664].

## 4. Prompt Generation, Model Architecture, and Integration

The VaMP framework is structured as follows:

- **Visual Encoder:** Frozen CLIP ViT extracts image embedding \( \bar f_x \in \mathbb R^{d_{vl}} \).
- **Posterior MLPs:** \( \{\phi_i\} \) produce posterior means and standard deviations for each of \( H \) prompt-injected transformer layers.
- **Prior MLPs:** \( \{\psi_i\} \) generate class-anchored Gaussian priors using the class prototype \( o_y \).
- **Prompt Injection:** Layer-wise prompt tokens \( z_i \) sampled from the posterior are concatenated to text encoder inputs at layers \( J, \dots, J+H-1 \).
- **Visual-side Prompts:** Optional deterministic visual prompts \( \tilde z_i \) may be injected into the image encoder, but uncertainty modeling is reserved for the text-side.
- **Prediction:** Prompted image and text features are projected and scored by cosine similarity, with softmax yielding class probabilities.

The architecture is designed to support efficient end-to-end training while retaining parameter efficiency [2511.22664].

## 5. Class-Aware Priors and Semantic Consistency

A distinct innovation in VaMP is the class-aware prior construction, where:

\[
p_\theta(z\mid x, y) = \prod_{i=J}^{J+H-1} \mathcal N(z_i; \hat\mu_i, \mathrm{diag}(\hat\sigma_i^2)), \quad [\hat\mu_i, \hat\sigma_i] = \psi_i(o_y)
\]

By computing \( o_y \) as the empirical mean of all embeddings from a class and mapping it through small prior MLPs, VaMP regularizes prompt representations such that samples from the same class are close in the latent space. This promotes intra-class coherence while supporting inter-class discrimination. Compared to standard Gaussian priors, this yields a measurable gain in downstream generalization metrics [2511.22664].

## 6. Empirical Results: Generalization, Robustness, and Efficiency

### Few-Shot and Domain Generalization
In evaluation across 11 few-shot (16-shot) vision-language settings, VaMP outperforms leading baselines such as MMRL, improving novel class accuracy from 77.16% to 78.67% and raising the harmonic mean by ≈1.17 points (81.20→82.37). On ImageNet variant domain generalization, average top-1 accuracy is 61.73% (+1.20% vs. MMRL), with similar gains on cross-dataset transfer (10 unseen datasets: 67.74% vs. 67.25%).

### Ablations and Uncertainty Analysis
Ablation studies demonstrate that sample-specific, probabilistic prompts (vs. deterministic or task-specific) yield a +1.04–1.28 HM gain. The use of class-aware priors rather than standard Gaussians contributes a +0.55 harmonic mean improvement [2511.22664].

### Latency and Scalability
With \( S=10 \) Monte Carlo samples at inference, the increase in latency is marginal (0.8 ms/image on NVIDIA V100), while measurable improvements in harmonic mean are achieved. The method is compatible with EVA-CLIP, SigLIP, SigLIP2, and other recent CLIP-style backbones.

### Task Extensions
VaMP demonstrates successful adaptation to open-vocabulary segmentation (CAT-Seg, +1.1 mIoU) and action recognition (FROSTER, +0.6 HM), indicating architectural generality [2511.22664].

## 7. Relationship to Variational Prompt Modeling and Future Directions

Prior work on modeling prompt distributional robustness, such as MVP (Modeling Variants of Prompts) [2503.08229], focuses on invariance to prompt template structure by decoupling templates and class names, then using a VAE to encode template structural variation. MVP's approach mitigates prompt sensitivity in natural language templates but does not address instance-specific, image-conditioned adaptation or uncertainty modeling. By contrast, VaMP broadens the variational principle: it enables prompt representations to be adapted per input instance, integrates a class-aware latent space, and unifies prompt tuning with semantic regularization.

Potential extensions for VaMP include more expressive priors (e.g., with normalizing flows), dynamic updating of class prototypes, and expanded application to multi-modal reasoning, compositional prompts, and adversarial robustness. This suggests that a unifying variational framework for both prompt robustness and input-aware adaptation is a promising trajectory for future vision-language research [2511.22664][2503.08229].

Source: https://www.emergentmind.com/topics/variational-multi-modal-prompt-learning-vamp