VLPrompt: Multi-Modal Prompting Techniques

• VLPrompt is a family of prompt-based vision-language methods that integrate structured textual and visual cues to leverage language priors and enhance model performance.
• It employs diverse prompt representations such as LLM-generated language prompts, learnable tokens, and factorized embeddings to tailor adaptation strategies.
• Empirical results demonstrate the effectiveness of VLPrompt in tasks like panoptic scene graph generation, federated CLIP adaptation, and generalized category discovery.

Searching arXiv for papers that explicitly use or contextualize “VLPrompt” and closely related vision-language prompting frameworks. VLPrompt denotes a family of prompt-based vision-language methods rather than a single standardized algorithm. In recent arXiv literature, the name is used for an attention-based Vision-Language Prompting model for panoptic scene graph generation, for Vision-Language Prompt Tuning in federated CLIP adaptation, and for a CLIP-based generalized category discovery framework under domain shifts. Across these usages, the common operation is to inject additional textual, visual, or cross-modal prompt structure so that pretrained multimodal models can exploit language priors, adapt under limited supervision, or remain robust under heterogeneity and shift (Zhou et al., 2023, 2505.23024, Wang et al., 29 Apr 2026).

1. Terminological scope

The label “VLPrompt” is used in multiple, task-specific senses. In panoptic scene graph generation, it names a model that combines vision features from panoptic segmentation with language features generated by an LLM for relation prediction, explicitly targeting the long-tail problem among relations. In federated learning, the paper does not introduce “VLPrompt” as a standalone algorithm name in the title; instead, it uses VLPT, or Vision-Language Prompt Tuning, for the joint use of vision prompt learning and language prompt learning in CLIP-style federated adaptation. In generalized category discovery, VLPrompt is the vision-language extension of the HiLo/HLPrompt lineage and combines factorized textual prompts, cross-modal consistency regularization, and CLIP-based adaptation under simultaneous semantic and domain shifts (Zhou et al., 2023, 2505.23024, Wang et al., 29 Apr 2026).

Usage	Task	Distinguishing prompt form
VLPrompt (Zhou et al., 2023)	Panoptic scene graph generation	LLM-generated pair-level and triplet-level language prompts fused with visual features
VLPT / Vision-Language Prompt Tuning (2505.23024)	Federated prompt learning for CLIP	Joint use of language prompt learning and vision prompt learning
VLPrompt (Wang et al., 29 Apr 2026)	Generalized category discovery under domain shifts	Factorized textual prompts with cross-modal consistency and spatial prompting

This terminological multiplicity is important because identical surface naming hides substantially different objectives, losses, and deployment settings. A plausible implication is that “VLPrompt” functions less as a single method family with one canonical architecture than as a reusable label for prompt-centric vision-language adaptation.

2. Prompt representations and architectural patterns

Across the cited papers, prompt representations range from natural-language descriptions to learnable tokens and factorized embeddings. In panoptic scene graph generation, VLPrompt uses two language prompt types generated by an LLM: a Relation Proposer Prompt for subject-object pairs and a Relation Judger Prompt for subject-object-relation triplets; these are embedded and fused with visual pair features through transformer decoder blocks (Zhou et al., 2023). In federated CLIP adaptation, Language Prompt Learning prepends learnable prompt tokens $\{v_1,\dots,v_L\}$ to the class token in the text input, while Vision Prompt Learning inserts learnable prompt vectors $\{u_1,\dots,u_L\}$ into the visual input; VLPT jointly uses both prompt types when label skew and domain shift coexist (2505.23024). In generalized category discovery, VLPrompt learns factorized textual prompts composed of shared context embeddings $\boldsymbol{\Theta}_{\text{ctx}}$ and category-specific embeddings $\boldsymbol{\Gamma}$, reducing prompt parameters from $K(N+1)D$ to $(N+K)D$ (Wang et al., 29 Apr 2026).

Related prompt-learning work shows how wide this design space has become. ProMPT uses class-conditional vision prompting, instance-conditional text prompting, and feature filtering inside a recurrent Multi-modal Iterative Evolution module (Qiu et al., 2024). APLe learns vision and language prompts independently in a sequential token-wise manner and then performs multi-modal token adaptation (Cao et al., 2024). APoLLo combines deep prompts with cross-attention-based adapter layers and consistency regularization on augmented inputs (Chowdhury et al., 2023). SDPT places a single set of unified prototype tokens in the fusion space of fusion-based VLPMs and maps them back into the text and image input spaces through fixed inverse linear projections (Zhou et al., 2024). MMLoP factorizes deep multi-modal prompts into low-rank components and reports only $11.5\text{K}$ trainable parameters (Ghiasvand et al., 24 Feb 2026). VaMP turns text-side prompts into latent variables sampled from a learned posterior and regularized by a class-aware prior (Cheng et al., 27 Nov 2025).

These variants suggest that the central modeling choice in VLPrompt-style systems is not merely whether prompting is used, but where the prompt lives: text token space, visual token space, fused cross-modal space, image space, or an LLM-mediated semantic channel.

3. Objectives, optimization, and alignment mechanisms

The optimization logic of VLPrompt depends on the task, but the dominant theme is alignment: client-to-server alignment in federated learning, image-to-text alignment in CLIP-style transfer, or vision-to-language alignment in relation reasoning.

In federated prompt learning, the protocol is explicitly a three-stage loop of distribution, optimization, and aggregation. The server sends the current global prompt $\mathbf{P}^g$ to each client, the client initializes $\mathbf{P}^m \leftarrow \mathbf{P}^g$, optimizes it on local private data $D^m$, and the server aggregates prompts by a weighted FedAvg-style update, $\{u_1,\dots,u_L\}$0. The underlying CLIP model is trained with the symmetric contrastive objective $\{u_1,\dots,u_L\}$1 (2505.23024).

In generalized category discovery, VLPrompt uses a composite objective

$\{u_1,\dots,u_L\}$2

Here $\{u_1,\dots,u_L\}$3 is semi-supervised classification over known and pseudo-labeled unknown categories, $\{u_1,\dots,u_L\}$4 is entropy maximization to prevent pseudo-label collapse, $\{u_1,\dots,u_L\}$5 minimizes mutual information between early and late visual features, $\{u_1,\dots,u_L\}$6 is a symmetric KL-based vision-language consistency term, and $\{u_1,\dots,u_L\}$7 extends PatchMix to both modalities (Wang et al., 29 Apr 2026).

In panoptic scene graph generation, the key optimization target is relation prediction over all ordered subject-object pairs. VLPrompt builds a vision prompting feature $\{u_1,\dots,u_L\}$8 from mask-pooled object features and spatial descriptors, then decodes against RP-language and RJ-language features to produce $\{u_1,\dots,u_L\}$9 and $\boldsymbol{\Theta}_{\text{ctx}}$0. Final prediction is fused through

$\boldsymbol{\Theta}_{\text{ctx}}$1

and the total loss is $\boldsymbol{\Theta}_{\text{ctx}}$2, where $\boldsymbol{\Theta}_{\text{ctx}}$3 is a binary cross-entropy loss applied to $\boldsymbol{\Theta}_{\text{ctx}}$4, $\boldsymbol{\Theta}_{\text{ctx}}$5, and the fused output $\boldsymbol{\Theta}_{\text{ctx}}$6 (Zhou et al., 2023).

A recurring pattern across broader multi-modal prompting research is that prompt learning is rarely left as a bare input-layer perturbation. ProVP-Ref regularizes prompted image features against the frozen CLIP feature distribution through contrastive feature re-formation (Xu et al., 2023). APoLLo adds intra-modal consistency on augmented branches (Chowdhury et al., 2023). MMLoP anchors prompted representations to zero-shot CLIP features through a self-regulating consistency loss and then removes global embedding drift through uniform drift correction (Ghiasvand et al., 24 Feb 2026). VaMP reframes prompt tuning as variational inference over latent prompts and optimizes an ELBO with reparameterized sampling (Cheng et al., 27 Nov 2025). This suggests that modern VLPrompt systems are increasingly defined by the regularizers that preserve pretrained geometry.

4. Empirical behavior across applications

The principal empirical findings differ by task, but the reported results consistently emphasize performance under difficult regimes: long-tail relations, non-IID federation, and domain-shifted discovery.

Setting	Main empirical pattern	Representative result
PSG relation prediction (Zhou et al., 2023)	Language priors improve rare relations	On PSG, VLPrompt reaches $\boldsymbol{\Theta}_{\text{ctx}}$7 and $\boldsymbol{\Theta}_{\text{ctx}}$8
Federated CLIP adaptation (2505.23024)	VPT dominates under label skew; LPT dominates under domain shift; combined VLPT is best in mixed heterogeneity	On CIFAR-100 label skew, VPT reaches about $\boldsymbol{\Theta}_{\text{ctx}}$9–$\boldsymbol{\Gamma}$0, while LPT is around $\boldsymbol{\Gamma}$1–$\boldsymbol{\Gamma}$2
GCD under domain shifts (Wang et al., 29 Apr 2026)	VLPrompt is strongest on large style/domain gaps; HLPrompt can be stronger on corruption-heavy SSB-C	On DomainNet-GCD, Real+Painting reaches $\boldsymbol{\Gamma}$3 All

In panoptic scene graph generation, the central result is the jump in mean recall. On PSG, the previous method HiLo reports $\boldsymbol{\Gamma}$4 and $\boldsymbol{\Gamma}$5, while VLPrompt reports $\boldsymbol{\Gamma}$6 and $\boldsymbol{\Gamma}$7, respectively. The large $\boldsymbol{\Gamma}$8 gain is especially salient because mean recall is sensitive to rare relations, which is precisely the regime where RP and RJ language prompting are intended to help (Zhou et al., 2023).

In federated prompt learning, the headline pattern is modality-specific robustness to distinct heterogeneity types. Under label skew implemented by Dirichlet partitioning, visual prompt learning consistently outperforms language prompt learning on CIFAR-100 across $\boldsymbol{\Gamma}$9. Under domain shift on Office-Home, Office31, and DomainNet, language prompt learning usually performs better than visual prompt learning. When label skew and domain shift coexist, VLPT, the combined vision-language prompt strategy, is consistently best, especially on DomainNet (2505.23024).

In generalized category discovery, VLPrompt is strongest on large style gaps in DomainNet-GCD: Real+Painting reaches $K(N+1)D$0 All versus $K(N+1)D$1 for HLPrompt and $K(N+1)D$2 for HiLo; Real+Sketch reaches $K(N+1)D$3 All versus $K(N+1)D$4 for HLPrompt; Real+Clipart reaches $K(N+1)D$5 All versus $K(N+1)D$6 for HLPrompt. The same paper explicitly notes that on SSB-C, HLPrompt is often stronger than VLPrompt, indicating complementary strengths between pure-vision disentanglement with foreground prompting and CLIP-based cross-modal priors (Wang et al., 29 Apr 2026).

Broader prompt-learning results show similar adaptation–generalization trade-offs. ProVP-Ref reports an average base-to-novel harmonic mean of $K(N+1)D$7 across 11 datasets (Xu et al., 2023). MMLoP reports $K(N+1)D$8 harmonic mean with $K(N+1)D$9 trainable parameters (Ghiasvand et al., 24 Feb 2026). VaMP reports $(N+K)D$0 harmonic mean and $(N+K)D$1 average novel accuracy (Cheng et al., 27 Nov 2025). SDPT reports COCO mAP $(N+K)D$2, LVIS AP $(N+K)D$3, and ODinW13 full-shot $(N+K)D$4 with $(N+K)D$5 trainable parameters at $(N+K)D$6 (Zhou et al., 2024). These neighboring results suggest that VLPrompt sits inside a rapidly diversifying parameter-efficient adaptation literature rather than outside it.

5. Robustness, efficiency, and failure modes

Robustness claims in VLPrompt-style systems are strongly task-dependent. In federated prompt learning, increasing communication rounds yields only marginal improvements; weighted and equal aggregation both work well, with weighted aggregation slightly better under label skew and equal aggregation slightly better under domain shift; varying the number of clients does not catastrophically hurt performance; and increasing prompt length from 16 to 32, 64, or 128 gives only modest gains. The paper’s practical recommendation is therefore conditional: if the main challenge is class imbalance, visual prompts are preferred; if the main challenge is domain mismatch, language prompts are preferred; if both coexist, combined vision-language prompts are recommended (2505.23024).

The generalized category discovery VLPrompt paper is explicit about limitations. It assumes domain-class independence, assumes the number of classes $(N+K)D$7 is known, and notes that very large domain gaps remain hard. It also reports that semantic-aware spatial prompting that works well in HLPrompt is less stable for CLIP because CLIP’s attention is more globally distributed and more sensitive to disruptive masking; for that reason, VLPrompt prefers boundary-based spatial prompting (Wang et al., 29 Apr 2026).

The panoptic scene graph VLPrompt paper identifies several failure modes: segmentation errors propagate into relation prediction, LLM outputs may be imperfect or inconsistent, and stuff-stuff or stuff-thing pairs are harder because models tend to default to spatial relations. Efficiency is also nontrivial: VLPrompt reports $(N+K)D$8 GFLOPs, $(N+K)D$9G params, and $11.5\text{K}$0 ms inference, compared with HiLo’s $11.5\text{K}$1 GFLOPs, $11.5\text{K}$2G params, and $11.5\text{K}$3 ms inference (Zhou et al., 2023).

Robustness to prompt configuration is itself an active research target. APLe reports average accuracy $11.5\text{K}$4 with standard deviation $11.5\text{K}$5 across prompt-length experiments, versus $11.5\text{K}$6 and $11.5\text{K}$7 for MaPLe, and presents this as evidence of reduced prompt-length sensitivity (Cao et al., 2024). MMLoP is motivated by the parameter explosion of deep multi-modal prompting and responds with low-rank factorization plus regularization (Ghiasvand et al., 24 Feb 2026). This suggests that the mature form of VLPrompt is no longer just “add prompts,” but “add prompts while constraining instability, drift, and parameter growth.”

6. Relation to the broader visual-language prompting ecosystem

The broader literature shows that VLPrompt is part of a wider shift from static, hand-crafted textual templates toward explicit, structured, and often image-conditional prompt mechanisms. Some of these prompts are not latent tokens at all. VRPTEST formalizes visual referring prompting and shows that prompt strategy can change large multimodal model accuracy by $11.5\text{K}$8 to $11.5\text{K}$9 across a benchmark of $\mathbf{P}^g$0 image-question pairs (Li et al., 2023). MedVP uses explicit image-space markers such as rectangles, scribbles, and ellipses to guide medical VLMs toward clinically relevant regions and reports VQA-RAD $\mathbf{P}^g$1 open and $\mathbf{P}^g$2 closed performance (Zhu et al., 4 Jan 2025). VPP-LLaVA injects a global axis-like visual position prompt and local DETR-based position-aware queries, and improves ReferIt zero-shot transfer from $\mathbf{P}^g$3 to $\mathbf{P}^g$4 on val/test for the 7B model (Tang et al., 19 Mar 2025). BBVPE treats visual prompt engineering as a black-box routing problem over object-based overlays and reduces hallucination on POPE and CHAIR (Woo et al., 30 Apr 2025). VAPS uses a learnable visual prompt repository and similarity-based retrieval to adapt CLIP for compositional zero-shot learning (Stein et al., 27 Feb 2025).

Within this larger ecosystem, VLPrompt can be understood as the vision-language branch of prompt learning in which prompts are not merely lexical prefixes but carriers of structure: relation priors from LLMs, factorized class anchors, federated lightweight adapters, uncertainty-aware latent variables, or explicit image-space guidance. A common misconception is to treat “VLPrompt” as the name of one canonical model. The literature instead shows a family resemblance: multiple methods share the prompt-centric adaptation principle while differing sharply in supervision regime, prompt parameterization, and target task (Zhou et al., 2023, 2505.23024, Wang et al., 29 Apr 2026).