UniPrompt: Unified Vision-Language Prompt Tuning

• UniPrompt is a unified prompt tuning framework for CLIP-style vision-language models that coordinates text and visual prompts via a shared seed and lightweight Transformer block.
• It addresses the limitations of unimodal tuning by balancing high intra-class visual variances and low inter-class text variances, improving few-shot learning and domain generalization.
• The method integrates unified prompts into CLIP’s pipeline, tuning only the prompt generator while keeping the image and text encoders frozen for efficiency.

UniPrompt, introduced as Unified Prompt Tuning (UPT) in “Unified Vision and Language Prompt Learning,” is a prompt-tuning framework for adapting frozen CLIP-style vision-LLMs by jointly learning text and visual prompts through a tiny shared network (Zang et al., 2022). It was motivated by a systematic comparison of text prompt tuning and visual prompt tuning: text prompt tuning fails on data with high intra-class visual variances, whereas visual prompt tuning cannot handle low inter-class variances. UPT addresses this mismatch by starting from a single seed prompt and using a lightweight Transformer block to generate coordinated prompts for both modalities, yielding a better trade-off on few-shot learning and domain generalization benchmarks.

1. Motivation, setting, and failure modes

UniPrompt is situated in the parameter- and data-efficient transfer setting for large vision-LLMs such as CLIP. In this setting, the image encoder $\phi$ and text encoder $\psi$ remain frozen, while adaptation is performed by modifying the model’s input space with prompts rather than by full fine-tuning. The paper studies two representative unimodal prompt tuning methods: text prompt tuning, exemplified by CoOp, and visual prompt tuning, exemplified by VPT (Zang et al., 2022).

Text prompt tuning learns a small set of continuous tokens $P_T \in \mathbb R^{d\times m}$ that are prepended to a hand-crafted template such as “a photo of a,” after which the frozen text encoder produces class-conditional prototypes. Visual prompt tuning injects learnable patch tokens $P_V^i \in \mathbb R^{d\times n}$ into one or more layers $i$ of a frozen ViT image encoder. These two mechanisms operate in different parts of the CLIP pipeline and exhibit different failure modes.

The central empirical observation is that neither unimodal approach performs consistently well. If images have high intra-class variance, such as satellite or food data, learned text prompts cannot move the text prototypes far enough apart to help $\phi(x)$ sort ambiguous visuals, and CoOp yields near-zero improvement. If class names are semantically similar, corresponding to low inter-class text-embedding variance, a visual encoder tweak alone cannot overcome the small angles between static text prototypes, so VPT becomes unreliable. The rationale for unification is therefore geometric as well as statistical: datasets occupy different regions of an intra-class visual variance versus inter-class text variance plane, and a single-modality tuner cannot be expected to cover all regimes.

2. Prompt parameterization and joint generator

UniPrompt retains the standard parameterizations of text and visual prompts, then couples them through a shared generator. For text prompts, the learnable context is

$$P_T = [p_{T,1},\dots,p_{T,m}] \in \mathbb R^{d\times m},$$

and for class $c$ the prompted text input is

$$\hat t_c = [\,p_{T,1},\dots,p_{T,m},\,\mathrm{Embed}({\tt CLASS}_c)\,] \in \mathbb R^{d\times (m+1)}.$$

The frozen text encoder yields the class prototype

$$w_c = \psi(\hat t_c) \in \mathbb R^d \quad (c=1\ldots k).$$

For visual prompts, the original Transformer tokens at layer $\psi$0 are

$\psi$1

and the inserted prompt tokens are

$\psi$2

forming

$\psi$3

The distinctive element of UPT is the unified prompt generator. Instead of independently optimizing $\psi$4 and $\psi$5, the framework begins with a single seed prompt

$\psi$6

shared across all layers and modalities. A lightweight Transformer block $\psi$7, consisting of one self-attention layer, one feed-forward network, and layer norms, transforms the seed prompt as

$\psi$8

For layer $\psi$9, $P_T \in \mathbb R^{d\times m}$0 is sliced into

$P_T \in \mathbb R^{d\times m}$1

which serve as that layer’s text and visual prompts. The entire generation process is written as

$P_T \in \mathbb R^{d\times m}$2

where $P_T \in \mathbb R^{d\times m}$3 is the number of layers to modulate. Both $P_T \in \mathbb R^{d\times m}$4 and $P_T \in \mathbb R^{d\times m}$5 are trainable, whereas $P_T \in \mathbb R^{d\times m}$6 and $P_T \in \mathbb R^{d\times m}$7 remain frozen (Zang et al., 2022).

This construction is intentionally minimal. The design ties the prompt collections together through a shared seed and permits information flow between modalities through self-attention in $P_T \in \mathbb R^{d\times m}$8. The paper’s interpretation is that this lets the text and visual prompts “talk” to each other, producing synergy rather than conflicting updates.

3. Integration into CLIP and optimization objective

UPT inserts its generated prompts directly into CLIP’s standard image-text classification pipeline. On the text side, at layer $P_T \in \mathbb R^{d\times m}$9 of $P_V^i \in \mathbb R^{d\times n}$0, the model prepends $P_V^i \in \mathbb R^{d\times n}$1 before the class-name embedding. On the visual side, at each inserted layer $P_V^i \in \mathbb R^{d\times n}$2 of $P_V^i \in \mathbb R^{d\times n}$3, the model splices $P_V^i \in \mathbb R^{d\times n}$4 among the patch tokens before self-attention (Zang et al., 2022).

Classification uses cosine-similarity logits with temperature $P_V^i \in \mathbb R^{d\times n}$5:

$P_V^i \in \mathbb R^{d\times n}$6

Given a few-shot training set $P_V^i \in \mathbb R^{d\times n}$7, the loss is the standard cross-entropy

$P_V^i \in \mathbb R^{d\times n}$8

Optimization is carried out only with respect to $P_V^i \in \mathbb R^{d\times n}$9. This means that the learnable adaptation capacity is concentrated entirely in the seed prompt and the tiny prompt generator, while the pretrained CLIP backbone is preserved.

In the reported implementation, few-shot learning is evaluated at 1, 2, 4, 8, and 16 shots per class on 11 standard vision benchmarks: ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVC-Aircraft, SUN397, DTD, EuroSAT, and UCF101. Testing is performed on the full validation split. Training uses SGD with learning rate $i$0 decayed by a cosine schedule and batch size $i$1. Prompt size is fixed at $i$2, corresponding to 4 context vectors for text and 4 patch tokens for vision per layer. The learnable parameter count is described as roughly $i$3 for the seed prompt plus $i$4 for the small Transformer block $i$5, with no extra regularization beyond standard weight decay and layer norm in $i$6.

4. Empirical results on few-shot learning and domain generalization

The main empirical claim is that UPT achieves a stronger trade-off than unimodal prompt tuning on both few-shot classification and domain generalization. On the 16-shot average over three runs across 11 datasets, UPT exceeds zero-shot CLIP, CoOp, and both shallow and deep variants of VPT (Zang et al., 2022).

Method	16-shot average
Zero-shot CLIP	70.0% (approx.)
CoOp	78.24%
VPT-shallow	73.18%
VPT-deep	79.39%
UPT	81.44%

On the ImageNet domain generalization benchmark, training is performed with 16 shots on ImageNet and evaluation is then carried out on IN-V2, IN-Sketch, IN-A, and IN-R. UPT achieves both the highest source-domain accuracy and the highest OOD average.

Method	Source IN	OOD avg
CoOp	71.51	59.28
CoCoOp	71.02	59.91
VPT-shallow	68.98	58.27
VPT-deep	70.57	57.40
UPT	72.63	59.98

The ablation study isolates the importance of the multi-modal design. Jointly tuning CoOp and VPT as two disjoint prompt sets reaches 78.70%. Sharing one prompt directly, with no self-attention or feed-forward transformation, reaches 77.88%. Replacing the Transformer block with a two-layer MLP reaches 78.73%. The full UPT design with self-attention and FFN reaches 81.44%. This pattern indicates that unification alone is insufficient; the specific cross-modal coupling mechanism matters.

The paper also reports two practical tendencies. Increasing prompt length $i$7 beyond 4 yields diminishing gains at the cost of more parameters. Injecting unified prompts into more layers improves few-shot accuracy but can mildly harm OOD generalization, and a single shared layer often suffices in practice.

5. Interpretation, qualitative behavior, and practical considerations

The paper’s explanation for UPT’s performance is explicitly complementary across modalities. When visual features are “noisy,” corresponding to high intra-class variance, text prompts adapt the semantic prototypes so that the induced decision regions become more separable. When text prototypes are “clustered,” corresponding to low inter-class separation, visual prompts sharpen the image embeddings in complementary directions. The self-attention in the generator is presented as the mechanism that coordinates these updates rather than letting them compete (Zang et al., 2022).

Qualitative analysis is summarized through DINO-style attention-map visualizations. In these visualizations, UPT’s learned visual prompts attend more strongly to object-centric patches, such as bird bodies and car wheels, than VPT’s. Its text prompts are described as shifting class names along semantically meaningful axes such as color and shape. This suggests that the model is not merely adding capacity, but rebalancing the image-text interface in a class-sensitive manner.

The implementation described in the details is correspondingly lightweight. A single Python module defines UnifiedPrompt with members P and TransformerBlock. A wrapper around CLIP’s encode_image and encode_text intercepts inputs at designated layers to splice in prompts. The training loop is identical to CoOp in that $i$8 and $i$9 are frozen and only prompt modules are tuned. Reproduction tips specify a ViT-B/16 backbone, prompt length 4, Transformer hidden size matching $\phi(x)$0, SGD with initial learning rate $\phi(x)$1, cosine decay, batch size $\phi(x)$2, and training for approximately 100 epochs in the 16-shot regime, described as about 10 minutes on one GPU.

Some extrapolations in the details are explicitly prospective rather than experimentally established. For example, for multi-label or detection tasks, the description states that one can splice unified prompts into the text encoder’s queries and the RoI heads of the vision encoder in exactly the same manner. A plausible implication is that the framework is intended as a general prompt-generation mechanism rather than as a dataset-specific CLIP heuristic.

6. Terminological scope and related uses of “UniPrompt”

The name “UniPrompt” is not unique to the visual-language framework of (Zang et al., 2022). In multilingual NLP, “Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt” uses a unified prompt for all languages, implemented through a model-based, language-agnostic prompt representation derived from XLM-RoBERTa-base and combined with soft label-word initialization (Huang et al., 2022). In prompt optimization, “Task Facet Learning: A Structured Approach to Prompt Optimization” uses the name UniPrompt for an algorithm that clusters the input space and edits prompts as semantic sections corresponding to task facets (Juneja et al., 2024).

Related nomenclature also appears in “On Unified Prompt Tuning for Request Quality Assurance in Public Code Review,” which proposes UniPCR under a masked-language-model formulation with text prompt tuning and code prefix tuning (Chen et al., 2024). In few-shot text classification, “Towards Unified Prompt Tuning for Few-shot Text Classification” introduces UPT with the Prompt-Options-Verbalizer paradigm and Knowledge-enhanced Selective Masked Language Modeling (Wang et al., 2022). In graph learning, “One Prompt Fits All: Universal Graph Adaptation for Pretrained Models” introduces UniPrompt as an input-level graph prompt based on a learnable adjacency matrix and bootstrapped fusion with the original topology (Huang et al., 26 Sep 2025).

Within this broader naming landscape, the 2022 vision-language UniPrompt is specifically the framework that unifies visual and textual prompt learning for frozen CLIP-style models through a shared seed prompt and a lightweight Transformer generator. Its defining contribution is not merely multimodal prompting, but the claim that cross-modal prompt coordination is necessary because the error modes of text-only and vision-only prompting are statistically complementary.