Visual Prompt Optimization

• Visual prompt optimization is a set of methods that use optimized visual cues to adapt frozen models with parameter-efficient strategies.
• Techniques include pixel-level prompts, token-space tuning, and cross-modal adaptations that align visual inputs with textual contexts.
• Applications range from image classification to text-to-image and text-to-3D synthesis, enhancing model robustness and performance.

Visual prompt optimization denotes a family of parameter-efficient adaptation methods in which a frozen or largely frozen model is steered by optimizing prompts associated with visual inputs, visual tokens, or visually grounded conditioning signals. In the literature, the term spans universal pixel-space prompts for classification, learnable prompt tokens in ViTs, attention-guided “functional” visual prompts in VLM debiasing, before–after image pairs in diffusion-based editing, visual-feedback loops for text-to-image prompt engineering, and 2D visual prompts for text-to-3D generation (Wu et al., 2022, Han et al., 2023, Gupta et al., 5 Jan 2026, Xu et al., 7 Jan 2025, Wu et al., 29 Jun 2025, Chen et al., 2024). Across these settings, the optimization target is not only prompt content but also prompt placement, label mapping, prompt distribution across layers, and joint visual–textual conditioning.

1. Scope and formal settings

In recognition and transfer, visual prompting usually means adapting a frozen source model by learning a universal input transformation and, in some formulations, an output mapping from source labels to target labels. A representative formulation writes the prompted image as

$$\tilde x_t = \mathcal{P}_p(x_t) = \text{InputScaling}_p(x_t) + \mathcal{M}_p \odot \sigma(\delta),$$

with frozen backbone $f_{\theta_s}$, trainable prompt $\delta$, and output mapping $g_\phi$ from source logits to target logits (Tsao et al., 2023). In CLIP-like VLMs, zero-shot prediction remains the usual compatibility computation between image and text embeddings,

$$p(y|x) = \mathrm{softmax}\!\left(\tau \cdot \mathrm{sim}(f_v(x), f_t(t_y))\right),$$

so prompt optimization can act on the visual side, the textual side, or both (Gupta et al., 5 Jan 2026).

The same phrase acquires broader meanings in generative settings. In diffusion-based image editing, a visual prompt is explicitly defined as a before–after image pair $(\mathbf{x}_0^b,\mathbf{x}_0^a)$ that encodes an edit transformation and is optimized into a reusable text embedding (Xu et al., 7 Jan 2025). In text-to-image synthesis, prompt optimization can be driven by visual feedback from generated images, using question answering over atomic concepts to revise the text prompt itself (Wu et al., 29 Jun 2025). In text-to-3D generation, a 2D image serves as a visual prompt that conditions SDS-style optimization and explicit visual-consistency rewards (Chen et al., 2024). The field is therefore best understood as a collection of closely related optimization problems that all use visual structure to steer frozen or minimally updated models.

2. Pixel-level prompting, input reprogramming, and output mapping

Pixel-level visual prompting treats the prompt as an image-space object. Early formulations emphasized model reprogramming with a universal prompt $\boldsymbol{\delta}$ added to every target image while keeping the source model frozen, but the label mapping from source classes to target classes was often fixed heuristically. The label-mapping perspective showed that mapping quality, assessed through mapping precision and explanation, can consistently improve the effectiveness of visual prompting, and ILM-VP formulated training as alternating optimization between prompt updates and iterative remapping of source labels to target labels (Chen et al., 2022). This established that prompt optimization is partly an output-space problem rather than a purely input-space one.

AutoVP systematized this perspective by defining a joint design space over prompt optimization, pre-trained model selection, and output mapping strategies, including FreqMap, SemanticMap, IterMap, and FullyMap. Its search space spans 222 configurations and the reported gains reach up to 6.7% over prior VP methods and a maximum performance increase of 27.5% compared to linear probing (Tsao et al., 2023). A central implication is that “visual prompt optimization” is not exhausted by learning prompt pixels; it also includes selecting the backbone, the prompt/image scale trade-off, and the label-space interface.

The input-side design was sharpened by work that argued the strategy of reconciling the prompt and the image matters. “Unleashing the Power of Visual Prompting At the Pixel Level” replaces additive perturbation by an independent prompt that wraps around a properly shrinked image, then re-introduces input diversity and gradient normalization from transferable adversarial examples. Using CLIP, it reports 82.8% average accuracy across 12 popular classification datasets, surpassing the prior art by +5.6% and linear probing by +2.1% (Wu et al., 2022). The operative lesson is that non-overlap between prompt and image content, together with stable optimization dynamics, materially changes what a pixel-space prompt can express.

LoR-VP pushes pixel-level prompting in a different direction by abandoning border-only prompting and learning a full-image low-rank prompt

$$\mathcal{P}(x) = \operatorname{Resize}_L(x) + B \cdot A,$$

with $B \in \mathbb{R}^{c \times L \times r}$ and $A \in \mathbb{R}^{c \times r \times L}$ (Jin et al., 2 Feb 2025). This factorization injects shared and patch-specific information across rows and columns of pixels, addressing the claim that pad prompting limits interaction with central patches and ignores shared structure across patches. Empirically, the method reports up to 6 times faster training times, 18 times fewer visual prompt parameters, and a 3.1% improvement in performance (Jin et al., 2 Feb 2025). Taken together, these works define pixel-level visual prompt optimization as joint control over prompt parameterization, prompt–image composition, and output remapping.

3. Token-space prompt tuning, adaptive experts, and prompt distribution

In transformer backbones, visual prompt optimization typically means appending learnable prompt tokens to the input sequence while freezing the backbone. For a ViT block, VPT augments the token sequence with $f_{\theta_s}$0, so queries, keys, and values are formed from the concatenation of image tokens and prompt tokens (Le et al., 31 Jan 2025). This architecture is parameter-efficient, but subsequent work identified several distinct optimization bottlenecks: prompt initialization, prompt length sensitivity, restricted expressiveness of static prompt tokens, and the choice of where prompt capacity should reside.

One line of work focuses on optimization inside attention. E$f_{\theta_s}$1VPT adds visual prompts at layer inputs and learnable key–value prompts inside self-attention, then prunes low-importance prompts via token-wise and segment-wise pruning. It reports 0.32% of model parameters on VTAB-1k while outperforming several state-of-the-art baselines (Han et al., 2023). CVPT modifies the interaction mechanism itself by computing cross-attention between prompt tokens and embedded tokens, using weight sharing from self-attention to avoid a large parameter increase; on VTAB-1K it outperforms VPT by over 4% in average accuracy (Huang et al., 2024). Both methods recast prompt optimization as a question of how prompts enter attention, not only how many prompt tokens are used.

A second line addresses prompt initialization and expressiveness. SPT begins from the observation that prompt tokens tend to share high mutual information with patch tokens during proficient training, and therefore initializes prompts with downstream token prototypes rather than random noise. It reports less than 0.4% learnable parameters, surpasses full fine-tuning in 19 out of 24 tasks, and substantially improves adaptation for self-supervised pretraining by at least 10% to 30% (Wang et al., 2024). VAPT goes further by interpreting each attention head as a mixture of experts and viewing VPT as the addition of constant, input-independent prompt experts. Its adaptive prompts become nonlinear functions of the current block features, and the paper states that VAPT surpasses fully fine-tuned baselines by 7.34% on VTAB-1K and 1.04% on FGVC while its theoretical analysis indicates optimal sample efficiency (Le et al., 31 Jan 2025). This suggests that a central limitation of static prompts is “restricted functional expressiveness,” not just parameter count.

A third line optimizes prompt allocation across depth. PRO-VPT formalizes adaptive distribution optimization as a nested problem in which prompt values and prompt locations across blocks must be optimized jointly. It uses iterative prompt relocation: identify and prune idle prompts, then relocate them to better blocks with a PPO agent. The method surpasses VPT by 1.6% average accuracy and leads prompt-based methods to state-of-the-art performance on VTAB-1k (Shang et al., 10 Mar 2025). The common theme across these transformer methods is that prompt optimization has decomposed into four coupled subproblems: token initialization, attention interface, prompt expressiveness, and prompt distribution.

4. Vision–language prompt optimization and bilateral adaptation

In VLMs, prompt optimization is increasingly bilateral rather than visual-only. BiPrompt is explicit on this point: visual debiasing is incomplete if textual anisotropy and priors remain fixed. Its bilateral prompt optimization framework combines structured attention-guided erasure on the visual side with Balanced Prompt Normalization on the textual side. The text prototype for class $f_{\theta_s}$2 is re-centered as

$f_{\theta_s}$3

while the visual side constructs foreground and background views via Grad-CAM masks and enforces consistency with the foreground and orthogonality with the background (Gupta et al., 5 Jan 2026). The paper interprets the combined effect as approximate minimization of $f_{\theta_s}$4, and reports consistent improvements in both average and worst-group accuracies over prior test-time debiasing methods (Gupta et al., 5 Jan 2026). Here the “visual prompt” is no longer a learnable patch or token, but a functional instruction implemented through structured erasure and prediction constraints.

A related textual line of work highlights that prompt optimization can harm generalization if it forgets essential general textual knowledge. KgCoOp constrains learned prompt embeddings toward the hand-crafted CLIP prompt embeddings by minimizing the discrepancy between learned and hand-crafted textual embeddings, with the explicit goal of improving unseen-class generalization (Yao et al., 2023). IPO replaces gradient-descent prompt vectors altogether with an interpretable LLM-based optimizer. Its Prompt Optimization Prompt stores past prompts with their performance metrics, and a large multimodal model generates image descriptions that provide visually grounded context. Across 11 datasets, IPO improves the accuracy of gradient-descent-based prompt learning methods while maintaining human-understandable prompts (Du et al., 2024). A plausible implication is that VLM prompt optimization is moving toward interpretable, dataset-specific, visually grounded language prompts rather than opaque continuous tokens alone.

5. Generative and cross-modal formulations

In diffusion-based image editing, visual prompt optimization takes an example-based form. “Textualize Visual Prompt for Image Editing via Diffusion Bridge” defines a visual prompt as a before–after image pair and optimizes a text embedding $f_{\theta_s}$5 so that a deterministic DDIM trajectory maps the latent inverted from the before-image to the after-image (Xu et al., 7 Jan 2025). The framework uses a diffusion bridge grounded in the probability-flow ODE view of diffusion, optimizes $f_{\theta_s}$6 with a time-aware per-timestep loss, and introduces differential attention control so that the learned embedding captures the edit transformation rather than the image content. The method reports, among other metrics, PSNR $f_{\theta_s}$7, SSIM $f_{\theta_s}$8, and LPIPS $f_{\theta_s}$9 in Table 1, together with stronger V-CLIP, DINO, VIE, and human scores than the baselines (Xu et al., 7 Jan 2025). In this regime, optimization transfers a visual example into prompt space.

Text-to-image prompt optimization has also become visually grounded. VisualPrompter is a training-free prompt engineering framework that first generates an image from the user prompt, then uses an automatic self-reflection module to identify missing concepts and a target-specific prompt optimization mechanism to revise the text in a fine-grained manner (Wu et al., 29 Jun 2025). Its feedback loop is explicit: prompt $\delta$0 image $\delta$1 DSG-style question answering over atomic concepts $\delta$2 prompt revision $\delta$3 final image. The paper states that it achieves new state-of-the-art performance on multiple benchmarks for text-image alignment and emphasizes that the framework is plug-and-play across multiple generative models (Wu et al., 29 Jun 2025). This shifts prompt optimization from static rewriting toward visually diagnosed semantic correction.

In text-to-3D, VP3D introduces a visual-prompt-guided SDS pipeline. A 2D image generated from the text acts as the visual prompt, is embedded into the diffusion conditioning space, and is paired with two additional differentiable rewards. The resulting objective is

$\delta$4

where the visual-consistency term aligns DINO features of the render and the visual prompt, and the human-feedback term uses ImageReward (Chen et al., 2024). The method also extends a single front-view prompt into left, right, and back prompts with Zero-1-to-3, so that prompt selection becomes view-dependent. The paper attributes higher visual fidelity, more detailed textures, and better semantic alignment on T$\delta$5Bench to this explicit 2D visual prompting strategy (Chen et al., 2024).

6. Evaluation regimes, limitations, and security

Evaluation protocols for visual prompt optimization are heterogeneous because the object being optimized differs across settings. Recognition work concentrates on VTAB-1K, FGVC, ImageNet-derived transfer, WILDS, and corruption benchmarks such as CIFAR-C, with metrics centered on average accuracy, worst-group accuracy, and cross-dataset generalization (Le et al., 31 Jan 2025, Wu et al., 2022, Gupta et al., 5 Jan 2026). Generative work uses PSNR, SSIM, LPIPS, V-CLIP, DINO, VIE, DSG, TIFA, AP, and mIoU depending on whether the task is image editing, text-image alignment, detection, or segmentation (Xu et al., 7 Jan 2025, Wu et al., 29 Jun 2025, Shang et al., 10 Mar 2025). This diversity complicates direct comparison, but it also clarifies that prompt optimization now spans recognition robustness, semantic faithfulness, visual fidelity, and even geometry.

Several recurring limitations emerge. Multiple methods depend on the quality of auxiliary structure: BiPrompt depends on Grad-CAM maps that must reasonably identify causal regions; Textualize Visual Prompt and VP3D depend on the prior of the underlying T2I model and on the quality of the generated or provided visual prompt; PRO-VPT still depends on the total prompt count even though it is more robust than VPT; and LoR-VP’s gains are tied to the inductive bias imposed by a low-rank full-image prompt (Gupta et al., 5 Jan 2026, Xu et al., 7 Jan 2025, Chen et al., 2024, Shang et al., 10 Mar 2025, Jin et al., 2 Feb 2025). A plausible synthesis is that optimization quality is increasingly bottlenecked by auxiliary modules—attention maps, view synthesis, LLM/VLM feedback, reward models—rather than by prompt parameters alone.

Security has also become part of the topic. “Prompt Backdoors in Visual Prompt Learning” studies a Visual Prompt as a Service setting in which a malicious provider returns a poisoned prompt instead of a benign one. The paper reports that poisoning $\delta$6 CIFAR10 training data leads to above $\delta$7 attack success rates with only negligible model accuracy drop by $\delta$8, and concludes that seven analyzed defenses are either ineffective or impractical to mitigate BadVisualPrompt (Huang et al., 2023). This indicates that visual prompt optimization is not only a PEFT problem but also an attack surface.

Taken together, the literature suggests three durable directions. First, prompt optimization is broadening from prompt content to prompt structure, routing, and cross-modal coordination. Second, visual grounding—through label mapping, patch-token prototypes, image descriptions, attention maps, generated views, or explicit visual prompts—has become the main route to improved robustness and generalization. Third, interpretability and security are now first-order concerns, because increasingly powerful prompt optimizers operate outside the backbone and can therefore be both modular and difficult to audit.