VPIP: Prompt-Guided Image Processing

• VPIP is a visual processing framework that uses paired input-target image prompts to define task-specific transformations.
• It employs dedicated prompt encoders and cross-attention modules to unify heterogeneous tasks like restoration, enhancement, and edge detection.
• VPIP methods vary in prompt representation and training regimes, delivering competitive accuracy and parameter efficiency across diverse vision challenges.

Visual Task Prompt-based Image Processing (VPIP) denotes a prompt-conditioned formulation of visual processing in which the task is specified by visual evidence rather than only by model parameters or textual labels. In its explicit low-level vision form, VPIP represents a task by an input-target image pair $[P_{\Omega_S}, P_{\Omega_T}]$, learns mappings $\mathcal{T}_{task}: \Omega_S \rightarrow \Omega_T$, and predicts outputs by conditioning an image-processing network on both the query image and the visual task prompt: $$I_{out} = \mathcal{F}(I_{in}, [P_{\Omega_S}, P_{\Omega_T}]; \Theta)$$ (Chen et al., 2024). Related work extends the same underlying idea to visual question-answering for image processing, inpainting-based in-context learning, prompt-aware multimodal encoding, retrieval with region prompts, visual-prompt image editing, and automated prompt discovery for LVLM perception (Liu et al., 2023, Xu et al., 2023, Wu et al., 2024, Nozawa et al., 2 Apr 2025, Xu et al., 7 Jan 2025, Kim et al., 17 Mar 2026).

1. Conceptual basis and historical emergence

The central motivation for VPIP is that unified visual processing is difficult when tasks differ not only in degradation type but also in output domain. The low-level vision literature formalizes this difficulty by distinguishing restoration, enhancement, edge detection, stylization, and feature extraction as mappings to different target domains rather than a single degradation-to-clean regime (Chen et al., 2024). This directly addresses a limitation of multi-task restoration systems such as AirNet and PromptIR, which are described as fundamentally designed for the degradation-to-clean setting $\Omega_{HQ}$, and a limitation of MAE-based prompting systems such as Painter and PromptGIP, which are described as tightly coupled to ViT and prone to suboptimal reconstruction quality, blocking artifacts, prompt-image sensitivity, and weaker handling of low-frequency tasks (Chen et al., 2024).

An important antecedent is PromptGIP, which reformulates image processing as a visual prompting question-answering problem. There the prompt is an input-output image pair, the query is a new image, and the model predicts the answer image by $Y_A = G(X_Q \mid \{P_Q, P_A\})$, enabling one model to cover image restoration, image enhancement, and image feature extraction tasks without task-specific finetuning (Liu et al., 2023). IMProv develops a related but distinct formulation, casting many computer vision problems as conditional image inpainting with visual examples, text instructions, or both provided at inference time (Xu et al., 2023).

This progression suggests a broader interpretation of VPIP: not merely a single architecture, but a family of methods in which task identity is carried by prompts embedded in the visual pipeline itself. The explicit VPIP framework introduced for GenLV makes that interpretation concrete for low-level vision, while adjacent work shows that the same design principle can be applied to classification, segmentation, retrieval, editing, and multimodal reasoning (Chen et al., 2024, Chen et al., 20 Jul 2025).

2. Prompt representations and task specification

Across the literature, VPIP systems differ primarily in how they encode the task prompt. PromptGIP uses one or more paired images $(P_Q, P_A)$ arranged as a $Q\!-\!A\!-\!Q\!-\!A$ sequence, arguing that this preserves question-answer adjacency and makes the transformation relation more explicit than Painter’s $Q\!-\!Q\!-\!A\!-\!A$ organization (Liu et al., 2023). The GenLV line adopts the same paired-example principle but makes it the formal task interface: the prompt is $[P_{\Omega_S}, P_{\Omega_T}]$, where one image exemplifies the source domain and the other the target domain (Chen et al., 2024). In image editing, "Textualize Visual Prompt for Image Editing via Diffusion Bridge" uses a before-image and after-image as the visual prompt, then optimizes a reusable text embedding that captures the transformation conveyed by that pair (Xu et al., 7 Jan 2025).

Other VPIP variants use region prompts rather than source-target exemplars. PHS supports point, box, and segmentation prompts, converts them into a binary prompt mask $\mathbf{M}_{\mathrm{input}}$, and uses the corresponding ROI token set $\mathcal{T}_{task}: \Omega_S \rightarrow \Omega_T$0 to select last-layer ViT attention heads aligned with the user’s intended object of interest (Nozawa et al., 2 Apr 2025). SEVEX defines a visual prompt as a combination of image-manipulation code and corresponding text prompts, so the prompt is executable rather than purely declarative (Kim et al., 17 Mar 2026). IMProv permits visual prompts, text prompts, or both, and explicitly argues that language removes ambiguity when visual examples alone under-specify the desired transformation (Xu et al., 2023). PIP-MM, by contrast, derives a prompt vector from the frozen LLM’s hidden state and inserts it into the ViT input as a prompt-conditioned class token, thereby making the image encoder itself prompt-aware (Wu et al., 2024).

System	Prompt representation	Primary setting
PromptGIP	input-output image pair in $\mathcal{T}_{task}: \Omega_S \rightarrow \Omega_T$1	universal image processing without task-specific finetuning
VPIP / GenLV	$\mathcal{T}_{task}: \Omega_S \rightarrow \Omega_T$2 paired visual task prompt	restoration, enhancement, edge detection, stylization, feature extraction
IMProv	visual examples, text instructions, or both	in-context inpainting for computer vision tasks
PHS	point, box, segmentation prompt mask	focus-oriented image retrieval
SEVEX	image-manipulation code plus text prompts	LVLM perception improvement

These prompt forms are not interchangeable. The paired-example format is used when the task is naturally expressed as a source-to-target transformation; region prompts are used when the task is object-focused; multimodal prompts are used when ambiguity must be reduced or instruction semantics matter. A plausible implication is that VPIP is best understood as a task-specification interface rather than a single prompt syntax.

3. Architectural realizations and interaction mechanisms

The explicit VPIP architecture contains three components: a main image-processing backbone, a prompt encoder network, and a prompt interaction mechanism (Chen et al., 2024). In GenLV, the backbone is X-Restormer, a U-shaped architecture with three downsampling and three upsampling stages, skip connections, and TSAB and SSAB blocks; the prompt encoder is a series of standard residual blocks with multiple downsampling operations; and interaction occurs through the Prompt Cross-Attention Block (PCAB) inserted into SSAB at the bottom of the U-shape architecture (Chen et al., 2024). In the scaled formulation, the attention is written as

$\mathcal{T}_{task}: \Omega_S \rightarrow \Omega_T$3

followed by

$\mathcal{T}_{task}: \Omega_S \rightarrow \Omega_T$4

so the input feature queries prompt features that separately encode source-domain and target-domain appearance (Chen et al., 20 Jul 2025).

PromptGIP instead uses a vanilla ViT-large backbone over four $\mathcal{T}_{task}: \Omega_S \rightarrow \Omega_T$5 images arranged in sequence and trains with a masked image modeling objective that masks answer patches at an 85% ratio (Liu et al., 2023). IMProv uses an encoder-decoder transformer similar to MAE-VQGAN, predicts discrete VQGAN tokens with cross-entropy loss, and injects text through cross-attention from image tokens to text tokens after each self-attention layer (Xu et al., 2023). These systems differ from VPIP chiefly in backbone flexibility: the GenLV papers emphasize that prompting should not be tied to ViT when pixel-level reconstruction is the main objective (Chen et al., 2024).

A distinct but related architectural principle appears in CVPT. That work diagnoses a weakness in Visual Prompt Tuning for vision: when prompt tokens are concatenated directly with image tokens, self-attention scales as $\mathcal{T}_{task}: \Omega_S \rightarrow \Omega_T$6, prompt tokens can absorb more than 80% of attention mass when many prompts are used, and the self-attention among embedded tokens is weakened (Huang et al., 2024). CVPT replaces prompt participation in shared self-attention with cross-attention after the frozen ViT self-attention block, using embedded image tokens as queries and prompt tokens as key-value sequences. The block is written as

$\mathcal{T}_{task}: \Omega_S \rightarrow \Omega_T$7

Although developed for parameter-efficient fine-tuning rather than low-level image-to-image translation, this design articulates a principle also seen in VPIP: prompts act as directed guidance signals rather than as competitors inside one shared attention pool (Huang et al., 2024).

PIP-MM pushes prompt conditioning earlier still. It vectorizes the prompt with a frozen LLM, aligns the resulting hidden state with a trainable MLP to form $\mathcal{T}_{task}: \Omega_S \rightarrow \Omega_T$8, and replaces the ViT CLS token so that prompt information participates in self-attention from the start of image encoding (Wu et al., 2024). PHS takes the opposite route: no retraining, no image alteration, and no prompt token insertion. Instead, it measures the alignment of each last-layer head with the prompt-defined ROI through $\mathcal{T}_{task}: \Omega_S \rightarrow \Omega_T$9, keeps the top $$I_{out} = \mathcal{F}(I_{in}, [P_{\Omega_S}, P_{\Omega_T}]; \Theta)$$0 heads, and zeroes the others (Nozawa et al., 2 Apr 2025). This suggests that VPIP mechanisms range from explicit prompt-conditioned encoding to prompt-guided selection over existing model structure.

4. Learning regimes, efficiency, and automation

VPIP-related methods occupy a wide range of training regimes. PromptGIP trains a single model on a mixed multi-task dataset from up to 15 image processing tasks, uses $$I_{out} = \mathcal{F}(I_{in}, [P_{\Omega_S}, P_{\Omega_T}]; \Theta)$$1 loss, AdamW, a learning rate of $$I_{out} = \mathcal{F}(I_{in}, [P_{\Omega_S}, P_{\Omega_T}]; \Theta)$$2, cosine learning rate decay, batch size 48, 50 epochs, and 8 Tesla V100 GPUs (Liu et al., 2023). The 30-task GenLV model uses $$I_{out} = \mathcal{F}(I_{in}, [P_{\Omega_S}, P_{\Omega_T}]; \Theta)$$3 inputs, $$I_{out} = \mathcal{F}(I_{in}, [P_{\Omega_S}, P_{\Omega_T}]; \Theta)$$4 loss, AdamW with $$I_{out} = \mathcal{F}(I_{in}, [P_{\Omega_S}, P_{\Omega_T}]; \Theta)$$5, $$I_{out} = \mathcal{F}(I_{in}, [P_{\Omega_S}, P_{\Omega_T}]; \Theta)$$6, learning rate $$I_{out} = \mathcal{F}(I_{in}, [P_{\Omega_S}, P_{\Omega_T}]; \Theta)$$7, batch size 64, and 30 epochs (Chen et al., 2024). The scalable extension introduces GLV-Bench with 101 tasks, trains Base, Large, and Huge variants for 50 epochs on 8 A100 GPUs, and widens the prompt encoder to handle greater task diversity (Chen et al., 20 Jul 2025).

Several works emphasize parameter efficiency. In CVPT, only prompt-related parameters and the final classifier are trainable in the basic setup, and weight sharing initializes cross-attention from self-attention so that the cross-attention layer is frozen together with self-attention after sharing weights (Huang et al., 2024). In ablations, frozen shared-weight cross-attention uses $$I_{out} = \mathcal{F}(I_{in}, [P_{\Omega_S}, P_{\Omega_T}]; \Theta)$$8M learnable parameters compared with $$I_{out} = \mathcal{F}(I_{in}, [P_{\Omega_S}, P_{\Omega_T}]; \Theta)$$9M for a learnable cross-attention module, yet performs better on FGVC (Huang et al., 2024). PIP-MM adds only a trainable MLP, reports trained parameters of less than 1% of the full model, and then trains the MLP alone during pre-training and the MLP plus adapter during instruction fine-tuning (Wu et al., 2024). LaViP likewise keeps the backbone frozen and trains only a small prompt generator with a low-rank construction, using about $\Omega_{HQ}$0M parameters on average while remaining applicable in black-box settings (Kunananthaseelan et al., 2023).

Other systems eliminate training or reduce it to prompt search. PHS is explicitly training-free and operates only at inference by prompt-guided head selection (Nozawa et al., 2 Apr 2025). AutoVP treats visual prompting as a searchable design space over prompt size, input scaling, source model, and output mapping. Its search covers 222 configurations, uses Ray Tune, grid search, and ASHA early stopping, and retains the top $\Omega_{HQ}$1 candidates after 2–5 tuning epochs per configuration before full training (Tsao et al., 2023). SEVEX goes further by automating prompt discovery itself. It iterates through selection, implementation and execution, semantic backpropagation, and expansion; uses Novelty-guided UCT; and repeats for 50 iterations to discover task-wise visual prompts through agent-driven experiments rather than manual trial-and-error (Kim et al., 17 Mar 2026).

This diversity of learning regimes indicates that VPIP is not synonymous with one optimization protocol. It includes end-to-end supervised training, in-context learning, parameter-efficient fine-tuning, training-free prompt-guided inference, and automated prompt exploration.

5. Task coverage and empirical performance

In low-level vision, the GenLV line is the clearest large-scale instantiation of VPIP. The initial model is trained on 30 tasks spanning 10 restoration tasks, 8 enhancement tasks, 3 edge-detection tasks, and 9 stylization tasks (Chen et al., 2024). On restoration, GenLV$\Omega_{HQ}$2 reports 28.99 on Gaussian noise versus 26.48 for PromptGIP$\Omega_{HQ}$3, 31.69 on Poisson noise versus 27.76, 36.63 on salt-and-pepper versus 28.08, and 28.91 on haze versus 24.55; the haze result is especially notable because X-Restormer$\Omega_{HQ}$4 without VPIP reaches only 16.73 dB (Chen et al., 2024). On enhancement and stylization, GenLV$\Omega_{HQ}$5 reaches 23.55 on low-light enhancement, 27.61 on LLF, 31.59 on MTM, 34.45 on SDR-to-HDR, and 35.92 on HDR-to-SDR (Chen et al., 2024). On edge detection, it reports 8.07 MAE for Canny, 1.27 for Laplacian, and 7.23 for PED, substantially better than the PromptGIP and Painter values listed in the same table (Chen et al., 2024). The scalable follow-up expands to 101 tasks in GLV-Bench and reports that GenLV-Huge improves over GenLV-Base by +1.17 dB on restoration and +0.86 dB on enhancement, while training on more tasks enhances generalization, particularly for tasks with limited data (Chen et al., 20 Jul 2025).

PromptGIP provides an earlier universal image-processing result with one model, one training stage, many tasks, and no task-specific retraining (Liu et al., 2023). It covers 10 degradation types in restoration, low-light enhancement and local Laplacian filtering in enhancement, and Canny and Laplacian edge detection in feature extraction (Liu et al., 2023). The paper reports some out-of-distribution behavior on mixed degradation restoration, style transfer, and colorization, but explicitly characterizes the model’s capability as preliminary generalization rather than full emergent reasoning (Liu et al., 2023).

Outside low-level image-to-image processing, VPIP-style mechanisms have produced competitive results in several adjacent domains. CVPT reports 76.2 average accuracy on VTAB-1K versus 72.0 for VPT-Deep, 90.3 average accuracy on FGVC versus 89.1 for VPT-Deep, and on ADE20K improves 10-prompt mIoU from 42.11/44.06 to 43.78/45.85 while reaching 45.66/47.92 with 200 prompts (Huang et al., 2024). IMProv reports that text conditioning and larger data improve in-context learning by over +10% AP for Foreground Segmentation, over +5% gains in AP for Single Object Detection, and almost 20% lower LPIPS in Colorization (Xu et al., 2023). PIP-MM reports an average improvement of about 2.7% over baselines across seven multimodal benchmarks and maintains excellent generation results even when half of the visual tokens are reduced (Wu et al., 2024). PHS improves focus-oriented retrieval on COCO, PASCAL VOC, and Visual Genome, for example raising COCO DINOv2-base MP@100 to 60.6 and MAP@100 to 64.1 while remaining training-free (Nozawa et al., 2 Apr 2025). "Textualize Visual Prompt for Image Editing via Diffusion Bridge" reports PSNR 24.57, SSIM 0.8091, LPIPS 0.1197, V-CLIP 0.2750, I-CLIP 0.8178, V-DINO 0.3234, I-DINO 0.9073, Edit Analogy 4.81, Fidelity 4.62, and Overall 4.72 for visual-prompt-driven image editing (Xu et al., 7 Jan 2025). SEVEX reports 78.9% average accuracy overall versus 71.6% for Naive and 64.6% for SketchPad, with inference cost about 10.9% higher than Naive but about 91.2% lower token cost than SketchPad (Kim et al., 17 Mar 2026).

Taken together, these results support two recurrent claims in the literature. First, visual prompts can unify heterogeneous task families when the prompt represents the transformation relation itself rather than a task name. Second, prompt-based methods are not uniformly weaker than adapter-based or fully task-specific alternatives; their competitiveness depends strongly on prompt representation, interaction design, and training regime.

6. Robustness, misconceptions, and open problems

A major misconception addressed in the literature is that prompt-based methods are inherently weak in vision. CVPT argues instead that VPT’s original formulation is structurally mismatched to vision because prompt tokens compete with sparse image patch tokens inside shared self-attention; with many prompts, VPT degrades sharply to 64.0 at 200 prompts, whereas CVPT continues improving to 74.8, indicating that the weakness lies in the interaction mechanism rather than in prompting per se (Huang et al., 2024). This reframing is consistent with the VPIP emphasis on explicit prompt-image interaction via cross-attention (Chen et al., 2024).

Prompt sensitivity remains a persistent issue. PromptGIP reports that prompt quality matters, richer texture and color structure tend to work better, and different prompts for the same task can yield different performance (Liu et al., 2023). The review literature likewise identifies prompt sensitivity, overfitting of learned prompts, domain shift, and the absence of a universal best prompt form as open problems in visual prompt engineering (Wang et al., 2023). By contrast, the GenLV papers report small standard deviations over 20 prompt samples, often around or below 0.1 dB, suggesting that prompt cross-attention and the chosen backbone reduce—but do not eliminate—prompt dependence (Chen et al., 2024).

Generalization is partial rather than unlimited. PromptGIP explicitly states that it does not reliably perform colorization and does not show strong emergent capabilities on unseen prompt types (Liu et al., 2023). The scalable GenLV study reports partial zero-shot transfer but still finds old photo restoration and real-world denoising on SIDD challenging; it also notes that feature extraction tasks can be harder because they require semantic understanding rather than only pixel transformation (Chen et al., 20 Jul 2025). PIP-MM improves most on prompt-localization-heavy benchmarks and is slightly lower than its baseline on MathVista, which the paper notes is less prompt-localization-heavy and more global or analytic (Wu et al., 2024).

Model specificity is another recurrent theme. SEVEX shows that visual prompts discovered for one LVLM may fail or even degrade another model, leading to the conclusion that visual prompting must be model-specific and discovered empirically rather than assumed transferable across backbones (Kim et al., 17 Mar 2026). PHS similarly depends on the pretrained ViT having enough semantic diversity across attention heads, which is why very small models benefit less (Nozawa et al., 2 Apr 2025).

The principal research directions stated in the VPIP literature are scale and modality expansion. The GenLV papers explicitly suggest larger models, broader and richer task sets, more data, and better generalization to unseen tasks (Chen et al., 2024). IMProv’s results suggest that vision and language prompts are complementary and that using both is advantageous for in-context learning (Xu et al., 2023). A plausible implication is that future VPIP systems will continue moving toward hybrid prompt interfaces in which source-target visual examples, region cues, and language semantics are combined within a single prompt-aware processing pipeline.