Vision-First Prompt Reformulation

• Vision-First Prompt Reformulation is a paradigm that reformulates vision tasks by injecting learnable modifications into a largely frozen backbone, minimizing full fine-tuning.
• It achieves parameter-efficient adaptation by training only prompts and prototypes, ensuring robust transfer and high task alignment with pre-trained generative models.
• The approach leverages techniques like masked visual token modeling and adaptive prompt tuning to optimize performance across multiple vision tasks and architectures.

Vision-First Prompt Reformulation is the paradigm of recasting downstream vision tasks as prompt-driven problems, where minimal, learnable, or programmatic visual or token-based modifications are injected into the input or intermediate representations of a frozen (or largely frozen) large-scale vision model. The defining feature is that the adaptation is “vision-first”—either operating directly on visual tokens, pixels, patches, or intermediate transformer states—rather than on language or meta-level instructions. The goal is to achieve parameter-efficient, robust, and consistent transfer from a pre-trained vision backbone to new tasks by exploiting prompts as the modifiable interface, often leveraging generative or mask reconstruction objectives for maximal task alignment and stability.

1. Conceptual Foundations and Motivation

Classic prompt learning originated in NLP, where downstream tasks are reformulated to match pre-training objectives via textual cues fed to powerful LLMs. In the vision domain, prompt reformulation has followed two main threads:

• Visual Prompt Learning for discriminative models: Learnable tokens or input overlays are appended to patch or pixel sequences, steering a fixed discriminative transformer (e.g., ViT) toward new tasks with minimal parameter updates.
• Generative-model-driven Visual Prompting: Downstream tasks are reformulated to mimic masked token modeling tasks of pre-trained generative backbones (e.g., VQGAN-transformer). This closes the gap between pre-training and downstream inference, offering improved alignment and robustness (Liao et al., 2023).

The motivation for vision-first prompt reformulation includes (i) maximizing parameter efficiency by freezing most backbone weights, (ii) aligning the adaptation process with the pre-trained model’s original generative or self-supervised dynamics, and (iii) enabling transfer to new domains or tasks with minimal supervision or human intervention.

2. Formal Frameworks and Mathematical Formulation

The core idea of vision-first prompt reformulation is to shift the model adaptation burden from full-parameter fine-tuning to learning a concise set of prompts (tokens, pixel overlays, or manipulation programs) injected at specific locations in the architecture. There are several canonical formulations:

2.1. Masked Visual Token Modeling (VPTM)

Let $x$ be an image, which a VQ encoder maps to visual tokens $z = \{z_1, \ldots, z_n\} \in V^n$. Mask positions $m \subset \{1,\ldots,n\}$ are selected, and unmasked tokens $z_{/m}$ are kept. Prompt tokens $p = \{p_1, \ldots, p_P\}$ (continuous) are interleaved with $z_{/m}$ as input.

The generative model predicts codebook posteriors for masked positions, using the cross-entropy loss

$$L_{VPTM} = - \frac{1}{|m|} \sum_{i \in m} \sum_{v \in V} \mathbf{1}[z_i = v] \log P_\theta(z_i = v | z_{/m}, p)$$

Only $p$ is trained; the backbone $\theta$ remains frozen. At inference, prompt tokens steer reconstruction, and outputs are mapped to class prototypes $\{\mu_k\}$ in the codebook embedding space using nearest-neighbor assignment (Liao et al., 2023).

2.2. Adaptive Prompt Tuning in Vision Transformers

Visual Prompt Tuning (VPT) inserts $z = \{z_1, \ldots, z_n\} \in V^n$0 learnable prompt embeddings $z = \{z_1, \ldots, z_n\} \in V^n$1 into the patch+token sequence in a ViT. Shallow VPT only modifies the initial block, while deep VPT injects prompts at each layer. Block-wise gates $z = \{z_1, \ldots, z_n\} \in V^n$2 can be learned for each transformer block $z = \{z_1, \ldots, z_n\} \in V^n$3 to control how much a prompt influences the representation:

$z = \{z_1, \ldots, z_n\} \in V^n$4

This allows per-layer prompt selection and adaptive weighting, leading to improved performance and sample efficiency, particularly in self-supervised ViT adaptation (Yoo et al., 2023).

2.3. Generative, Instance-Adaptive, and Input-Space Prompting

Prompt Generation Networks (PGN) generate unique prompt embeddings for each input image via an auxiliary lightweight network $z = \{z_1, \ldots, z_n\} \in V^n$5, selecting from or synthesizing library tokens. Prompts can be projected back to RGB input space and concatenated as image patches, enabling prompt-based adaptation even in deployment scenarios where model internals are inaccessible (Loedeman et al., 2022).

2.4. Vision Prompt Programs and Manipulation

For large vision-LLMs (LVLMs), vision-first prompting extends beyond tokens to full image manipulation "programs": $z = \{z_1, \ldots, z_n\} \in V^n$6, a block of executable code (e.g., crop, draw), plus a paired textual instruction $z = \{z_1, \ldots, z_n\} \in V^n$7 form a visual prompt $z = \{z_1, \ldots, z_n\} \in V^n$8 that maximizes expected accuracy when passed through both image transformation and LVLM inference (Kim et al., 17 Mar 2026).

3. Prototypical Verbalizers and Output Decoding

A distinctive component of vision-first prompt reformulation—especially in generative architectures—is the use of a prototypical verbalizer to map the outputs (visual tokens/codes) to discrete downstream class labels. Each class $z = \{z_1, \ldots, z_n\} \in V^n$9 is represented as a prototype embedding $m \subset \{1,\ldots,n\}$0, where $m \subset \{1,\ldots,n\}$1 is a support set and $m \subset \{1,\ldots,n\}$2 is a codebook embedding. Given output predictions $m \subset \{1,\ldots,n\}$3 (probabilities over codebook entries), the expected embedding $m \subset \{1,\ldots,n\}$4 is computed and averaged, and the predicted class is assigned via nearest-prototype in the embedding space:

$m \subset \{1,\ldots,n\}$5

This decouples model output from the explicit classifier head and supports efficient adaptation to new or open-set classes by adding prototypes without modifying the rest of the pipeline (Liao et al., 2023).

4. Design, Implementation, and Empirical Insights

Vision-first prompt methods exhibit several architectural choices and performance advantages:

• Prompt Location and Length Robustness: Performance is invariant ($m \subset \{1,\ldots,n\}$6 drop) to prompt insertion locations (early/shallow vs. deep layers) and stable to prompt length in moderate regimes ($m \subset \{1,\ldots,n\}$7 variation for $m \subset \{1,\ldots,n\}$8).
• Parameter Efficiency: Only the parameters for prompts and, optionally, prototypes are trained (typically $m \subset \{1,\ldots,n\}$9 of the model).
• Masking Strategy: Masked positions often correspond to spatial regions providing maximal information for the downstream task.
• Fine-Grained Control: In advanced VPTMs, prompt tokens and masking can be tailored per sample, enabling fine class resolution or robustness under distribution shift (Liao et al., 2023).
• Prompt Selection in Deep ViTs: Empirical sweeps show that the optimal prompt-injection block is often mid-to-late in self-supervised ViTs, contrasting with shallow insertions in supervised ones. Block gating further automates this selection, boosting downstream accuracy by 8–36 pp depending on the backbone and task (Yoo et al., 2023).
• Instance-Adaptivity: By making prompts depend not only on layer but also on input image content, “progressive” (e.g., ProVP) and adaptive prompting (e.g., VAPT) architectures approach meta-network performance, propagating data-driven adjustments through the transformer while keeping parameter count minimal (Xu et al., 2023, Le et al., 31 Jan 2025).

In terms of empirical results, VPTM achieves mean accuracy increases of $z_{/m}$0 over classical visual prompt learning and $z_{/m}$1 over linear probing, with strong cross-dataset generalization (ImageNet, CIFAR-100, Flowers, Birds) and negligible sensitivity to hyperparameters such as prompt location or length (Liao et al., 2023). Gated and adaptive prompting variants consistently outperform both shallower designs and full fine-tuning under strict parameter budgets (Yoo et al., 2023, Le et al., 31 Jan 2025).

5. Applications and Extensions

Vision-first prompt reformulation underpins a wide array of tasks and architectures:

• Classification and Recognition: Paradigms such as VPTM, VPT, ProVP, and Pro-tuning support robust transfer of pre-trained backbones to new classification tasks, including extreme class imbalance, image corruption, adversarial attacks, and out-of-distribution scenarios (Liao et al., 2023, Nie et al., 2022).
• Vision-Language Grounding: Prompts act both as visual cues and textual queries, and approaches such as Position-guided Text Prompt (PTP) decompose images into blocks and encourage explicit visual-text grounding, yielding recall@1 gains of $z_{/m}$2 and $z_{/m}$3 on ViLT/BLIP Flickr30K retrieval, with zero inference-time overhead (Wang et al., 2022).
• Multi-modal Compositionality: SEVEX shows that vision-first prompts instantiated as image-manipulation programs discovered via semantic tree search can diagnose and rectify LVLM perception failures, providing 14.3% accuracy improvement over tool-driven baselines on challenging visual reasoning benchmarks (Kim et al., 17 Mar 2026).
• Generative Alignment: Adaptive prompt optimization and inference-time prompt scaling (PRIS) exploit element-level factual verifiers to iteratively revise prompts and maximize attribute realization in T2I and T2V generation (+15% VBench 2.0, +7.1% VQA-Score on GenAI-Bench), outperforming fixed-prompt best-of-N sampling (Kim et al., 3 Dec 2025).
• Universal and Personalized Generation: Prompt rewriting with user-specific retrieval and black-box LLMs, or self-rewarding LVLM loops, tune prompt content for maximal aesthetic/alignment reward, measured via CLIP, preference, and PickScore metrics (Chen et al., 2023, Yang et al., 22 May 2025).
• Dense Prediction, Segmentation, and Editing: Unified LPG (left-prompt-guided) paradigms concatenate visual prompt and target input along spatial axes, adapting powerful inpainting T2I models to tasks ranging from segmentation to semantic/attribute editing, achieving strong data efficiency and outperformance over commercial solutions (Xie et al., 16 Feb 2025).

6. Deployment Guidelines, Challenges, and Future Directions

Best practices for vision-first prompt reformulation draw from observed empirical robustness and model efficiency:

• Use a moderate number of prompt tokens (P = 8–32) and mask positions to balance computational cost and expressivity (Liao et al., 2023).
• Prefer shallow to mid-layer prompt insertion unless diagnostics indicate deeper features are more task-relevant (Yoo et al., 2023).
• Leverage prototypical verbalizers and class prototypes to accommodate new or fine-grained categories with negligible retraining (Liao et al., 2023).
• For inference-constrained scenarios or black-box deployments, employ prompt generation networks with RGB-space prompt inversion (Loedeman et al., 2022).
• In generative settings, iteratively verify attribute realization and revise prompts at inference time to overcome prompt scaling plateaus (Kim et al., 3 Dec 2025).
• Maintain architectural modularity: with approaches such as LoRA (AnyRefill) or per-layer prompt blocks (Pro-tuning), composing/editing task modules is trivial and efficient (Xie et al., 16 Feb 2025, Nie et al., 2022).

Open challenges include black-box adaptation without access to patch embedding projections (PGN), extending input-based prompt formulation to dense prediction (PGN), robustness under extreme domain shift, unified frameworks for multimodal and programmatic visual prompting, and scaling vision-first prompt reformulation with automated, semantics-aware discovery mechanisms such as SEVEX (Kim et al., 17 Mar 2026).

Vision-First Prompt Reformulation has thus emerged as a cornerstone methodology for parameter-efficient adaptation, robust transfer, and interpretability in modern computer vision, supporting both discriminative and generative models across an expanding array of tasks (Liao et al., 2023, Kim et al., 17 Mar 2026, Yoo et al., 2023, Xie et al., 16 Feb 2025, Kim et al., 3 Dec 2025, Wang et al., 2022, Le et al., 31 Jan 2025, Chen et al., 2023, Yang et al., 22 May 2025, Nie et al., 2022, Xu et al., 2023).