Visual Instance-aware Prompt Tuning (ViaPT)

• The paper introduces ViaPT, combining instance-aware and dataset-level prompts to overcome limitations of fixed visual prompting.
• It employs a lightweight image-conditioned generator and PCA-based prompt propagation to balance shallow and deep transfer across layers.
• Empirical evaluations on 34 datasets show that ViaPT outperforms static prompt baselines and even full fine-tuning with minimal parameter tuning.

Searching arXiv for the exact ViaPT paper and closely related prompt-tuning work to ground the article in current literature. Visual Instance-aware Prompt Tuning (ViaPT) is a parameter-efficient transfer method for vision transformers that replaces the conventional assumption of one fixed prompt set per downstream dataset with a hybrid prompt design that combines dataset-level prompts and instance-aware prompts generated from each individual input. Proposed for pretrained ViTs with frozen backbones, ViaPT uses image-conditioned prompt generation at the first layer and PCA-based prompt propagation across deeper layers, thereby framing VPT-Shallow and VPT-Deep as two corner cases of a single prompt-information continuum (Xiao et al., 10 Jul 2025).

1. From static visual prompting to instance-aware prompting

ViaPT emerges from the visual prompt tuning literature initiated by Visual Prompt Tuning, which introduced learnable prompt tokens for frozen vision transformers and emphasized that standard VPT prompts are static, task-level parameters shared across all inputs of a task (Jia et al., 2022). That static design is efficient, but it assumes that a single global prompt bank can serve every image equally well. ViaPT is motivated by the claim that this assumption becomes sub-optimal when downstream datasets exhibit high variance, including fine-grained differences, domain shift, clutter, and instance-specific local cues (Xiao et al., 10 Jul 2025).

Several adjacent prompt-tuning directions sharpen this motivation. Dynamic Visual Prompt Tuning argued that existing PETL methods “do not take into account instance-specific visual clues” and generated dynamic instance-wise tokens from each image via a Meta-Net (Ruan et al., 2023). V$^2$APT replaced static prompt tokens with VAE-generated input-dependent prompts and combined them with domain-specific prompts (Xiao et al., 22 Mar 2025). At the theoretical end, VAPT reinterpreted prompt tokens as experts in an MoE view of attention and identified the restricted expressiveness of static prompt experts as a core limitation, proposing input-conditioned prompt functions instead (Le et al., 31 Jan 2025). In CLIP-like settings, DPT introduced class-aware visual prompts generated dynamically from text features and image tokens, while SEP refined visual prompts at each layer from representative pretrained visual tokens, making prompt behavior explicitly dependent on the current instance (Xing et al., 2022, Yao et al., 2024).

Within that landscape, ViaPT is distinctive in two respects. First, it does not discard dataset-level prompting; instead, it fuses dataset-level and instance-level prompt information. Second, it does not treat prompt propagation across layers as a binary choice between shallow reuse and deep replacement. Instead, it formalizes prompt transfer as a continuum controlled by a PCA bottleneck (Xiao et al., 10 Jul 2025). This suggests that ViaPT is best understood as a hybridization of task-global and sample-specific control rather than as a simple dynamic replacement of VPT.

2. Hybrid prompt architecture and instance-conditioned generation

ViaPT starts from the standard ViT tokenization pipeline. An input image is converted into patch tokens $\mathbf{E}_0 \in \mathbb{R}^{k \times d}$ together with a class token $\mathbf{x}_0$. The first-layer prompt bank is split into two disjoint components: instance-aware prompt tokens and dataset-level prompt tokens. The paper writes

$$\mathbf{P}_0^{\mathrm{ins}} := \{\mathbf{p}_0^1, \mathbf{p}_0^2, \ldots, \mathbf{p}_0^\lambda\},$$

$$\mathbf{P}_0^{\mathrm{dom}} := \{\mathbf{p}_0^{\lambda+1}, \mathbf{p}_0^{\lambda+2}, \ldots, \mathbf{p}_0^p\}.$$

The effective first-layer prompt set is their concatenation (Xiao et al., 10 Jul 2025).

Instance-aware prompt generation is image-conditioned but does not operate on raw pixels. Instead, a lightweight generator $g$ consumes the initial tokenized image representation $\mathbf{E}_0$. The intended mapping is

$$g: \mathbb{R}^{k \times d} \rightarrow \mathbb{R}^{\lambda \times d},$$

but the implementation parameterizes this more compactly through Gaussian statistics: $$(\boldsymbol{\mu}, \boldsymbol{\sigma}) = g(\mathbf{E}_0),$$ with $$\boldsymbol{\mu}, \boldsymbol{\sigma} \in \mathbb{R}^{d}$$. Then, for sampled latent vectors $\mathbf{E}_0 \in \mathbb{R}^{k \times d}$0, the instance-aware prompt tokens are constructed as

$\mathbf{E}_0 \in \mathbb{R}^{k \times d}$1

Each image therefore receives its own first-layer instance-aware prompts, while the remaining prompt tokens remain dataset-level parameters (Xiao et al., 10 Jul 2025).

The first transformer layer then processes the usual concatenated token stream: $\mathbf{E}_0 \in \mathbb{R}^{k \times d}$2 The paper’s conceptual claim is that the dataset-level prompts encode global task priors, while the generated prompts adapt those priors to each instance. That division is central to ViaPT’s identity: unlike purely static VPT, it is not limited to one prompt bank per dataset; unlike purely dynamic prompting, it preserves a stable task-conditioned component (Xiao et al., 10 Jul 2025).

3. PCA-based prompt propagation and the unification of VPT-Shallow and VPT-Deep

ViaPT’s second defining mechanism is its treatment of prompt transfer across depth. Standard VPT provides two canonical choices. In VPT-Shallow, prompts are inserted once and then fully propagated through all later layers. In VPT-Deep, fresh prompts are inserted at every layer and previous prompt outputs are not used as explicit prompt inputs (Jia et al., 2022). ViaPT reframes those as extreme points of a more general propagation rule (Xiao et al., 10 Jul 2025).

Let $\mathbf{E}_0 \in \mathbb{R}^{k \times d}$3 be the prompt outputs from layer $\mathbf{E}_0 \in \mathbb{R}^{k \times d}$4. ViaPT applies a PCA mapping

$\mathbf{E}_0 \in \mathbb{R}^{k \times d}$5

yielding

$\mathbf{E}_0 \in \mathbb{R}^{k \times d}$6

This reduced prompt representation is then concatenated with newly learned parameters

$\mathbf{E}_0 \in \mathbb{R}^{k \times d}$7

to form the next layer’s prompt input: $\mathbf{E}_0 \in \mathbb{R}^{k \times d}$8 The next transformer layer receives

$\mathbf{E}_0 \in \mathbb{R}^{k \times d}$9

PCA thus reduces the feature dimension of each prompt token, not the number of prompt tokens (Xiao et al., 10 Jul 2025).

The paper’s unifying claim follows directly from the dimensionality parameter $\mathbf{x}_0$0. If $\mathbf{x}_0$1, no prompt information is preserved from the previous layer and the method reduces to VPT-Deep. If $\mathbf{x}_0$2, all prompt dimensions are retained and the method reduces to VPT-Shallow. ViaPT occupies the interior regime $\mathbf{x}_0$3, where some prior-layer prompt information is preserved and some fresh layer-specific capacity is learned (Xiao et al., 10 Jul 2025).

This continuum is one of ViaPT’s most consequential conceptual contributions. It turns the usual shallow-versus-deep dichotomy into an information-allocation problem: how much of the prompt state should be inherited, and how much should be relearned. The paper argues that random dimension reduction yields performance between the two extremes, whereas PCA provides a principled bottleneck because it retains high-variance prompt directions rather than arbitrary coordinates (Xiao et al., 10 Jul 2025).

4. Optimization, parameterization, and implementation details

ViaPT is trained with standard downstream supervision plus a latent regularizer. The paper gives the objective

$\mathbf{x}_0$4

where $\mathbf{x}_0$5 is the classification cross-entropy and the KL term regularizes the image-conditioned prompt distribution toward a standard Gaussian prior (Xiao et al., 10 Jul 2025).

The main implementation targets ViT-Base/16 pretrained on ImageNet-21k. The appendix reports $\mathbf{x}_0$6, 12 transformer layers, 196 image patches, batch size $\mathbf{x}_0$7, $\mathbf{x}_0$8 training epochs, AdamW with learning rate $\mathbf{x}_0$9, weight decay $$\mathbf{P}_0^{\mathrm{ins}} := \{\mathbf{p}_0^1, \mathbf{p}_0^2, \ldots, \mathbf{p}_0^\lambda\},$$0, cosine annealing, and 10 warmup epochs. In the principal configuration, the prompt design uses 50 prompt tokens, typically split into 25 instance-aware and 25 domain-level tokens, with PCA retaining $$\mathbf{P}_0^{\mathrm{ins}} := \{\mathbf{p}_0^1, \mathbf{p}_0^2, \ldots, \mathbf{p}_0^\lambda\},$$1 dimensions per token across deeper layers and $$\mathbf{P}_0^{\mathrm{ins}} := \{\mathbf{p}_0^1, \mathbf{p}_0^2, \ldots, \mathbf{p}_0^\lambda\},$$2 for KL regularization (Xiao et al., 10 Jul 2025).

The first-layer instance-aware generator is described in the main text as a lightweight 2-layer convolutional encoder, while the appendix at one point refers to a lightweight MLP encoder. That discrepancy is part of the paper’s implementation record and is relevant when reconstructing the method exactly (Xiao et al., 10 Jul 2025).

Inference can be stochastic or deterministic. For stochastic inference, the final class probability is averaged over $$\mathbf{P}_0^{\mathrm{ins}} := \{\mathbf{p}_0^1, \mathbf{p}_0^2, \ldots, \mathbf{p}_0^\lambda\},$$3 forward passes: $$\mathbf{P}_0^{\mathrm{ins}} := \{\mathbf{p}_0^1, \mathbf{p}_0^2, \ldots, \mathbf{p}_0^\lambda\},$$4 The paper reports $$\mathbf{P}_0^{\mathrm{ins}} := \{\mathbf{p}_0^1, \mathbf{p}_0^2, \ldots, \mathbf{p}_0^\lambda\},$$5 as a practical choice and also mentions deterministic alternatives, including fixed latent samples shared across test images or direct prompt generation without sampling (Xiao et al., 10 Jul 2025).

5. Empirical behavior across 34 datasets

ViaPT is evaluated on 34 datasets spanning FGVC, HTA, and VTAB-1k. On ViT-Base/16, the reported means are as follows (Xiao et al., 10 Jul 2025):

Benchmark	ViaPT	VPT-Deep
FGVC mean	91.40	89.11
HTA mean	92.20	85.5
VTAB-1k mean	76.36	69.43

In the same comparison, full fine-tuning attains 88.54 on FGVC, 85.8 on HTA, and 65.57 on VTAB-1k, while ViaPT also records 61.25 on VTAB Structured, compared with 60.80 for SA$$\mathbf{P}_0^{\mathrm{ins}} := \{\mathbf{p}_0^1, \mathbf{p}_0^2, \ldots, \mathbf{p}_0^\lambda\},$$6VP and 60.19 for VFPT (Xiao et al., 10 Jul 2025). The paper therefore presents ViaPT as outperforming prior prompt baselines across all three benchmark suites and as exceeding full fine-tuning on the aggregated means reported for the ViT-Base setting.

The method also transfers beyond vanilla ViTs. On Swin-Base, the reported VTAB-1k category means are 85.29 on Natural, 86.21 on Specialized, and 62.24 on Structured (Xiao et al., 10 Jul 2025). Under self-supervised pretraining, the paper reports competitive MAE performance and the best MoCo v3 results across Natural, Specialized, and Structured VTAB groups (Xiao et al., 10 Jul 2025).

The ablations isolate both major components. On VTAB-1k, removing PCA yields 74.35, removing the instance prompt yields 75.03, removing both yields 73.18, and replacing the learned instance-aware component with random prompts yields 73.78, compared with 76.36 for the full model (Xiao et al., 10 Jul 2025). A separate study shows that performance peaks around a medium bottleneck, especially $$\mathbf{P}_0^{\mathrm{ins}} := \{\mathbf{p}_0^1, \mathbf{p}_0^2, \ldots, \mathbf{p}_0^\lambda\},$$7, consistent with the interpretation that neither full reuse nor full replacement is optimal (Xiao et al., 10 Jul 2025). On CUB-200, latent sampling gives 89.1, compared with 88.7 for a fixed 25-prompt deterministic alternative and 88.5 for VPT-Deep, indicating that the probabilistic prompt generator contributes beyond simple prompt count adjustment (Xiao et al., 10 Jul 2025).

The paper also reports qualitative analyses. Grad-CAM visualizations are described as more focused and semantically aligned than VPT-Deep, t-SNE shows more compact and separated class clusters on CUB-200, prompt-token similarity maps are more concentrated, and appendix mutual-information estimates report slightly lower $$\mathbf{P}_0^{\mathrm{ins}} := \{\mathbf{p}_0^1, \mathbf{p}_0^2, \ldots, \mathbf{p}_0^\lambda\},$$8 and substantially higher $$\mathbf{P}_0^{\mathrm{ins}} := \{\mathbf{p}_0^1, \mathbf{p}_0^2, \ldots, \mathbf{p}_0^\lambda\},$$9 in the top three layers, consistent with an information-bottleneck interpretation (Xiao et al., 10 Jul 2025).

6. Interpretation, literature position, and limitations

A common misconception is that ViaPT is simply VPT-Deep with extra generated tokens. The paper argues otherwise on two grounds. First, the prompt generator is only one half of the method; the other half is the PCA-guided propagation rule that interpolates between VPT-Shallow and VPT-Deep. Second, parameter count is not the main explanation for the gains: ViaPT is reported at $$\mathbf{P}_0^{\mathrm{dom}} := \{\mathbf{p}_0^{\lambda+1}, \mathbf{p}_0^{\lambda+2}, \ldots, \mathbf{p}_0^p\}.$$0 tuned/total parameters on the main ViT-Base setting, compared with $$\mathbf{P}_0^{\mathrm{dom}} := \{\mathbf{p}_0^{\lambda+1}, \mathbf{p}_0^{\lambda+2}, \ldots, \mathbf{p}_0^p\}.$$1 for VPT-Deep, while still outperforming it (Xiao et al., 10 Jul 2025).

A second misconception is that instance-aware prompting in ViaPT replaces task-global prompting. It does not. The first-layer prompt budget is explicitly divided between image-conditioned and dataset-level components, and the method’s empirical framing depends on that combination. This places ViaPT close to other hybrid designs. DVPT also makes prompts image-conditioned, but does so through a Meta-Net that adds instance-wise visual clues to a shared base prompt and remains shallow rather than PCA-mediated across depth (Ruan et al., 2023). V$$\mathbf{P}_0^{\mathrm{dom}} := \{\mathbf{p}_0^{\lambda+1}, \mathbf{p}_0^{\lambda+2}, \ldots, \mathbf{p}_0^p\}.$$2APT similarly combines generated instance prompts and static domain prompts, but does so with a VAE prompt generator rather than with ViaPT’s first-layer Gaussian generator plus PCA propagation (Xiao et al., 22 Mar 2025). In CLIP-like models, DPT’s CAVPT yields class-aware visual prompts by cross-attending text prompt features and image patch embeddings, offering a multimodal rather than purely visual route to instance dependence (Xing et al., 2022). VAPT, by contrast, provides a theoretical account of why static prompt experts are limited and replaces them with input-conditioned prompt functions $$\mathbf{P}_0^{\mathrm{dom}} := \{\mathbf{p}_0^{\lambda+1}, \mathbf{p}_0^{\lambda+2}, \ldots, \mathbf{p}_0^p\}.$$3, which is closely aligned with ViaPT’s instance-aware premise even though the mechanism differs (Le et al., 31 Jan 2025).

The limitations reported for ViaPT are correspondingly specific. The instance-aware generator adds extra overhead relative to fixed prompts; PCA can suppress subtle but useful prompt dimensions when compression is too strong; sampling introduces stochasticity in training and inference; and structured tasks such as SmallNORB azimuth and dSprites orientation remain difficult (Xiao et al., 10 Jul 2025). The paper also contains the generator-description inconsistency noted above. These caveats matter because they delimit ViaPT’s contribution: it is a hybrid PEFT method that improves prompt adaptivity and prompt-state transfer, not a complete solution to spatial or geometric reasoning.

Taken together, ViaPT marks a specific stage in the evolution of visual prompting. Standard VPT established prompt tokens as a strong control interface for frozen vision transformers (Jia et al., 2022). Later work showed that prompts could be made image-conditioned, multimodally conditioned, or theoretically adaptive (Ruan et al., 2023, Xing et al., 2022, Le et al., 31 Jan 2025). ViaPT’s contribution is to combine per-instance prompt generation with a principled account of how prompt information should be retained across layers. In that sense, it is both a method and a framework: a concrete algorithm for dynamic visual prompting, and a unifying reinterpretation of shallow and deep prompting as endpoints of a prompt-information bottleneck (Xiao et al., 10 Jul 2025).