Image-Adaptive Prompt Learning

Updated 7 July 2026

Image-Adaptive Prompt Learning is a method that customizes prompts based on individual image features, replacing static, one-size-fits-all conditioning.
It encompasses diverse techniques—from token-level adaptation in Vision Transformers to Bayesian and particle-based approaches in generative models—to tailor supervision and improve robustness.
By mitigating overfitting and mode collapse, IAPL enhances performance across zero-shot generation, image detection, segmentation, and visuomotor control.

Image-Adaptive Prompt Learning (IAPL) denotes a family of prompt-learning methods in which prompts, prompt-conditioned features, or prompt-governed parameters are adapted to the current image rather than fixed once training ends. In the literature, the term appears explicitly in zero-shot generative model adaptation and AI-generated image detection, but the same underlying idea also appears under related names in continual learning, federated vision-language adaptation, interpretable image enhancement, few-shot segmentation, fine-grained recognition, and visuomotor control. Across these settings, the common objective is to replace homogeneous, image-agnostic conditioning with instance-conditioned prompt behavior so that supervision, representation, or control signals vary with image content or with image-derived latent factors (Guo et al., 2023, Li et al., 3 Aug 2025).

1. Conceptual scope and defining characteristics

IAPL is not a single architecture. In the cited literature, it appears as image-specific continuous text prompts for CLIP-guided generator adaptation, per-image and per-token prompt matching inside a frozen Vision Transformer, class-conditional image-space masking and padding, prompt-guided image-adaptive enhancement parameters, Bayesian posterior distributions over image-conditioned prompt residuals, domain-aware mixtures of domain-specific prompts, support-derived visual prompts for SAM, and cross-attentive refinement of text features by image patches (Guo et al., 2023, Han et al., 2024, Wu et al., 2023, Kosugi, 2024, Derakhshani et al., 2022, Wei et al., 2023, Shen et al., 2024, Brouwer et al., 2024).

A common simplification is to equate IAPL with text-only soft prompts in CLIP. The literature is broader. In some works, the adapted object is a text prompt sequence; in others it is a visual prefix token, a prompt key/value pair, a support-derived prototype set, or an image-adaptive parameter vector whose semantics are constrained by fixed natural-language prompt pairs. What unifies these variants is the replacement of static prompting by image-conditioned prompting, whether conditioning occurs through a latent mapper, cosine matching over token keys, cross-attention, Bayesian posterior sampling, or test-time entropy minimization (Han et al., 2024, Kosugi, 2024, Li et al., 3 Aug 2025).

The principal motivation is also recurrent. Static prompts or fixed adaptation directions can overfit dominant correlations, ignore image-specific structure, or impose identical supervision on heterogeneous samples. IAPL methods therefore seek to increase flexibility, improve out-of-distribution or cross-domain generalization, preserve instance-specific content, and reduce failure modes such as mode collapse, catastrophic forgetting, or brittle prompt selection (Guo et al., 2023, Derakhshani et al., 2022, Cho et al., 2024).

2. Foundational formulation in zero-shot generative adaptation

A canonical formulation appears in “Zero-shot Generative Model Adaptation via Image-specific Prompt Learning” (Guo et al., 2023), where IAPL is introduced as IPL. The setting assumes a source generator $G_s$ that produces source-domain images labeled by $Y_s$ , an unseen target domain labeled only by $Y_t$ , and frozen CLIP encoders $E_I$ and $E_T$ . For latent codes $w^i$ , source images are generated as $x_i = G_s(w^i)$ , and a latent mapper $F$ predicts continuous prompt vectors

$p_i = \{[V]^i_1, [V]^i_2, \dots, [V]^i_m\} \in \mathbb{R}^{m \times k}.$

These vectors are concatenated with the source or target label embedding to form image-specific prompt matrices $M_s^i$ and $Y_s$ 0.

The method is explicitly two-stage. Stage 1 learns image-specific prompts by contrastive alignment between each source image embedding and its own prompt embedding, with positive pairs at $Y_s$ 1 and negatives at $Y_s$ 2. The contrastive term is

$Y_s$ 3

where $Y_s$ 4 is cosine similarity between normalized CLIP image and text embeddings. A domain regularization term constrains learned prompts to remain compatible with the target label,

$Y_s$ 5

and the Stage 1 objective is $Y_s$ 6 (Guo et al., 2023).

Stage 2 freezes the prompt mapper and updates the target generator $Y_s$ 7 with an adaptive directional CLIP loss. For each image,

$Y_s$ 8

$Y_s$ 9

and

$Y_t$ 0

This replaces the fixed directional loss used by NADA with per-image supervision. The central claim is that fixed-prompt methods yield a single shared $Y_t$ 1, which enforces near-identical gradient directions across latents and fosters mode collapse, whereas IPL yields varying $Y_t$ 2 and thus more diverse outputs (Guo et al., 2023).

This formulation is notable for its architectural independence. The same text-side prompt mechanism is used with StyleGAN-based adaptation and with diffusion autoencoders; the CLIP encoders remain frozen; no adversarial training and no additional identity, LPIPS, or feature penalties are introduced. The default prompt length is $Y_t$ 3, prompt vectors are initialized from “a photo of a,” and the latent mapper is a 4-layer fully connected network (Guo et al., 2023).

3. Mechanistic variants: token-level, image-space, and support-derived prompting

Later work broadens IAPL from per-image text prompts to finer-grained and structurally different mechanisms. In continual learning, I-Prompt performs image-adaptive prompting at token granularity inside a frozen ViT-B/16. At each prompted layer, it computes cosine similarities between self-attention keys of image tokens and learnable prompt keys,

$Y_t$ 4

aggregates prompts as $Y_t$ 5, and injects the result directly into intermediate image tokens by additive modulation. Prompting is applied to layers 1–5, uses a pool size of 100 and prompt length 2, and removes task prediction by selecting prompts on the fly in a single forward pass (Han et al., 2024).

In adaptive multi-modality prompt learning, image prompting is moved into pixel space. For each class hypothesis, a learned probability tensor drives Bernoulli masking of “meaningless patches,” the masked regions are padded with learnable tensors, and text-derived information is injected into the padding through lightweight networks. The masked-and-padded image is then passed to a frozen CLIP image encoder, while class-wise text prompts are updated by image-derived scalars. The objective remains a CLIP-style cross-entropy; the stochastic masking and text-informed padding are the adaptive prompt mechanism (Wu et al., 2023).

Few-shot segmentation with SAM introduces a different form of IAPL: visual prompts are not manually specified as points or boxes but computed from the support image–mask pair. APL-SAM extracts support features, downsamples the support mask, initializes superpixel centroids with MaskSLIC, refines them by iterative clustering, and uses the resulting centroids as visual prompts. The number of prompts is adapted to foreground size through

$Y_t$ 6

with $Y_t$ 7 chosen by ablation. These prompts are fused with learned tokens by self-attention and transferred to query features by cross-attention, then consumed by a multi-source, multi-level mask decoder (Shen et al., 2024).

For fine-grained few-shot classification, Adaptive Prompt Tuning conditions text-side features on image patches via cross-attention. Frozen CLIP text features act as queries, frozen ViT patch embeddings act as keys and values, and a trainable cross-attention block with layer normalization, dropout, residual connections, and an MLP produces tuned text features per image. This makes prompt refinement depend directly on the current patch-level evidence rather than on a static context vector (Brouwer et al., 2024).

4. Bayesian, data-dependent, and domain-aware formulations

A second major line of work interprets IAPL probabilistically. Bayesian Prompt Learning replaces deterministic image-conditioned residual prompts with a learned posterior distribution,

$Y_t$ 8

and optimizes an evidence lower bound

$Y_t$ 9

with a standard normal prior $E_I$ 0. The prompt becomes stochastic and image-conditioned, and Monte Carlo sampling enlarges prompt-space coverage under distribution shift (Derakhshani et al., 2022).

APP, or Adaptive Particle-based Prompt learning, makes the prior itself data-dependent. It defines a Gaussian prior centered at the average prior mean produced by a network $E_I$ 1, estimates a multimodal posterior over prompts by Wasserstein Gradient Flow and SVGD particles, and adapts at test time through

$E_I$ 2

Here prompts are explicitly image-adaptive at inference because the text feature is blended with the image-conditioned prior mean of the test image (Cho et al., 2024).

Domain-aware prompt learning in federated CLIP offers a third variant. ADAPT maintains domain-specific textual prompts and shared visual prompts. In the last self-attention layer, affinities between the image’s [CLS] query and visual prompt keys produce per-image mixture weights over domains; these weights then mix the text embeddings generated from each domain-specific textual prompt. The effective prompt is therefore a per-image weighted composition of domain prompts rather than a single global or domain-local prompt (Wei et al., 2023).

A further extension appears in cross-embodiment visuomotor policy learning. CAPO learns a pool of domain-factor-specific visual prompts with hybrid contrastive objectives and then uses a dual-branch attention mechanism to compute observation-conditioned weights $E_I$ 3 for prompt orchestration. The fused state representation is

$E_I$ 4

where the text-prompted feature bypasses attention to preserve goal semantics. This suggests that IAPL can function ոչ merely as a classifier-side or text-side adaptation technique, but as a general strategy for constructing instance-conditioned state representations (Zhang et al., 1 Feb 2026).

5. Application areas and empirical record

In zero-shot generative adaptation, IPL improves both quality and diversity over NADA across multiple domains. On Photo→Disney, Inception Score rises from 2.721 to 3.089, SCS from 0.407 to 0.448, and ID from 0.782 to 0.801; on AFHQ Photo→Cartoon, IS rises from 6.505 to 8.658, SCS from 0.407 to 0.563, and ID from 0.925 to 0.941. A user study with 1210 responses reports that, on average, 80.5% of users prefer IPL over NADA. End-to-end adaptation requires approximately 10–20 minutes on a single NVIDIA RTX 3090 GPU in the GAN setup (Guo et al., 2023).

In continual learning, I-Prompt reports both accuracy and efficiency gains. On task-imbalanced ImageNet-R B100-Inc5, it reaches 73.96/67.30 Avg/Last versus CODA-Prompt 71.24/64.20, and on task-balanced CIFAR-100 B0-Inc10 it reaches 91.75/87.63 versus 91.45/86.19. Its trainable parameter ratio is approximately 1.43%, with training time 5.86 ms/image and inference 2.50 ms/image, compared with 9.47 and 4.90 for CODA-Prompt (Han et al., 2024).

In interpretable image enhancement, PG-IA-NILUT couples fixed prompt pairs with image-adaptive parameters. On MIT-Adobe FiveK at 480p it reports PSNR 25.22, SSIM 0.930, $E_I$ 5 7.76, and 1.9 ms runtime; at 4K it reports PSNR 25.05, SSIM 0.934, $E_I$ 6 7.88, and 2.0 ms runtime. On PPR10K at 360p it reports PSNR 26.00, SSIM 0.957, and $E_I$ 7 6.81 (Kosugi, 2024).

In federated domain generalization, ADAPT reaches 68.4% average accuracy over six domains in DomainNet while learning and sharing only 0.08M parameters, improving the original CLIP by 14.8%. The reported per-domain accuracies are 77.5 on clipart, 63.1 on infograph, 70.5 on painting, 41.6 on quickdraw, 85.7 on real, and 72.1 on sketch (Wei et al., 2023).

In few-shot SPM image segmentation, APL-SAM substantially exceeds vanilla SAM and nnUNet. Mean DSC rises from 22.58% for SAM with 1 point and 55.24% for SAM with 20 points to 89.53% for APL-SAM; mean IoU rises from 18.08% and 46.29% to 81.95%. The paper characterizes this as over a 30% improvement in Dice Similarity Coefficient with only one-shot guidance (Shen et al., 2024).

In fine-grained few-shot recognition, Adaptive Prompt Tuning reports Top-1 accuracy gains over static prompt tuning. On FGVC Aircraft it rises from 27% at 1-shot to 47% at 16-shot, against a zero-shot CLIP baseline of 17%; on Oxford Flowers it reaches 84% at 1-shot and 97% at 16-shot; on CUBirds it reaches 56% at 1-shot and 77% at 16-shot. The same work integrates Monte-Carlo Dropout and reports Expected Calibration Error decreasing with more shots, with Flowers and CUBirds stabilizing around approximately 0.01–0.02 for 16-shot (Brouwer et al., 2024).

In AI-generated image detection, IAPL combines a Conditional Information Learner with Confidence-Driven Adaptive Prediction on top of a frozen CLIP ViT-L/14. It achieves 95.61% mean accuracy and 99.32% mean average precision on UniversalFakeDetect, and 96.7% mean accuracy on GenImage. The ablation trail is explicit: a frozen CLIP plus classifier gives 84.19/93.52 mAcc/mAP on UFD, adding MLP-based adapters yields 89.02/97.66, adding CIL yields 93.51/98.24, and adding CDAP yields 95.61/99.32 (Li et al., 3 Aug 2025).

In visuomotor control, CAPO reports source / seen target / unseen target success rates of 97.9 / 90.9 / 86.4, compared with 96.1 / 83.3 / 80.5 for ConPE and 86.5 / 77.7 / 59.8 for PPO, together with better SPL and lower navigation error. The prompt pool size is best at $E_I$ 8, matching the ten modeled domain factors (Zhang et al., 1 Feb 2026).

6. Limitations, misconceptions, and open directions

The limitations reported across the literature are heterogeneous but consistent in structure. In zero-shot generative adaptation, large domain shifts such as Human→Cat remain challenging, learned continuous prompts do not map cleanly to single vocabulary words, and performance is sensitive to the domain regularization weight $E_I$ 9, the number of prompt tokens $E_T$ 0, and biases in CLIP or in the chosen domain labels (Guo et al., 2023). In continual learning, prompt pools and memory grow linearly with the number of tasks, inference latency increases accordingly, and performance is sensitive to prompt pool size and prompt length (Han et al., 2024). In prompt-guided enhancement, prompt wording is dataset-dependent, CLIP-based attribute scores can reflect VLM bias, and the prompt guidance loss slightly reduces reconstruction metrics while improving interpretability (Kosugi, 2024).

Bayesian and particle-based methods reduce overfitting but introduce their own constraints. BPL adds inference cost through Monte Carlo sampling and may slightly reduce in-domain accuracy relative to CoOp (Derakhshani et al., 2022). APP uses a fixed isotropic prior covariance, can be sensitive to the blending weight $E_T$ 1, and depends on kernel bandwidth selection for SVGD (Cho et al., 2024). Domain-aware federated prompting can suffer when domain correspondence detection is ambiguous or diffuse (Wei et al., 2023). Support-derived prompting in APL-SAM depends on support mask quality, requires tuning of $E_T$ 2, and can collapse to a single prototype for very small targets (Shen et al., 2024). Image-adaptive detection relies on a single texture-rich patch for conditioning and incurs multi-view test-time overhead; globally diffuse artifacts or severe shift can therefore remain difficult (Li et al., 3 Aug 2025). CAPO is bounded by the diversity of source-domain factors represented in its prompt pool, and large $E_T$ 3 can increase compute and induce redundancy or overfitting (Zhang et al., 1 Feb 2026).

A recurrent misconception is that IAPL is merely a more flexible form of soft prompt tuning. The surveyed work indicates a broader interpretation: IAPL can alter supervision geometry in CLIP space, select or weight prompts per token, rewrite the image itself through masking and padding, derive scale-aware support prototypes, impose semantic disentanglement on adaptive enhancement controls, or perform test-time prompt optimization on a single sample. This suggests that IAPL is best understood as a design principle for instance-conditioned prompting rather than as a single prompt parameterization.

Open directions are already present in the primary sources. For zero-shot generation, proposed extensions include multi-prompt ensembles, semantic clustering, adversarial prompt regularization, curriculum adaptation, and hybrid losses for stricter content retention (Guo et al., 2023). For interpretable enhancement, future work includes automatic prompt selection, content-aware prompt weighting, improved constraints, video consistency, local context injection, and hybrid FiLM-style modulation (Kosugi, 2024). For control, proposed directions include online or meta adaptation of prompt parameters, multimodal orchestration, and hierarchical prompts for long-horizon tasks (Zhang et al., 1 Feb 2026). A plausible implication is that future IAPL systems will increasingly combine image-conditioned prompting with structured priors, selective test-time adaptation, and modality-specific prompt semantics rather than relying on a single universal prompt mechanism.