Prompt-Level Regularization in Adaptation

• Prompt-level regularization is a set of methods that explicitly constrain the prompt or its induced representation to prevent overfitting and retain pretrained, task-agnostic knowledge.
• Techniques include anchoring outputs to frozen predictions, aligning prompted features with pretrained representations, enforcing probabilistic and geometric constraints, and regulating gradient updates.
• These methods trade task-specific memorization for improved transfer, robustness under scarce data and distribution shifts, while demanding a delicate balance between regularization strength and prompt flexibility.

Prompt-level regularization denotes a family of adaptation strategies in which the prompt itself, the prompt-induced representation, or the prompt-update rule is explicitly constrained so that downstream specialization does not erase pretrained generality. Across recent work, the term covers several distinct but related interventions: anchoring prompted outputs to frozen prompt predictions, matching prompted features to pretrained representations, placing priors over prompt distributions, regularizing prompt-update gradients, balancing prompt experts, smoothing prompt optimization with perturbations, and constraining discrete prompt rewrites in text space (Zhu et al., 2023, Derakhshani et al., 2022, Khattak et al., 2023, Park et al., 2024, Chen et al., 2023, Fu et al., 20 May 2026).

1. Conceptual scope and taxonomy

A central theme in the literature is that freezing a pretrained backbone does not by itself prevent forgetting. In CLIP-style adaptation, the backbone may remain frozen while learned prompts still steer the model toward narrow task-specific regions of representation space; in few-shot language-model prompting, only prompt parameters may be trainable while optimization remains sharp, unstable, or overly sample-specific (Khattak et al., 2023, Zhu et al., 2023, Chen et al., 2023). Prompt-level regularization therefore targets the interface by which downstream data condition a pretrained model, rather than the full parameter set.

The regularized object varies substantially across papers. In some methods it is the output distribution induced by prompting a frozen model; in others it is the prompt posterior, the prompted feature geometry, the prompt gradient, the routing distribution over prompt experts, or the natural-language prompt itself. This diversity is not incidental. It reflects different diagnoses of prompt failure: overfitting to downstream labels, collapse of prompt diversity, prompt-space undercoverage, unstable optimization, expert under-utilization, or prompt distributional overfitting (Derakhshani et al., 2022, Jiang et al., 2023, Fu et al., 20 May 2026).

Representative method	Regularized object	Core mechanism
ProReg (Zhu et al., 2023)	Fine-tuned prediction vs. frozen prompt prediction	KL regularization with adaptive sample-wise weight
Bayesian Prompt Learning (Derakhshani et al., 2022)	Distribution over prompt residuals	Variational posterior with KL-to-prior
PromptSRC (Khattak et al., 2023)	Prompted image/text features and logits	Self-consistency to frozen CLIP, GPA, textual diversity
ProMetaR (Park et al., 2024)	Prompt-induced representation drift	Meta-learned modulation of regularizer gradients
SUPMER (Pan et al., 2023)	Prompt-update gradient	Meta-gradient transformation and gating
Prompt-OT (Chen et al., 11 Mar 2025)	Joint image-text feature distribution	Optimal transport to zero-shot features
PTP (Chen et al., 2023)	Prompt optimization landscape	Random or adversarial perturbation training
TextReg (Fu et al., 20 May 2026)	Discrete prompt representation	Regularized textual gradients via inefficiency control

A common misconception is that prompt-level regularization is synonymous with a simple penalty on prompt parameters. The literature contradicts this. Several influential methods regularize prompt effects rather than prompt norms: output agreement in ProReg, feature/logit agreement in PromptSRC, OT-based joint feature alignment in Prompt-OT, and gradient shaping in SUPMER, ProMetaR, and GRMP-IQA (Zhu et al., 2023, Khattak et al., 2023, Chen et al., 11 Mar 2025, Pan et al., 2023, Park et al., 2024, Li et al., 2024).

2. Anchoring prompted models to pretrained knowledge

One major line of work treats prompt regularization as a knowledge-preservation problem. The premise is that downstream prompt tuning overfits because it is driven solely by biased or narrow task labels, whereas the pretrained model retains broader “encyclopedic” or task-agnostic knowledge. Prompt-level regularization then anchors the adapted model to a frozen prompt-accessible prior rather than allowing unrestricted drift (Zhu et al., 2023, Khattak et al., 2023).

In ProReg, the adapted model prediction $f(\mathbf{x};\theta)$ is regularized toward a prompt-induced distribution $\mathbf{y}$ produced by the pretrained model. The training loss is

$$L = (1-w)L_{\mathrm{CE}} + \alpha\, w\, L_{\mathrm{KL}}, \qquad w=\frac{f_t}{f_t+y_t},$$

where $f_t$ and $y_t$ are the probabilities assigned by the fine-tuned model and the prompt model to the ground-truth class $t$ (Zhu et al., 2023). This is not ordinary distillation from a separate teacher. The teacher is the model’s own frozen prompt-based predictor, and the KL weight is sample-wise and adaptive rather than constant. The same paper reports that ProReg reaches $54.89\%$ on VQA-CP, compared with FT++ at $46.34\%$ and zero-shot at $43.62\%$, and improves OOD accuracy on BAR, NICO, PACS, and DomainNet (Zhu et al., 2023).

PromptSRC takes a closely related but broader approach in CLIP prompt learning. It adds self-consistency losses between prompted and frozen CLIP image features, text features, and logits,

$$\mathcal{L}_{\text{final}}=\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{SCL}},$$

with $\mathbf{y}$0 combining $\mathbf{y}$1, $\mathbf{y}$2, and $\mathbf{y}$3 (Khattak et al., 2023). Two additional regularizers extend this anchor: Gaussian weighted prompt aggregation over the training trajectory, and textual diversity via an average over $\mathbf{y}$4 frozen CLIP prompt templates. The reported average base-to-novel performance over 11 datasets is Base $\mathbf{y}$5, Novel $\mathbf{y}$6, HM $\mathbf{y}$7 (Khattak et al., 2023).

A stronger distributional version appears in Prompt-OT. Instead of matching prompted and frozen features pointwise, Prompt-OT concatenates image and text features into a joint representation $\mathbf{y}$8 and regularizes the prompted batch distribution toward the frozen zero-shot batch distribution via

$\mathbf{y}$9

The full objective is $$L = (1-w)L_{\mathrm{CE}} + \alpha\, w\, L_{\mathrm{KL}}, \qquad w=\frac{f_t}{f_t+y_t},$$0 (Chen et al., 11 Mar 2025). The paper argues that OT preserves global feature geometry while enlarging the feasible prompt-parameter region relative to rigid pointwise constraints. On base-to-novel generalization over 11 datasets, it reports Base $$L = (1-w)L_{\mathrm{CE}} + \alpha\, w\, L_{\mathrm{KL}}, \qquad w=\frac{f_t}{f_t+y_t},$$1, Novel $$L = (1-w)L_{\mathrm{CE}} + \alpha\, w\, L_{\mathrm{KL}}, \qquad w=\frac{f_t}{f_t+y_t},$$2, HM $$L = (1-w)L_{\mathrm{CE}} + \alpha\, w\, L_{\mathrm{KL}}, \qquad w=\frac{f_t}{f_t+y_t},$$3 (Chen et al., 11 Mar 2025).

SPTR and CCPL show that the anchor need not be the unprompted model alone. SPTR aligns the tuned textual feature distribution to multiple hand-crafted prompt features via optimal transport,

$$L = (1-w)L_{\mathrm{CE}} + \alpha\, w\, L_{\mathrm{KL}}, \qquad w=\frac{f_t}{f_t+y_t},$$4

and adds a similarity-paradigm loss

$$L = (1-w)L_{\mathrm{CE}} + \alpha\, w\, L_{\mathrm{KL}}, \qquad w=\frac{f_t}{f_t+y_t},$$5

with total loss $$L = (1-w)L_{\mathrm{CE}} + \alpha\, w\, L_{\mathrm{KL}}, \qquad w=\frac{f_t}{f_t+y_t},$$6 (Cui et al., 20 Feb 2025). CCPL instead anchors learnable class-prompt embeddings to frozen concept prototypes using a text-space cosine consistency loss,

$$L = (1-w)L_{\mathrm{CE}} + \alpha\, w\, L_{\mathrm{KL}}, \qquad w=\frac{f_t}{f_t+y_t},$$7

with default $$L = (1-w)L_{\mathrm{CE}} + \alpha\, w\, L_{\mathrm{KL}}, \qquad w=\frac{f_t}{f_t+y_t},$$8, concept dropout $$L = (1-w)L_{\mathrm{CE}} + \alpha\, w\, L_{\mathrm{KL}}, \qquad w=\frac{f_t}{f_t+y_t},$$9, and no KL-based prediction consistency term (Sang et al., 21 Jun 2026). This suggests a broader principle: prompt-level regularization often works by choosing a semantically richer target than the downstream label alone.

3. Probabilistic and geometric control of prompt space

A second line of work regularizes not merely prompted outputs but the prompt space itself. In Bayesian Prompt Learning, the prompt is treated as a latent random variable rather than a single deterministic embedding. For the conditional version, the model introduces a latent residual prompt $f_t$0 and optimizes the ELBO

$f_t$1

with $f_t$2 and prior $f_t$3 (Derakhshani et al., 2022). The regularizer is therefore a KL on the prompt posterior itself. The paper explicitly frames this as prompt-level regularization because the prompt random variable, not merely network weights, is constrained. Reported gains include unseen-prompt average $f_t$4 versus $f_t$5 for CoCoOp and $f_t$6 for ProDA (Derakhshani et al., 2022).

This probabilistic view generalizes the notion of prompt-space coverage. Deterministic ERM prompt tuning identifies one sharp point in prompt space; Bayesian prompt learning forces a family of plausible prompt realizations and uses test-time prompt sampling as an ensemble (Derakhshani et al., 2022). The same logic reappears in discrete prompt theory: the PAC-Bayes analysis in “Prompts Generalize with Low Data” treats the prompt space as the hypothesis space and shows that informative or data-dependent priors tighten prompt generalization bounds through the term $f_t$7 (Madras et al., 9 Oct 2025). When $f_t$8 is a point mass on one prompt, the complexity penalty becomes $f_t$9, making prompt naturalness or low conditional perplexity an explicit regularizer rather than a mere heuristic (Madras et al., 9 Oct 2025).

Geometric prompt-space regularization also appears in diffusion personalization. MCPL learns multiple object-level pseudo-words from a single sentence-image pair and augments the denoising loss with prompt-level constraints. The attention-masked diffusion loss

$y_t$0

uses cross-attention-derived masks to spatially focus supervision, while the prompt contrastive loss

$y_t$1

separates embeddings of distinct learned concepts (Jin et al., 2023). Bind Adjective extends this by tying noun-like learned tokens to adjective anchors already well localized by the pretrained model (Jin et al., 2023). The result is a prompt-space regularizer aimed at semantic disentanglement and region grounding rather than classification robustness.

A related but more implicit form appears in graph prompting. SUPT does not introduce a bespoke regularization term in its main objective, but its subgraph-level basis prompts, graph-aware assignment scores, frozen backbone, and extremely small trainable parameter set act as a structural regularizer on the adaptation hypothesis class (Lee et al., 2024). The paper explicitly reports much lower parameter counts than fine-tuning and stronger few-shot gains, which suggests that prompt parameterization itself can serve as prompt-level regularization when it enforces shared, context-sensitive structure (Lee et al., 2024).

4. Gradient- and optimization-level regularization

A third family moves regularization from prompt states to prompt updates. Here the prompt is regularized by transforming, purifying, or projecting the gradient used to adapt it.

SUPMER is a canonical example. In few-shot soft prompt tuning for frozen LLMs, the raw support-set gradient $y_t$2 is passed through a meta-learned transformation

$y_t$3

and the inner-loop prompt update becomes

$y_t$4

The gate $y_t$5 makes the transformation input-dependent, and the outer loop learns $y_t$6 so that transformed support gradients align with query performance under induced support-query shift (Pan et al., 2023). The paper reports average T5-base few-shot accuracy $y_t$7 versus $y_t$8 without meta-gradient regularization, and domain generalization $y_t$9 versus $t$0 (Pan et al., 2023).

ProMetaR applies the same general idea to vision-language prompt tuning. It regularizes prompted representation drift through $t$1 and $t$2, but does not use a fixed $t$3. Instead it meta-learns a modulation function over the regularizer gradient: $t$4 Task augmentation via manifold mixup then generates virtual validation tasks to reduce meta-overfitting (Park et al., 2024). The reported average base-to-new HM over 11 datasets is $t$5, compared with $t$6 for IVLP and $t$7 for PromptSRC (Park et al., 2024).

GRMP-IQA extends gradient-level prompt regularization to blind image quality assessment. It defines a quality gradient $t$8 and a semantic gradient $t$9, where $54.89\%$0 measures deviation from the original CLIP semantic distribution. The regularized update is

$54.89\%$1

This subtracts the semantic-aligned component of the quality gradient when the two point in a similar direction (Li et al., 2024). With limited data, the paper reports SRCC $54.89\%$2 versus $54.89\%$3 on LIVEC and $54.89\%$4 versus $54.89\%$5 on KonIQ, and states that with $54.89\%$6 of the training data GRMP-IQA outperforms most existing fully supervised BIQA methods (Li et al., 2024).

Abstract-level descriptions of GRAM in vision-language learning follow the same pattern. GRAM is described as jointly meta-learning “an efficient soft prompt initialization” and “a lightweight gradient regulating function” for better cross-domain generalizability, and as improving several prompt tuning methods over 11 datasets in few-shot and cross-domain settings (Li et al., 2023). The supplied content does not provide the exact formulation, but it is consistent with this gradient-regulation line.

Optimization-level regularization also includes prompt-landscape smoothing. PTP argues that vanilla prompt tuning has a precipitous loss landscape and reduces prompt instability by training on perturbed inputs, either with random noise or adversarial perturbations in text or embedding space (Chen et al., 2023). The paper reports improvements of $54.89\%$7 on SuperGLUE and $54.89\%$8 on FewGLUE, together with strong variance reductions such as RTE $54.89\%$9 under PGD in few-shot settings (Chen et al., 2023). This suggests that prompt-level regularization can operate by reshaping the optimization neighborhood seen by the prompt parameters, even when the prompt vectors themselves are not directly penalized.

5. Structural, routing, and discrete prompt regularization

Prompt-level regularization is not confined to soft prompt learning over frozen encoders. It also appears in prompt routing, hard prompt search, and inference-time prompt control.

In multimodal fusion, Conditional Prompt Tuning introduces a mixture of prompt experts (MoPE) and an explicit regularizer to prevent routing collapse. For each layer, prompt experts $46.34\%$0 are combined by soft routing

$46.34\%$1

and the expert-balancing regularizer is defined through batch-wise importance

$46.34\%$2

with

$46.34\%$3

This is a load-balancing regularizer over prompt experts, not a sparsity penalty (Jiang et al., 2023). The paper shows that without the importance loss a few experts dominate, while with it expert usage becomes balanced and increasing $46.34\%$4 scales performance better than increasing prompt length (Jiang et al., 2023).

For hard prompt learning with reinforcement learning, PIN regularizes the token-selection policy itself. Replacing Shannon-entropy regularization with sparse Tsallis entropy yields the sparse policy

$46.34\%$5

and an additional ignorable-token filter removes low-probability tokens under the base LLM (Choi et al., 2024). The objective is not to make prompt embeddings smoother, but to suppress the dense exploration that produces opaque “secret language” prompts. The paper reports average few-shot text classification improvement from $46.34\%$6 to $46.34\%$7, together with markedly more interpretable prompts (Choi et al., 2024).

TextReg brings prompt-level regularization into fully discrete prompt rewriting. It models prompt inefficiency as

$46.34\%$8

the product of capacity cost and scope narrowness, and realizes a soft penalty objective through Dual-Evidence Gradient Purification, Semantic Edit Regularization, and Regularization-Guided Prompt Update (Fu et al., 20 May 2026). The reported OOD gains reach up to $46.34\%$9 over TextGrad and $43.62\%$0 over REVOLVE (Fu et al., 20 May 2026). This line makes explicit that prompt regularization in discrete text space can play the role that complexity control plays for model parameters.

Inference-time prompt regularization appears in zero-shot anomaly segmentation as well. SAA+ combines language prompts, property prompts, saliency prompts, and confidence prompts to constrain frozen GroundingDINO and SAM without training (Cao et al., 2023). Its “hybrid prompt regularization” is not a loss term but a multi-stage restriction of the feasible prediction set. The paper reports $43.62\%$1 total $43.62\%$2 for SAA+ versus $43.62\%$3 for the unregularized SAA assembly (Cao et al., 2023). This suggests that prompt-level regularization can be purely procedural: prompting as semantic restriction, geometric filtering, contextual rescoring, and sparse selection.

6. Empirical patterns, limitations, and recurring controversies

Across these papers, a recurrent empirical pattern is that prompt-level regularization is most valuable when adaptation data are scarce or evaluation is distribution-shifted. Bayesian Prompt Learning improves unseen-prompt, cross-dataset, and cross-domain generalization but is not best on in-domain performance (Derakhshani et al., 2022). ProReg shows the same trade-off: larger $43.62\%$4 improves OOD but reduces ID on DomainNet (Zhu et al., 2023). SPTR finds the best harmonic mean at intermediate $43.62\%$5 rather than stronger regularization (Cui et al., 20 Feb 2025). CCPL likewise shows that stronger inference fusion raises new-class accuracy while lowering base accuracy, making $43.62\%$6 “dataset- and protocol-sensitive” (Sang et al., 21 Jun 2026). These results support a common interpretation: prompt regularization usually trades memorization for transfer.

Another recurring pattern is that anchor quality matters. ProReg assumes prompt-based zero-shot predictions are less biased than downstream data, but notes that if prompt quality is poor or the pretrained model lacks target-domain knowledge, the regularizer is less helpful (Zhu et al., 2023). CCPL attributes its strongest gains to EuroSAT, where class-specific scene concepts align naturally with visible semantics, and reports near-neutral change on OxfordPets, where generic concept phrases fail to capture fine-grained breed cues (Sang et al., 21 Jun 2026). SPTR similarly emphasizes multiple hand-crafted prompts and reports modest gains except on novel or robustness-sensitive settings (Cui et al., 20 Feb 2025). Prompt-level regularization is therefore not intrinsically beneficial; its success depends on whether the regularizing prompt source encodes a semantically useful prior.

The literature also shows that “freezing the backbone” is not an answer to forgetting. PromptSRC, ProMetaR, Prompt-OT, and ProReg all start from frozen or mostly frozen pretrained models yet still document prompt overfitting and loss of task-agnostic structure (Khattak et al., 2023, Park et al., 2024, Chen et al., 11 Mar 2025, Zhu et al., 2023). A related misconception is that prompt-level regularization must be an explicit additive penalty. The field includes explicit KL or OT penalties, but also update-rule regularizers, perturbation-based smoothing, trajectory aggregation, sparse prompt policies, and inference-time prompt filtering (Zhu et al., 2023, Chen et al., 11 Mar 2025, Pan et al., 2023, Chen et al., 2023, Choi et al., 2024, Cao et al., 2023).

Costs and limitations are likewise method-specific. Bayesian prompt learning requires Monte Carlo prompt sampling at inference (Derakhshani et al., 2022). Prompt-OT adds a second frozen forward path and mini-batch OT during training, though not at inference (Chen et al., 11 Mar 2025). PTP incurs extra perturbation-generation cost and is sensitive to perturbation strength (Chen et al., 2023). TextReg depends on several LLM-mediated judgments and currently targets single-turn reasoning with well-defined behavioral rules (Fu et al., 20 May 2026). SUPMER requires substantial self-supervised meta-training, reported as about 140 GPU hours on 8 RTX 3090 GPUs (Pan et al., 2023). These constraints indicate that prompt-level regularization is not a single lightweight add-on; it is a design axis that trades off robustness, compute, supervision type, and architectural complexity.

The broader significance of the literature is that prompts are no longer treated as passive textual wrappers or small trainable prefixes. They are treated as adaptive objects with their own geometry, dynamics, routing statistics, and generalization behavior. Prompt-level regularization is the set of methods that make those properties explicit and controllable. In current work, that control is exercised by pretrained anchors, priors over prompt space, gradient transformations, transport constraints, routing balance, sparse prompt policies, or semantic compression. The field’s unifying claim is that prompt optimization needs regularization for the same reason model optimization does: without it, prompts overfit.