ANPrompt: Anti-noise Prompt Tuning for VLMs

• The paper demonstrates that integrating clustered weak-noise features with learnable prompt tokens significantly improves robustness in CLIP-based vision-language models.
• It introduces a deep prompt injection mechanism and noise-resistant visual prompt prototype to mitigate semantic perturbations in both image and text inputs.
• Empirical results on 11 diverse datasets show enhanced base-to-novel generalization and cross-dataset transfer compared to earlier prompt tuning methods.

Searching arXiv for the exact ANPrompt paper and closely related prompt-learning work to ground the article. ANPrompt is a prompt tuning framework for vision-LLMs, particularly CLIP-like architectures, that is designed to improve robustness to weak semantic perturbations in both image and text inputs. It addresses a limitation attributed to earlier prompt tuning methods: although they are parameter-efficient and often effective for base-to-novel transfer, they remain vulnerable to subtle semantic noise such as background clutter, co-occurring objects, or linguistically misleading phrasing that preserves nominal class identity while shifting nearby semantics. ANPrompt responds by constructing weak-noise text features, clustering them into noise prompts, integrating those prompts with learnable prompt tokens, injecting the resulting anti-noise prompts into deeper layers of both the image and text encoders, and optimizing the system with cross-entropy, similarity, and weak semantic noise alignment objectives (Gao et al., 6 Aug 2025).

1. Definition and problem formulation

ANPrompt is introduced as Anti-noise Prompt Tuning for Vision-LLMs, a prompt-learning method for CLIP with the explicit goal of improving robustness under weak semantic perturbations (Gao et al., 6 Aug 2025). In the paper’s formulation, the standard CLIP prediction rule remains unchanged:

$$p(y = c \mid f) = \frac{\exp(\text{sim}(f, w_c)/\tau)}{\sum_{i=1}^C \exp(\text{sim}(f, w_i)/\tau)}$$

where $f$ is the image feature, $w_c$ is the text feature for class $c$, $\tau$ is a learnable temperature, and $\text{sim}(\cdot)$ is cosine similarity (Gao et al., 6 Aug 2025). ANPrompt does not replace this scoring rule; instead, it changes how the prompt representations are constructed and aligned.

The motivating problem is not gross corruption but weak semantic perturbation. On the image side, the paper identifies background clutter, irrelevant regions, co-occurring objects, and attention drifting to non-discriminative regions. On the text side, it identifies subtle linguistic shifts that preserve class identity but nudge semantics toward adjacent categories, such as wording that moves “cat” toward “tiger”-like semantics. The paper’s central claim is that prior prompt tuning approaches improve efficiency or transferability but generally do not explicitly model such perturbations (Gao et al., 6 Aug 2025).

This places ANPrompt within a broader line of vision-language prompt learning methods that seek improved generalization beyond seen categories. Relative to CoOp, CoCoOp, MaPLe, PromptSRC, CoPrompt, MMRL, and TAC, ANPrompt is specifically framed around robustness to weak semantic noise rather than only base-to-novel transfer or cross-dataset adaptation (Gao et al., 6 Aug 2025).

2. Core mechanism: weak-noise features and anti-noise prompts

The first stage of ANPrompt constructs weak-noise frozen text features from two intra-class descriptions sampled from a language cache. One is a main text encoding the core semantics, and the other is a noise text introducing small linguistic variation. These are encoded by the frozen CLIP text encoder to obtain $f_m$ and $f_n$, and then fused as

$f_w = f_m + \alpha f_n$

with perturbation strength $\alpha$ (Gao et al., 6 Aug 2025). The implementation sets $f$0, and the ablation reported in the paper finds the best harmonic mean at this value, with higher weights reducing novel-class accuracy (Gao et al., 6 Aug 2025).

The weak-noise text features $f$1 are clustered with K-means to form noise prompts $f$2. The paper states that the weak-noise frozen text features lie in $f$3, and the clustered noise prompts $f$4, where the number of clusters matches the number of learnable prompt tokens $f$5 (Gao et al., 6 Aug 2025). The paper does not provide an explicit K-means objective, initialization scheme, distance metric, or clustering schedule, so those details remain unspecified.

The anti-noise prompt is then defined as

$f$6

where $f$7 denotes learnable prompt tokens, $f$8 the clustered noise prompts, and $f$9 the perturbation injection strength (Gao et al., 6 Aug 2025). The paper further states that $w_c$0 is projected through separate fully connected layers into modality-specific anti-noise prompts for the image and text encoders, though the explicit projection formulas are not given.

This design reflects a broader anchor-based tendency in prompt learning, but with a distinct emphasis. ATPrompt, for example, augments textual prompt learning by inserting explicit universal attribute anchors such as “shape” or “color” into CLIP prompts and selecting attribute sets through differentiable search (Li et al., 2024). AnchorOPT later replaces fixed textual anchors with learned implicit anchors and optimizes anchor positions with a learnable position matrix (Li et al., 26 Nov 2025). ANPrompt differs from both in that its auxiliary prompt source is not attribute vocabulary or latent anchor tokens but clustered weak semantic perturbation features derived from intra-class textual variation (Gao et al., 6 Aug 2025).

3. Deep prompt injection and Noise-Resistant Visual Prompt Prototype

ANPrompt is a deep prompt tuning method. The paper states that anti-noise prompts are injected into the deeper layers of both the image and text encoders, rather than only appended as shallow input-level context (Gao et al., 6 Aug 2025). The pretrained encoder weights remain frozen, and only prompt-related parameters are learned. The exact number of prompted layers is not specified.

The token-sequence notation in the paper is partially malformed, but its intended structure is clear. For the visual encoder, the input sequence contains anti-noise image prompts together with the class token and patch tokens. For the text encoder, the sequence contains the SOS token, anti-noise text prompts, text tokens, class token(s), and EOS token (Gao et al., 6 Aug 2025). No layerwise residual update equations are provided.

From the output of the visual prompt tokens, ANPrompt computes the Noise-Resistant Visual Prompt Prototype (NRVPP). If the output visual prompt token features are denoted $w_c$1, then

$w_c$2

This prototype is intended to suppress token-level variation, mitigate visual semantic distraction, and reduce sensitivity to irrelevant attended regions (Gao et al., 6 Aug 2025). The prompted image feature $w_c$3 is taken from the final image class token, and the prompted text feature $w_c$4 from the final EOS token (Gao et al., 6 Aug 2025).

The model then defines three logit families:

$w_c$5

$w_c$6

$w_c$7

The final classification logits are

$w_c$8

with $w_c$9 in the implementation (Gao et al., 6 Aug 2025). This makes the final decision depend jointly on prompted image-text alignment and robustness against weak-noise text features.

A plausible implication is that ANPrompt operates simultaneously at two levels: prompt-space perturbation modeling and logit-space robustness calibration. That contrasts with methods that improve prompt generalization mainly through textual structure or richer language priors. For example, AAPE in “Aggregate-and-Adapt Natural Language Prompts” distills rich natural-language prompt summaries into an image-conditional prompt embedding for improved few-shot and out-of-distribution generalization, but it does not formulate weak semantic perturbation as the central object of optimization (Yang et al., 2024).

4. Training objectives and optimization behavior

ANPrompt uses three losses: standard cross-entropy, a cosine-similarity-based sim loss, and the Weak semantic noise Alignment Loss (WALoss) (Gao et al., 6 Aug 2025). The exact formula for the sim loss is not provided in the paper text, but it is described as a cosine loss between frozen and prompted features used to preserve semantic consistency.

WALoss is the most distinctive objective. It is defined using the robustness logits $c$0 and weak-noise logits $c$1:

$c$2

where

$c$3

The paper describes $c$4 as a variance-adaptive coefficient that stabilizes optimization across different confidence regimes (Gao et al., 6 Aug 2025). The total objective is written as

$c$5

though the paper also defines $c$6 itself with a multiplicative $c$7, creating a notational ambiguity (Gao et al., 6 Aug 2025).

The training configuration uses CLIP ViT-B/16 as backbone, Adam with learning rate $c$8, 10 epochs, batch size 4, and averaging over 3 random seeds on a single NVIDIA RTX 3090 Ti GPU (Gao et al., 6 Aug 2025). The language cache incorporates prompts from CoPrompt and HPT, and weak perturbations are formed by randomly sampling two intra-class descriptions from this cache (Gao et al., 6 Aug 2025).

The combination of prompt construction and alignment loss should be distinguished from other robustness-oriented prompt methods. APT for anomaly detection, for instance, also uses prompt adaptation plus semantic alignment, but in a different setting: few-shot anomaly detection with synthetic anomaly generation, learnable normality and abnormality prompts, and gradient-calibrated alignment to meta-prompts (Chen et al., 22 Aug 2025). ANPrompt instead addresses classification robustness under weak semantic noise in CLIP-style vision-language learning (Gao et al., 6 Aug 2025).

5. Empirical results and ablations

ANPrompt is evaluated on the standard 11-dataset CoOp suite: ImageNet, Caltech101, OxfordPets, Flowers102, Food101, StanfordCars, FGVCAircraft, EuroSAT, UCF101, DTD, and SUN397 (Gao et al., 6 Aug 2025). The evaluation protocols are base-to-novel generalization, cross-dataset transfer from ImageNet, and domain generalization to ImageNet-V2, ImageNet-Sketch, ImageNet-A, and ImageNet-R (Gao et al., 6 Aug 2025).

In base-to-novel generalization, the reported average results are:

Method	Base	Novel	HM
ANPrompt	86.15	77.70	81.70
MMRL	85.68	77.16	81.20
CoPrompt	84.00	77.23	80.48
PromptSRC	84.26	76.10	79.97

ANPrompt therefore achieves the best average harmonic mean, with reported gains over MMRL of $c$9 on base, $\tau$0 on novel, and $\tau$1 on harmonic mean (Gao et al., 6 Aug 2025). Dataset-level highlights include EuroSAT at 95.53 / 87.33 / 91.21, StanfordCars with the best base accuracy 83.57 and HM 78.85, FGVCAircraft with the best base accuracy 49.67 and HM 42.14, UCF101 with the best base accuracy 88.80 and HM 84.21, and Caltech101 with tied-best base accuracy 98.97 and best novel and HM of 94.73 and 96.80 (Gao et al., 6 Aug 2025). The paper also notes that ANPrompt is not best on every dataset; for example, MMRL is slightly better on ImageNet novel/HM, and CoPrompt or MMRL are slightly stronger on OxfordPets (Gao et al., 6 Aug 2025).

In cross-dataset transfer, ANPrompt reports the best average accuracy at 67.14, ahead of CoPrompt at 67.00, TAC at 66.53, MaPLe at 66.10, PromptSRC at 65.81, CoCoOp at 65.74, and CoOp at 63.88 (Gao et al., 6 Aug 2025). It is strongest on six of ten target datasets and is particularly strong on FGVCAircraft, EuroSAT, OxfordPets, StanfordCars, Flowers, and Caltech (Gao et al., 6 Aug 2025).

In domain generalization, the result is more mixed. ANPrompt reports an average of 60.40 on ImageNet-derived targets, compared with PromptSRC at 60.63, MMRL at 60.59, CoPrompt at 60.42, and MaPLe at 60.26 (Gao et al., 6 Aug 2025). It is best on ImageNet-V2 with 64.63 but not best on average.

The ablation results clarify where the gains come from. Removing all anti-noise components yields HM 80.91. Adding WALoss alone gives 80.97, Anti-Prompt alone 81.10, TextNoise + WALoss 81.25, WALoss + Anti-Prompt 81.27, TextNoise + Anti-Prompt 81.31, and the full method 81.70 (Gao et al., 6 Aug 2025). This shows that the gains are complementary rather than attributable to a single component.

Prompt token length is also studied: $\tau$2 gives HM 80.53, $\tau$3 80.30, $\tau$4 80.45, $\tau$5 80.66, $\tau$6 81.70, and $\tau$7 81.66, with best performance at 5 prompt tokens (Gao et al., 6 Aug 2025). Varying the weak-noise text feature weight gives HM 81.70 at 0.001, 81.21 at 0.01, 81.00 at 0.1, and 80.43 at 1.0, indicating that subtle perturbation helps while stronger perturbation degrades generalization (Gao et al., 6 Aug 2025).

6. Position within prompt-learning research and methodological interpretation

ANPrompt belongs to a broader family of prompt-learning methods that use auxiliary semantics to improve transfer. ATPrompt extends CLIP textual prompt learning by embedding universal attributes as explicit semantic anchors, moving from category-only alignment to attribute-category hybrid prompting and using differentiable search over attribute combinations (Li et al., 2024). AnchorOPT later argues that anchors should be dynamic in both value and position, replacing fixed textual attribute anchors with learned implicit anchor tokens and a learnable position matrix (Li et al., 26 Nov 2025). AAPE instead distills knowledge from multiple natural-language prompts into an image-conditional prompt embedding through aggregation and adaptation (Yang et al., 2024).

Within that landscape, ANPrompt’s distinctive move is to treat weak semantic perturbation itself as a constructive supervisory resource. It neither relies on explicit hand-authored attributes, as in ATPrompt, nor on latent anchors learned from class-description distillation, as in AnchorOPT, nor on prompt-summary distillation from large prompt sets, as in AAPE. Instead, it builds a perturbation-aware prompt space from intra-class textual variation and then ties that space to both encoder prompting and logit-level alignment (Gao et al., 6 Aug 2025).

This suggests a specific interpretation of its contribution. The paper is not primarily about richer class semantics or broader textual knowledge, but about making prompt-tuned VLMs less brittle in the vicinity of semantically neighboring alternatives. That helps explain why the strongest gains appear in settings where subtle semantic distinctions matter, such as EuroSAT, FGVCAircraft, and StanfordCars (Gao et al., 6 Aug 2025).

The method also has clear limitations. The paper does not report a dedicated semantic-noise corruption benchmark, so its robustness claims are supported indirectly through downstream generalization and ablation gains rather than explicit perturbation-severity evaluation (Gao et al., 6 Aug 2025). Several implementation details remain underspecified, including clustering configuration, exact prompt injection depth, and the sim-loss formula (Gao et al., 6 Aug 2025). Domain generalization gains are mixed rather than uniformly superior. The weak-noise weight must remain small, and larger perturbation weights reduce novel-class accuracy (Gao et al., 6 Aug 2025).

A plausible implication is that ANPrompt is most useful when the failure mode of prompt tuning is semantic brittleness rather than gross domain mismatch. Where the primary issue is underrepresented class knowledge, attribute anchoring or prompt-summary distillation may be more central (Li et al., 2024, Yang et al., 2024). Where the issue is fixed anchor rigidity, dynamic anchor optimization may be more relevant (Li et al., 26 Nov 2025). ANPrompt’s specific niche is robustness to weak semantic drift in few-shot prompt-tuned CLIP classification (Gao et al., 6 Aug 2025).

In that sense, ANPrompt advances prompt learning by reframing noise not as an external corruption to be filtered away, but as a nearby semantic neighborhood that prompt representations should explicitly model and survive.