Pre-trained Model Guided AFT (PMG-AFT)

• The paper demonstrates that using a frozen pre-trained CLIP model as guidance during adversarial fine-tuning improves zero-shot robustness while mitigating overfitting.
• PMG-AFT leverages dual training branches—one for adversarial robustness and another for preserving generalizable features—to balance clean and robust accuracy.
• Empirical results show PMG-AFT outperforms baselines like FT-TeCoA, achieving notable gains in both robust and clean accuracies across various datasets.

Pre-trained Model Guided Adversarial Fine-Tuning (PMG-AFT) is a defense method for improving the zero-shot adversarial robustness of CLIP-style vision-LLMs while preserving the broad generalization induced by large-scale pre-training. It is introduced in the paper "Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness" (Wang et al., 2024). The method is motivated by the observation that direct adversarial fine-tuning can improve robustness on a fine-tuning dataset but can also move the model too far from the pre-trained solution, leading to overfitting and degraded transfer. PMG-AFT addresses this by using the original frozen pre-trained CLIP model as a guide during adversarial fine-tuning, so that the fine-tuned model is optimized for robustness without discarding the generalizable representations already captured during pre-training.

1. Problem setting and motivation

PMG-AFT is formulated for CLIP under a zero-shot adversarial robustness setting. The starting point is that large-scale pre-trained vision-LLMs such as CLIP exhibit strong zero-shot generalization, yet remain vulnerable to imperceptible adversarial examples. Existing defense methods typically rely on adversarial training or adversarial fine-tuning, but direct application to CLIP can produce overfitting, especially when fine-tuning is performed on a relatively small downstream dataset (Wang et al., 2024).

The paper identifies several aspects of this overfitting problem. Fine-tuning on clean data already causes the target model to generalize poorly outside the fine-tuning dataset. Adversarial fine-tuning aggravates this effect because adversarial examples lie far from the clean data manifold and therefore push the model further away from its pre-trained initialization. Existing methods such as FT-TeCoA improve zero-shot robust accuracy, but still exhibit a substantial drop in clean accuracy, which the paper interprets as evidence of overfitting to the target dataset.

The central objective of PMG-AFT is therefore not merely to increase robustness on adversarially perturbed inputs, but to preserve the useful generalizable features learned during pre-training while adapting the model for robustness. This suggests a robustness–generalization tradeoff is intrinsic to adversarial fine-tuning of large pre-trained models, and that PMG-AFT is designed as a constraint on parameter drift rather than as a replacement for adversarial training itself.

2. Training architecture and guiding supervision

PMG-AFT keeps the CLIP text encoder frozen and fine-tunes only the image encoder. During training, the method is organized around two conceptual branches.

The first branch is the robustness information branch. This branch corresponds to standard adversarial fine-tuning: the target model is trained to classify adversarial examples correctly. In isolation, this branch would improve adversarial performance but would not explicitly preserve the pre-trained model’s transfer behavior.

The second branch is the generalization information branch, which is the key novelty of PMG-AFT. The original frozen pre-trained CLIP image encoder is used as an auxiliary teacher. The same adversarial examples are passed through both the target image encoder being fine-tuned and the original pre-trained image encoder. The outputs of the target model are then encouraged to match those of the original model. The paper describes this as an auxiliary generalization information branch that “minimizes the distance between the adversarial example outputs in the target model and the pre-trained model” (Wang et al., 2024).

Operationally, the original CLIP model serves as a source of supervision on adversarial examples. The target model is thus optimized under dual pressures: it must become robust enough to classify perturbed inputs correctly, but it must also remain aligned with the output behavior of the pre-trained model on those same inputs. A plausible implication is that the original model functions as a stabilizing prior over fine-tuning dynamics, limiting over-specialization to the fine-tuning dataset.

3. Objective function and optimization

PMG-AFT is built on top of adversarial training. For CLIP, the image encoder is denoted by $F_e(\cdot)$, the text encoder by $T_o(\cdot)$, and classification is defined through image-text similarity and cross-entropy over the induced class probabilities (Wang et al., 2024).

Adversarial examples are generated with PGD:

$$x_{k+1} = x_k + \alpha \nabla_x L(x_k, t, y), \qquad x_{k+1} = P_{B(x,\epsilon)}(x_{k+1}), \qquad k=0,\ldots,K-1,$$

with perturbation ball

$B(x,\epsilon)=\{x \mid \|x-x_0\|_p<\epsilon\}.$

For a batch of image embeddings $I$ and text embeddings $T$, the similarity matrix is

$S = I \cdot T^{\top},$

the class probabilities are

$$P_{ij} = \frac{\exp(S_{ij})}{\sum_{k=1}^{c} \exp(S_{ik})},$$

and the cross-entropy loss is

$$L_{CE}(X,T,Y) = -\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{c} Y_{ij}\log(P_{ij}).$$

The robustness information branch uses

$L_{\text{robust}} = L_{CE}(X_a, T, Y).$

For the generalization information branch, let $T_o(\cdot)$0 denote the adversarial-output probabilities of the target model and $T_o(\cdot)$1 those of the original pre-trained CLIP model:

$T_o(\cdot)$2

Using

$T_o(\cdot)$3

the auxiliary term is

$T_o(\cdot)$4

The method also introduces a clean-output regularizer. Let

$T_o(\cdot)$5

and define

$T_o(\cdot)$6

The full objective is

$T_o(\cdot)$7

with the paper using $T_o(\cdot)$8 and $T_o(\cdot)$9 as the best setting. In the paper’s interpretation, the robustness branch enforces adversarial classification, the generalization branch preserves the pre-trained model’s output behavior on adversarial inputs, and the clean regularization term encourages consistency between adversarial and clean outputs.

4. Evaluation protocol, datasets, and attack settings

The principal evaluation protocol fine-tunes CLIP on TinyImageNet and then measures zero-shot transfer on 15 additional datasets, plus TinyImageNet itself, for a total of 16 datasets (Wang et al., 2024). The 15 zero-shot datasets are CIFAR10, CIFAR100, STL10, OxfordPets, Food101, SUN397, Flowers102, DTD, EuroSAT, FGVCAircraft, ImageNet, Caltech101, Caltech256, StanfordCars, and PCAM. In the paper, these are grouped into general object recognition, fine-grained recognition, scene recognition, domain-specific data, and medical image classification.

The main attack setting is PGD-10. The paper also reports results under AutoAttack. Perturbation bounds include $$x_{k+1} = x_k + \alpha \nabla_x L(x_k, t, y), \qquad x_{k+1} = P_{B(x,\epsilon)}(x_{k+1}), \qquad k=0,\ldots,K-1,$$0, and additional studies use $$x_{k+1} = x_k + \alpha \nabla_x L(x_k, t, y), \qquad x_{k+1} = P_{B(x,\epsilon)}(x_{k+1}), \qquad k=0,\ldots,K-1,$$1 and $$x_{k+1} = x_k + \alpha \nabla_x L(x_k, t, y), \qquad x_{k+1} = P_{B(x,\epsilon)}(x_{k+1}), \qquad k=0,\ldots,K-1,$$2. The evaluation is therefore designed to test both transfer across heterogeneous datasets and robustness under stronger attack protocols than a single PGD configuration.

This setup is significant because PMG-AFT is not evaluated only on the fine-tuning dataset. Instead, the paper explicitly targets the preservation of zero-shot transfer while improving adversarial performance, which aligns the experimental design with the method’s stated purpose.

5. Empirical performance and ablation findings

Across 15 zero-shot datasets, PMG-AFT significantly outperforms the state-of-the-art method FT-TeCoA in the reported experiments (Wang et al., 2024). The headline results are an average robust accuracy improvement of $$x_{k+1} = x_k + \alpha \nabla_x L(x_k, t, y), \qquad x_{k+1} = P_{B(x,\epsilon)}(x_{k+1}), \qquad k=0,\ldots,K-1,$$3 over the original CLIP, an average robust accuracy improvement of $$x_{k+1} = x_k + \alpha \nabla_x L(x_k, t, y), \qquad x_{k+1} = P_{B(x,\epsilon)}(x_{k+1}), \qquad k=0,\ldots,K-1,$$4 over FT-TeCoA, and an average clean accuracy improvement of $$x_{k+1} = x_k + \alpha \nabla_x L(x_k, t, y), \qquad x_{k+1} = P_{B(x,\epsilon)}(x_{k+1}), \qquad k=0,\ldots,K-1,$$5 over the adversarial-fine-tuning baseline stated in the abstract.

The paper reports that, in Table 1 under PGD-10 with $$x_{k+1} = x_k + \alpha \nabla_x L(x_k, t, y), \qquad x_{k+1} = P_{B(x,\epsilon)}(x_{k+1}), \qquad k=0,\ldots,K-1,$$6, FT-TeCoA attains an average robust accuracy of $$x_{k+1} = x_k + \alpha \nabla_x L(x_k, t, y), \qquad x_{k+1} = P_{B(x,\epsilon)}(x_{k+1}), \qquad k=0,\ldots,K-1,$$7, whereas PMG-AFT reaches $$x_{k+1} = x_k + \alpha \nabla_x L(x_k, t, y), \qquad x_{k+1} = P_{B(x,\epsilon)}(x_{k+1}), \qquad k=0,\ldots,K-1,$$8. In Table 2 for clean accuracy, FT-TeCoA achieves $$x_{k+1} = x_k + \alpha \nabla_x L(x_k, t, y), \qquad x_{k+1} = P_{B(x,\epsilon)}(x_{k+1}), \qquad k=0,\ldots,K-1,$$9, whereas PMG-AFT attains $B(x,\epsilon)=\{x \mid \|x-x_0\|_p<\epsilon\}.$0. Under AutoAttack, PMG-AFT also remains clearly better than CLIP and FT-TeCoA, indicating that the gains are not limited to the PGD-specific setting.

The ablation results clarify the role of the individual loss terms. Adding $B(x,\epsilon)=\{x \mid \|x-x_0\|_p<\epsilon\}.$1 improves both robust and clean accuracy over FT-TeCoA. Adding $B(x,\epsilon)=\{x \mid \|x-x_0\|_p<\epsilon\}.$2 further boosts robust accuracy. The paper reports the following progression: FT-TeCoA yields $B(x,\epsilon)=\{x \mid \|x-x_0\|_p<\epsilon\}.$3 robust and $B(x,\epsilon)=\{x \mid \|x-x_0\|_p<\epsilon\}.$4 clean accuracy; adding $B(x,\epsilon)=\{x \mid \|x-x_0\|_p<\epsilon\}.$5 yields $B(x,\epsilon)=\{x \mid \|x-x_0\|_p<\epsilon\}.$6 robust and $B(x,\epsilon)=\{x \mid \|x-x_0\|_p<\epsilon\}.$7 clean accuracy; adding $B(x,\epsilon)=\{x \mid \|x-x_0\|_p<\epsilon\}.$8 yields $B(x,\epsilon)=\{x \mid \|x-x_0\|_p<\epsilon\}.$9 robust and $I$0 clean accuracy.

The paper also studies feature-layer choice and distance metric. The best setting is the output layer with KL divergence. Alternatives such as $I$1, cosine similarity, or the use of penultimate features perform worse. This suggests that output-space alignment to the original pre-trained model is more effective than intermediate-feature matching in the reported setting.

6. Relation to prior methods, implementation details, and limitations

The paper compares PMG-AFT against the original zero-shot CLIP, FT-Standard, FT-TeCoA, and prompt-based variants such as VP-TeCoA and VPT-PMG-AFT (Wang et al., 2024). Its explanation for the gain over FT-TeCoA is specific: FT-TeCoA mainly addresses robustness through adversarial example generation and text supervision, but does not sufficiently constrain the fine-tuned model to preserve the original CLIP’s generalized representation. PMG-AFT adds explicit supervision from the pre-trained model itself in feature or output space, thereby targeting overfitting directly. The paper also notes that PMG-AFT can be combined with adversarial example generation methods from prior work because it improves the parameter update phase rather than replacing the attack-generation strategy.

The implementation details reported in the paper are concrete. The backbone is CLIP ViT-B/32. Optimization uses SGD. Fine-tuning runs for 10 epochs. The learning rate for image encoder fine-tuning is $I$2. The hyperparameters are $I$3 and $I$4. Training uses two NVIDIA GeForce RTX 3090 GPUs. The auxiliary branch introduces additional compute overhead of about 298 seconds per training epoch.

The paper also delineates the method’s scope and limitations. PMG-AFT incurs additional computational cost because the original pre-trained model must run as an auxiliary teacher during training. The focus is on image-modality attacks: the text encoder is frozen and not updated. The main experiments are centered on CLIP fine-tuned on TinyImageNet, although CIFAR100 is also reported in supplementary results. These limitations indicate that the method is tailored to a particular fine-tuning regime and does not directly address text-side perturbations or fully joint multimodal adversarial adaptation.

Taken together, PMG-AFT can be situated as a pre-trained-model-guided adversarial fine-tuning procedure for CLIP in which robustness is improved by constraining the fine-tuned model to remain close to the original model’s adversarial-output behavior. In the reported experiments, this yields a better balance between zero-shot adversarial robustness and clean generalization than direct adversarial fine-tuning or prior baselines, while introducing a measurable computational overhead and retaining a frozen-text-encoder training design.