AdvFLYP: Adversarial Fine-Tune Like You Pretrain

• The paper demonstrates that AdvFLYP preserves pre-trained data distribution, objective structure, and representation geometry to enhance robust performance.
• Methods like RIFT and CLIP AdvFLYP employ conditional mutual information and contrastive objectives to maintain robust, reusable representations during fine-tuning.
• Empirical results indicate that preserving pre-trained invariants mitigates catastrophic forgetting, boosting both clean and adversarial robustness across various models.

Searching arXiv for the cited papers and related AdvFLYP work. arXiv search query: "AdvFLYP adversarial fine-tune like you pretrain (Xing et al., 13 Apr 2026) RIFT (Dong et al., 2021)" Adversarial Fine-Tune Like You Pretrain (AdvFLYP) denotes a design principle for adversarial adaptation of pre-trained models: robustification should preserve the data distribution, objective structure, and representation geometry established during pre-training rather than treating pre-training as mere initialization. The term appears explicitly in two distinct but conceptually related lines of work. In NLP, the 2021 paper "How Should Pre-Trained LLMs Be Fine-Tuned Towards Adversarial Robustness?" argues that direct adversarial fine-tuning of pre-trained LLMs suffers from catastrophic forgetting and proposes Robust Informative Fine-Tuning (RIFT) to retain pre-trained information throughout fine-tuning (Dong et al., 2021). In vision-language modeling, the 2026 paper "Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-LLMs" introduces AdvFLYP as a concrete CLIP finetuning paradigm that uses web image-text pairs and a contrastive objective matching CLIP’s pretraining recipe (Xing et al., 13 Apr 2026). Across these formulations, the recurring thesis is that adversarial robustness improves when fine-tuning preserves reusable, robust features learned during pre-training instead of overwriting them.

1. Conceptual scope and historical framing

AdvFLYP is not a single algorithmic template shared verbatim across domains. It is a family resemblance among methods that regard adversarial fine-tuning as a continuation of pre-training rather than a downstream-only correction. The strongest explicit formulation is given in the CLIP-based work, where AdvFLYP is short for Adversarially Finetune Like You Pretrain and is motivated by the claim that existing adversarial finetuning pipelines robustify CLIP as if it were a classifier, even though CLIP was pretrained on web-scale image-text pairs with a contrastive objective (Xing et al., 13 Apr 2026).

A related but earlier conceptual message appears in RIFT. That paper states that conventional adversarial fine-tuning is misaligned with the fine-tuning setting because it intensifies catastrophic forgetting: fine-tuning already moves the model away from the pre-trained solution, adversarial examples are often out-of-distribution relative to pre-training, and the resulting updates can make the model forget generic and robust linguistic features learned during pre-training (Dong et al., 2021). The paper explicitly describes this as the spirit of “Adversarial Fine-Tune Like You Pretrain (AdvFLYP)”, namely adversarially fine-tuning while continuously retaining the general and robust representations learned during pre-training (Dong et al., 2021).

This suggests that AdvFLYP is best understood as a robustness-oriented continuation-learning principle. A plausible implication is that the label applies most naturally to methods that preserve pre-trained invariants in representation space, objective space, or data distribution, rather than simply starting from pre-trained weights.

2. The central problem: why naive adversarial fine-tuning fails

The core failure mode identified in the literature is mismatch. In RIFT, the mismatch is between adversarial training and the fine-tuning dynamics of pre-trained LLMs. Standard fine-tuning uses a pre-trained encoder $F_t(\cdot)$ only to initialize an objective model $F_s(\cdot)$, which is then optimized on a small labeled downstream dataset. Standard adversarial fine-tuning augments this with adversarially perturbed inputs $\hat{x}$, typically through an objective of the form

$$-\log q(y|F_s(x)) + \beta \, \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(\hat{x}))\big).$$

The paper argues that this direct procedure intensifies catastrophic forgetting because adversarial examples are often out-of-distribution relative to pre-training, and thus can push the model far from the pre-trained solution (Dong et al., 2021).

In the CLIP setting, the mismatch is formulated more concretely as both data mismatch and objective mismatch. Mainstream adversarial finetuning, exemplified by TeCoA, uses a labeled proxy dataset such as ImageNet and cross-entropy classification loss. AdvFLYP argues that this departs from CLIP pretraining in two ways: ImageNet-style labeled data are different from noisy web image-text pairs, and cross-entropy on class labels is not CLIP’s native objective, which is contrastive image-text alignment (Xing et al., 13 Apr 2026). According to the paper, such mismatch improves robustness on the proxy classification benchmark but can hurt zero-shot performance and limit transferability across domains (Xing et al., 13 Apr 2026).

Related CLIP work reinforces the same diagnosis in a different form. PMG-AFT states that straightforward adversarial fine-tuning tends to overfit the downstream fine-tuning dataset, moving the target image encoder too far from the original CLIP representation and reducing zero-shot transfer (Wang et al., 2024). Gen-AF, in the self-supervised encoder setting, similarly attributes failures of existing defenses to domain shift between pre-training and downstream data and to sensitivity of encoder parameters, which creates a trade-off between robustness and preserving pre-trained structure (Zhou et al., 2024).

3. RIFT as an AdvFLYP formulation for pre-trained LLMs

RIFT formalizes the retention principle through an information-theoretic objective. Let $X$ denote text input, $Y$ the label, $T = F_t(X)$ the feature from the fixed pre-trained model, and $S = F_s(X)$ the feature from the fine-tuned objective model. The method aims to maximize

$\max I(S; Y, T),$

with decomposition

$I(S;Y,T)=I(S;Y)+I(S;T\mid Y).$

The first term, $F_s(\cdot)$0, corresponds to task-discriminative information and is optimized via standard classification loss. The second term, $F_s(\cdot)$1, is the novel retention term: it encourages the fine-tuned representation to preserve information from the pre-trained model conditioned on class label (Dong et al., 2021).

The conditional formulation is central. RIFT argues that maximizing unconditional $F_s(\cdot)$2 encourages alignment plus uniformity over the whole data manifold, which may conflict with supervised class separation. By contrast, $F_s(\cdot)$3 encourages uniformity within each class-specific sub-manifold, making it more compatible with downstream classification (Dong et al., 2021). The lower bound is implemented via a class-conditional contrastive objective with score function

$F_s(\cdot)$4

where $F_s(\cdot)$5 are MLP projections and $F_s(\cdot)$6 is the temperature (Dong et al., 2021).

In the adversarial setting, RIFT generates adversarial examples self-supervisedly to avoid label leaking: $F_s(\cdot)$7 It then combines a robust task loss and a robust information-retention loss. The robust task term is

$F_s(\cdot)$8

while the robust information-retention term aligns adversarial features from the student with clean features from the fixed pre-trained teacher: $F_s(\cdot)$9 The paper emphasizes that the pre-trained model is evaluated on clean inputs $\hat{x}$0, not adversarial inputs $\hat{x}$1, reflecting the idea that the teacher should be queried in-domain (Dong et al., 2021). The final objective is

$\hat{x}$2

RIFT differs from parameter-space regularization methods such as pre-trained weight decay and Mixout by constraining function or representation space rather than weight space. The paper argues that this better reflects retained knowledge (Dong et al., 2021). This suggests an important definitional property of AdvFLYP-style methods: preservation is imposed on behavior or representation, not merely on parameters.

4. AdvFLYP for CLIP: matching data and objective to pretraining

The 2026 AdvFLYP paper operationalizes the pretraining-continuation principle directly for CLIP (Xing et al., 13 Apr 2026). CLIP notation is given by a vision encoder $\hat{x}$3, a text encoder $\hat{x}$4, and within-batch cosine similarities

$\hat{x}$5

The CLIP pretraining loss is the symmetric contrastive loss

$\hat{x}$6

AdvFLYP keeps both the pretraining-style data source and the objective. Because CLIP’s original 400M image-text corpus is not public, the method samples 1 million web image-text pairs from LAION-400M with reachable URLs (Xing et al., 13 Apr 2026). For each batch $\hat{x}$7, it constructs perturbations jointly by maximizing the CLIP contrastive loss over the whole batch: $\hat{x}$8 The batchwise optimization is essential because contrastive learning depends on within-batch negatives (Xing et al., 13 Apr 2026).

Finetuning updates only the vision encoder, leaving the text encoder frozen, by minimizing the same CLIP loss on adversarial image-text pairs: $\hat{x}$9 The conceptual difference from classifier-style adversarial training is explicit: adversarial images are aligned with their own texts, while all other texts in the batch serve as negatives (Xing et al., 13 Apr 2026).

The paper further introduces two regularizers to address noisy web data. Feature-level regularization penalizes drift in normalized image embeddings: $$-\log q(y|F_s(x)) + \beta \, \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(\hat{x}))\big).$$0 where $$-\log q(y|F_s(x)) + \beta \, \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(\hat{x}))\big).$$1 is the frozen original CLIP vision encoder (Xing et al., 13 Apr 2026). Logit-level regularization penalizes divergence between similarity-to-text distributions: $$-\log q(y|F_s(x)) + \beta \, \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(\hat{x}))\big).$$2 The full objective is

$$-\log q(y|F_s(x)) + \beta \, \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(\hat{x}))\big).$$3

and the regularized model is denoted $$-\log q(y|F_s(x)) + \beta \, \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(\hat{x}))\big).$$4 (Xing et al., 13 Apr 2026).

5. Empirical results and ablations

The empirical evidence for AdvFLYP-style methods is domain-specific but consistent in pattern: robustness improves most reliably when fine-tuning explicitly preserves pre-trained information.

RIFT results

RIFT is evaluated on IMDB sentiment analysis and SNLI natural language inference using BERT-base-uncased and RoBERTa-base, under Genetic attack and PWWS (Dong et al., 2021). Robust accuracy is reported on 1000 random test examples from each dataset, alongside vanilla clean accuracy. RIFT is described as best across all datasets, pre-trained LLMs, and attacks (Dong et al., 2021).

Setting	Robust accuracy
IMDB, BERT, Genetic	77.2
IMDB, BERT, PWWS	70.1
SNLI, RoBERTa, Genetic	83.5
SNLI, RoBERTa, PWWS	81.1

The paper also reports clean accuracies including 78.3 for IMDB BERT, 84.2 for IMDB RoBERTa, 80.5 for SNLI BERT, and 87.9 for SNLI RoBERTa (Dong et al., 2021). It states that the method improves or preserves clean accuracy better than adversarial baselines, rather than merely trading clean accuracy for robustness (Dong et al., 2021).

The ablations are particularly relevant to the AdvFLYP interpretation. Conditional mutual information $$-\log q(y|F_s(x)) + \beta \, \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(\hat{x}))\big).$$5 consistently outperforms unconditional $$-\log q(y|F_s(x)) + \beta \, \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(\hat{x}))\big).$$6; for example, on IMDB with BERT, $$-\log q(y|F_s(x)) + \beta \, \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(\hat{x}))\big).$$7 gives 77.2 / 70.1 under Genetic/PWWS, while $$-\log q(y|F_s(x)) + \beta \, \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(\hat{x}))\big).$$8 gives 76.1 / 69.4, and on SNLI with RoBERTa, $$-\log q(y|F_s(x)) + \beta \, \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(\hat{x}))\big).$$9 gives 83.5 / 81.1 versus 82.5 / 79.4 for $X$0 (Dong et al., 2021). The $X$1 ablation shows that increasing the weight on information retention initially improves both clean and robust performance, but too large an $X$2 can overemphasize preservation and underfit the downstream task (Dong et al., 2021).

CLIP AdvFLYP results

The CLIP AdvFLYP paper evaluates zero-shot adversarial robustness on 14 downstream datasets using CLIP ViT-B/32, with PGD-2 adversarial generation during finetuning, attack budget $X$3 during training and many evaluations, and stronger evaluations at $X$4 and $X$5 (Xing et al., 13 Apr 2026). Only the vision encoder is finetuned; the text encoder is frozen (Xing et al., 13 Apr 2026).

At $X$6, averaged over 14 datasets and three attacks, the paper reports:

Method	Average robust accuracy	Clean accuracy
TeCoA	33.00%	51.35%
PMG-AFT	34.87%	54.25%
AdvFLYP	35.61%	54.27%
AdvFLYP$X$7	38.39%	55.84%

The gains persist under stronger perturbations. At $X$8, average robust accuracy is 17.71% for TeCoA, 18.31% for PMG-AFT, 20.07% for AdvFLYP, and 21.69% for AdvFLYP$X$9. At $Y$0, the corresponding values are 4.09%, 4.00%, 5.87%, and 5.93% (Xing et al., 13 Apr 2026). The paper also states that AdvFLYP$Y$1 shows better or comparable transfer across ImageNet-R, ImageNet-A, ImageNet-Sketch, and ObjectNet (Xing et al., 13 Apr 2026).

The regularization ablation shows a division of labor between the two regularizers. The paper reports:

• AdvFLYP: AutoAttack $Y$2, clean $Y$3
• AdvFLYP + $Y$4: AutoAttack $Y$5, clean $Y$6
• AdvFLYP + $Y$7: AutoAttack $Y$8, clean $Y$9
• AdvFLYP + both: AutoAttack $T = F_t(X)$0, clean $T = F_t(X)$1 (Xing et al., 13 Apr 2026)

This indicates that $T = F_t(X)$2 primarily improves robustness transfer, while $T = F_t(X)$3 primarily preserves clean zero-shot accuracy (Xing et al., 13 Apr 2026). The paper also reports that more web image-text pairs improve both robustness and clean accuracy, though 1M pairs are fixed for efficiency, and that very small batch sizes hurt performance because contrastive learning relies on within-batch negatives (Xing et al., 13 Apr 2026).

6. Related methods, adjacent formulations, and common misconceptions

Several adjacent methods instantiate parts of the AdvFLYP logic without adopting the exact label. PMG-AFT uses a frozen original CLIP as a guide during adversarial fine-tuning, minimizing the KL divergence between adversarial outputs of the target model and the original model, plus a regularizer comparing adversarial and clean outputs: $T = F_t(X)$4 The paper reports that on TinyImageNet fine-tuning, PMG-AFT improves average robust accuracy to 31.95% versus 26.96% for FT-TeCoA, with 55.71% average clean accuracy, and states a +4.99% average robust improvement over FT-TeCoA (Wang et al., 2024). PMG-AFT therefore exemplifies pre-trained-model guidance via output-space alignment rather than pretraining-recipe matching.

Gen-AF addresses publicly available self-supervised encoders under downstream-agnostic adversarial examples. Its Stage I combines adversarial training with graph-based genetic regularization,

$T = F_t(X)$5

while Stage II selectively fine-tunes top-$T = F_t(X)$6 robust-redundant layers to recover clean accuracy (Zhou et al., 2024). The paper argues that robustness in the pre-training paradigm must preserve inherited representation geometry, not merely apply generic adversarial training (Zhou et al., 2024).

The 2020 adversarial fine-tuning paper on image classification gives a different route: clean pretraining followed by short adversarial fine-tuning with a slow start, fast decay learning-rate schedule (Jeddi et al., 2020). Its emphasis is not explicit representation matching, but it still shares the view that a pretrained model already has a useful natural-data representation and should not be adversarially retrained from scratch. On ImageNet, AFT (ResNet50-SWSL) achieves 74.5 clean and 50.5 PGD robustness at $T = F_t(X)$7, while using 32 h rather than the roughly 280 h reported for PGD AT (Jeddi et al., 2020).

A common misconception is that AdvFLYP is synonymous with “any adversarial fine-tuning starting from pretrained weights.” The papers do not support that broad usage. In the stricter sense used by RIFT and the 2026 CLIP paper, the defining property is ongoing preservation of pre-trained information or explicit fidelity to the pretraining recipe, not initialization alone (Dong et al., 2021, Xing et al., 13 Apr 2026).

A second misconception is that AdvFLYP always means feature matching. The evidence is more heterogeneous. RIFT uses conditional mutual information in representation space (Dong et al., 2021); PMG-AFT finds that output-layer KL is better than feature-level alternatives (Wang et al., 2024); CLIP AdvFLYP uses both feature-level and logit-level regularization, with different effects (Xing et al., 13 Apr 2026). This suggests that the principle is broader than any single preservation mechanism.

7. Limitations, open questions, and broader significance

The papers are explicit that AdvFLYP-style methods do not remove the robustness–generalization tension entirely. In RIFT, too large an information-retention coefficient $T = F_t(X)$8 can underfit the downstream task (Dong et al., 2021). In CLIP AdvFLYP, the method depends on noisy web-collected image-text pairs that are not the original CLIP corpus, and the approach is demonstrated mainly on CLIP ViT-B/32, with only preliminary evidence on ViT-B/16 (Xing et al., 13 Apr 2026). The paper also notes that larger batches and more data would likely help further but are constrained by compute (Xing et al., 13 Apr 2026).

Another limitation is domain specificity. RIFT addresses text classification under word-substitution attacks such as Genetic attack and PWWS (Dong et al., 2021). The CLIP AdvFLYP work is focused on zero-shot adversarial robustness in vision-LLMs (Xing et al., 13 Apr 2026). Gen-AF studies publicly available self-supervised encoders under downstream-agnostic universal perturbations (Zhou et al., 2024). These are related but not interchangeable threat models.

A broader open question is what aspect of pre-training must be preserved: objective, data distribution, feature geometry, output distribution, parameter topology, or some combination. The existing literature supports several answers. RIFT emphasizes class-conditional mutual information with pre-trained representations (Dong et al., 2021). PMG-AFT emphasizes output-level agreement with the frozen original model (Wang et al., 2024). CLIP AdvFLYP emphasizes fidelity to both pretraining data distribution and contrastive objective (Xing et al., 13 Apr 2026). Gen-AF emphasizes topology stability and cautious encoder updates (Zhou et al., 2024).

Taken together, these works position AdvFLYP as a general research direction in robust transfer learning. Its central claim is that adversarial robustness in pre-trained systems depends not only on adversarial optimization itself, but on whether fine-tuning continues the inductive biases of pre-training. The current evidence indicates that methods faithful to pre-training can improve robustness while preserving clean or zero-shot performance more effectively than downstream-only adversarial adaptation (Dong et al., 2021, Xing et al., 13 Apr 2026, Wang et al., 2024, Zhou et al., 2024).