---
title: 'AdvFLYP: Adversarial Fine-Tune Like You Pretrain'
url: https://www.emergentmind.com/topics/adversarial-fine-tune-like-you-pretrain-advflyp
type: topic
---

# AdvFLYP: Adversarial Fine-Tune Like You Pretrain

Searching arXiv for the cited papers and related AdvFLYP work.
arXiv search query: "AdvFLYP adversarial fine-tune like you pretrain 2604.11576 RIFT 2112.11668"
Adversarial Fine-Tune Like You Pretrain (AdvFLYP) denotes a design principle for adversarial adaptation of pre-trained models: robustification should preserve the data distribution, objective structure, and representation geometry established during pre-training rather than treating pre-training as mere initialization. The term appears explicitly in two distinct but conceptually related lines of work. In NLP, the 2021 paper "How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?" argues that direct adversarial fine-tuning of pre-trained language models suffers from catastrophic forgetting and proposes Robust Informative Fine-Tuning (RIFT) to retain pre-trained information throughout fine-tuning [2112.11668]. In vision-language modeling, the 2026 paper "Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models" introduces AdvFLYP as a concrete CLIP finetuning paradigm that uses web image-text pairs and a contrastive objective matching CLIP’s pretraining recipe [2604.11576]. Across these formulations, the recurring thesis is that adversarial robustness improves when fine-tuning preserves reusable, robust features learned during pre-training instead of overwriting them.

## 1. Conceptual scope and historical framing

AdvFLYP is not a single algorithmic template shared verbatim across domains. It is a family resemblance among methods that regard adversarial fine-tuning as a continuation of pre-training rather than a downstream-only correction. The strongest explicit formulation is given in the CLIP-based work, where AdvFLYP is short for **Adversarially Finetune Like You Pretrain** and is motivated by the claim that existing adversarial finetuning pipelines robustify CLIP as if it were a classifier, even though CLIP was pretrained on web-scale image-text pairs with a contrastive objective [2604.11576].

A related but earlier conceptual message appears in RIFT. That paper states that conventional adversarial fine-tuning is misaligned with the fine-tuning setting because it intensifies catastrophic forgetting: fine-tuning already moves the model away from the pre-trained solution, adversarial examples are often out-of-distribution relative to pre-training, and the resulting updates can make the model forget generic and robust linguistic features learned during pre-training [2112.11668]. The paper explicitly describes this as the spirit of **“Adversarial Fine-Tune Like You Pretrain (AdvFLYP)”**, namely adversarially fine-tuning while continuously retaining the general and robust representations learned during pre-training [2112.11668].

This suggests that AdvFLYP is best understood as a robustness-oriented continuation-learning principle. A plausible implication is that the label applies most naturally to methods that preserve pre-trained invariants in representation space, objective space, or data distribution, rather than simply starting from pre-trained weights.

## 2. The central problem: why naive adversarial fine-tuning fails

The core failure mode identified in the literature is mismatch. In RIFT, the mismatch is between adversarial training and the fine-tuning dynamics of pre-trained language models. Standard fine-tuning uses a pre-trained encoder \(F_t(\cdot)\) only to initialize an objective model \(F_s(\cdot)\), which is then optimized on a small labeled downstream dataset. Standard adversarial fine-tuning augments this with adversarially perturbed inputs \(\hat{x}\), typically through an objective of the form
\[
-\log q(y|F_s(x)) + \beta \, \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(\hat{x}))\big).
\]
The paper argues that this direct procedure intensifies catastrophic forgetting because adversarial examples are often out-of-distribution relative to pre-training, and thus can push the model far from the pre-trained solution [2112.11668].

In the CLIP setting, the mismatch is formulated more concretely as both **data mismatch** and **objective mismatch**. Mainstream adversarial finetuning, exemplified by TeCoA, uses a labeled proxy dataset such as ImageNet and cross-entropy classification loss. AdvFLYP argues that this departs from CLIP pretraining in two ways: ImageNet-style labeled data are different from noisy web image-text pairs, and cross-entropy on class labels is not CLIP’s native objective, which is contrastive image-text alignment [2604.11576]. According to the paper, such mismatch improves robustness on the proxy classification benchmark but can hurt zero-shot performance and limit transferability across domains [2604.11576].

Related CLIP work reinforces the same diagnosis in a different form. PMG-AFT states that straightforward adversarial fine-tuning tends to overfit the downstream fine-tuning dataset, moving the target image encoder too far from the original CLIP representation and reducing zero-shot transfer [2401.04350]. Gen-AF, in the self-supervised encoder setting, similarly attributes failures of existing defenses to domain shift between pre-training and downstream data and to sensitivity of encoder parameters, which creates a trade-off between robustness and preserving pre-trained structure [2403.10801].

## 3. RIFT as an AdvFLYP formulation for pre-trained language models

RIFT formalizes the retention principle through an information-theoretic objective. Let \(X\) denote text input, \(Y\) the label, \(T = F_t(X)\) the feature from the fixed pre-trained model, and \(S = F_s(X)\) the feature from the fine-tuned objective model. The method aims to maximize
\[
\max I(S; Y, T),
\]
with decomposition
\[
I(S;Y,T)=I(S;Y)+I(S;T\mid Y).
\]
The first term, \(I(S;Y)\), corresponds to task-discriminative information and is optimized via standard classification loss. The second term, \(I(S;T\mid Y)\), is the novel retention term: it encourages the fine-tuned representation to preserve information from the pre-trained model conditioned on class label [2112.11668].

The conditional formulation is central. RIFT argues that maximizing unconditional \(I(S;T)\) encourages alignment plus uniformity over the whole data manifold, which may conflict with supervised class separation. By contrast, \(I(S;T\mid Y)\) encourages uniformity within each class-specific sub-manifold, making it more compatible with downstream classification [2112.11668]. The lower bound is implemented via a class-conditional contrastive objective with score function
\[
f_y(a,b)=\frac{1}{\tau}\frac{\langle g_y^1(a),g_y^2(b)\rangle}{\|g_y^1(a)\|_2\,\|g_y^2(b)\|_2},
\]
where \(g_y^1,g_y^2\) are MLP projections and \(\tau\) is the temperature [2112.11668].

In the adversarial setting, RIFT generates adversarial examples self-supervisedly to avoid label leaking:
\[
\hat{x} = \arg\max_{x'\in\mathbb B(x)} \mathrm{KL}\big(q(\cdot|F_s(x)) \,\|\, q(\cdot|F_s(x'))\big).
\]
It then combines a robust task loss and a robust information-retention loss. The robust task term is
\[
\mathcal{L}_{\text{r-task}} = \mathbb E_{x,y\sim p_{\mathcal D}} \Big[ -\log q(y|F_s(x)) + \beta\,\mathrm{KL}\big(q(\cdot|F_s(x))\|q(\cdot|F_s(\hat{x}))\big) \Big],
\]
while the robust information-retention term aligns adversarial features from the student with clean features from the fixed pre-trained teacher:
\[
\hat{s}_i = F_s(\hat{x}_i), \qquad t_i = F_t(x_i).
\]
The paper emphasizes that the pre-trained model is evaluated on clean inputs \(x_i\), not adversarial inputs \(\hat{x}_i\), reflecting the idea that the teacher should be queried in-domain [2112.11668]. The final objective is
\[
\min_{\theta,\phi,\varphi}\ \mathcal L_{\text{r-task} }+ \alpha \mathcal L_{\text{r-info}}.
\]

RIFT differs from parameter-space regularization methods such as pre-trained weight decay and Mixout by constraining function or representation space rather than weight space. The paper argues that this better reflects retained knowledge [2112.11668]. This suggests an important definitional property of AdvFLYP-style methods: preservation is imposed on behavior or representation, not merely on parameters.

## 4. AdvFLYP for CLIP: matching data and objective to pretraining

The 2026 AdvFLYP paper operationalizes the pretraining-continuation principle directly for CLIP [2604.11576]. CLIP notation is given by a vision encoder \(f_\theta(\cdot)\in\mathbb{R}^d\), a text encoder \(g_\phi(\cdot)\in\mathbb{R}^d\), and within-batch cosine similarities
\[
s_{ij}=\frac{f_{\theta}(x_i)^\intercal g_\phi(t_j)}{\parallel f_{\theta}(x_i)\parallel\parallel g_\phi(t_j)\parallel}.
\]
The CLIP pretraining loss is the symmetric contrastive loss
\[
\mathcal{L}_{CLIP} \left( \{(x_i,t_i)\}_{i=1}^N \,\vert\, \theta,\phi \right) = -\frac{1}{2N}\sum_{i=1}^{N}\left[ \log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^{N}\exp(s_{ij}/\tau)} + \log\frac{\exp(s_{ii}/\tau)}{\sum_{j=1}^{N}\exp(s_{ji}/\tau)} \right].
\]

AdvFLYP keeps both the pretraining-style data source and the objective. Because CLIP’s original 400M image-text corpus is not public, the method samples **1 million web image-text pairs** from **LAION-400M** with reachable URLs [2604.11576]. For each batch \(\{(x_i,t_i)\}_{i=1}^N\), it constructs perturbations jointly by maximizing the CLIP contrastive loss over the whole batch:
\[
\boldsymbol{\delta}_{adv} = \arg\max_{\{\delta_1,\dots,\delta_N\}} \mathcal{L}_{CLIP} \left(\{(x_i+\delta_i,t_i)\}_{i=1}^N \,\vert\, \theta,\phi \right), \quad s.t.\, \left\{ \|\delta_i\|_{\infty}\leq \epsilon \,|\, i=1,\dots,N \right\}.
\]
The batchwise optimization is essential because contrastive learning depends on within-batch negatives [2604.11576].

Finetuning updates only the vision encoder, leaving the text encoder frozen, by minimizing the same CLIP loss on adversarial image-text pairs:
\[
\theta' = \arg\min_{\theta} \mathcal{L}_{CLIP} \left( \{(x_i+\delta_i^{adv},t_i)\}_{i=1}^N \,\vert\, \theta,\phi \right).
\]
The conceptual difference from classifier-style adversarial training is explicit: adversarial images are aligned with their own texts, while all other texts in the batch serve as negatives [2604.11576].

The paper further introduces two regularizers to address noisy web data. Feature-level regularization penalizes drift in normalized image embeddings:
\[
\mathcal{L}_{feat}=\frac{1}{N}\left[ \|X_\theta^{adv}-X_{\theta_0}^{adv}\|_F + \|X_\theta^{adv}-X_\theta^{clean}\|_F \right],
\]
where \(F_{\theta_0}\) is the frozen original CLIP vision encoder [2604.11576]. Logit-level regularization penalizes divergence between similarity-to-text distributions:
\[
\mathcal{L}_{logit}=\frac{1}{N}\left[\mathrm{KL}(P_\theta^{adv}\|P_{\theta_0}^{adv}) + \mathrm{KL}(P_\theta^{adv}\|P_\theta^{clean})\right].
\]
The full objective is
\[
\theta' = \arg\min_\theta \{ \mathcal{L}_{CLIP} \left( \{(x_i+\delta_i,t_i)\}_{i=1}^N \,\vert\, \theta,\phi \right) + \mathcal{L}_{logit} + \mathcal{L}_{feat} \},
\]
and the regularized model is denoted \(\mathrm{AdvFLYP}_{full}\) [2604.11576].

## 5. Empirical results and ablations

The empirical evidence for AdvFLYP-style methods is domain-specific but consistent in pattern: robustness improves most reliably when fine-tuning explicitly preserves pre-trained information.

### RIFT results

RIFT is evaluated on **IMDB** sentiment analysis and **SNLI** natural language inference using **BERT-base-uncased** and **RoBERTa-base**, under **Genetic attack** and **PWWS** [2112.11668]. Robust accuracy is reported on 1000 random test examples from each dataset, alongside vanilla clean accuracy. RIFT is described as best across all datasets, pre-trained language models, and attacks [2112.11668].

| Setting | Robust accuracy |
|---|---:|
| IMDB, BERT, Genetic | 77.2 |
| IMDB, BERT, PWWS | 70.1 |
| SNLI, RoBERTa, Genetic | 83.5 |
| SNLI, RoBERTa, PWWS | 81.1 |

The paper also reports clean accuracies including **78.3** for IMDB BERT, **84.2** for IMDB RoBERTa, **80.5** for SNLI BERT, and **87.9** for SNLI RoBERTa [2112.11668]. It states that the method improves or preserves clean accuracy better than adversarial baselines, rather than merely trading clean accuracy for robustness [2112.11668].

The ablations are particularly relevant to the AdvFLYP interpretation. Conditional mutual information \(I(S;T\mid Y)\) consistently outperforms unconditional \(I(S;T)\); for example, on IMDB with BERT, \(I(S;T\mid Y)\) gives **77.2 / 70.1** under Genetic/PWWS, while \(I(S;T)\) gives **76.1 / 69.4**, and on SNLI with RoBERTa, \(I(S;T\mid Y)\) gives **83.5 / 81.1** versus **82.5 / 79.4** for \(I(S;T)\) [2112.11668]. The \(\alpha\) ablation shows that increasing the weight on information retention initially improves both clean and robust performance, but too large an \(\alpha\) can overemphasize preservation and underfit the downstream task [2112.11668].

### CLIP AdvFLYP results

The CLIP AdvFLYP paper evaluates zero-shot adversarial robustness on **14 downstream datasets** using **CLIP ViT-B/32**, with **PGD-2** adversarial generation during finetuning, attack budget **\(\epsilon=1/255\)** during training and many evaluations, and stronger evaluations at **\(\epsilon=2/255\)** and **\(\epsilon=4/255\)** [2604.11576]. Only the vision encoder is finetuned; the text encoder is frozen [2604.11576].

At \(\epsilon=1/255\), averaged over 14 datasets and three attacks, the paper reports:

| Method | Average robust accuracy | Clean accuracy |
|---|---:|---:|
| TeCoA | 33.00% | 51.35% |
| PMG-AFT | 34.87% | 54.25% |
| AdvFLYP | 35.61% | 54.27% |
| AdvFLYP\(_{full}\) | 38.39% | 55.84% |

The gains persist under stronger perturbations. At \(\epsilon=2/255\), average robust accuracy is **17.71%** for TeCoA, **18.31%** for PMG-AFT, **20.07%** for AdvFLYP, and **21.69%** for AdvFLYP\(_{full}\). At \(\epsilon=4/255\), the corresponding values are **4.09%**, **4.00%**, **5.87%**, and **5.93%** [2604.11576]. The paper also states that AdvFLYP\(_{full}\) shows better or comparable transfer across ImageNet-R, ImageNet-A, ImageNet-Sketch, and ObjectNet [2604.11576].

The regularization ablation shows a division of labor between the two regularizers. The paper reports:
- **AdvFLYP**: AutoAttack \(34.94\%\), clean \(54.27\%\)
- **AdvFLYP + \(\mathcal{L}_{logit}\)**: AutoAttack \(37.40\%\), clean \(53.86\%\)
- **AdvFLYP + \(\mathcal{L}_{feat}\)**: AutoAttack \(35.01\%\), clean \(57.50\%\)
- **AdvFLYP + both**: AutoAttack \(37.74\%\), clean \(55.84\%\) [2604.11576]

This indicates that \(\mathcal{L}_{logit}\) primarily improves robustness transfer, while \(\mathcal{L}_{feat}\) primarily preserves clean zero-shot accuracy [2604.11576]. The paper also reports that more web image-text pairs improve both robustness and clean accuracy, though **1M** pairs are fixed for efficiency, and that very small batch sizes hurt performance because contrastive learning relies on within-batch negatives [2604.11576].

## 6. Related methods, adjacent formulations, and common misconceptions

Several adjacent methods instantiate parts of the AdvFLYP logic without adopting the exact label. PMG-AFT uses a frozen original CLIP as a guide during adversarial fine-tuning, minimizing the KL divergence between adversarial outputs of the target model and the original model, plus a regularizer comparing adversarial and clean outputs:
\[
L = L_{robust} + \alpha L_{general} + \beta L_{clean}.
\]
The paper reports that on TinyImageNet fine-tuning, PMG-AFT improves average robust accuracy to **31.95%** versus **26.96%** for FT-TeCoA, with **55.71%** average clean accuracy, and states a **+4.99%** average robust improvement over FT-TeCoA [2401.04350]. PMG-AFT therefore exemplifies pre-trained-model guidance via output-space alignment rather than pretraining-recipe matching.

Gen-AF addresses publicly available self-supervised encoders under downstream-agnostic adversarial examples. Its Stage I combines adversarial training with graph-based genetic regularization,
\[
\mathcal{L}_{GDAT} = \mathcal{L}_{dat} + \lambda \mathcal{L}_{gr},
\]
while Stage II selectively fine-tunes top-\(k\) robust-redundant layers to recover clean accuracy [2403.10801]. The paper argues that robustness in the pre-training paradigm must preserve inherited representation geometry, not merely apply generic adversarial training [2403.10801].

The 2020 adversarial fine-tuning paper on image classification gives a different route: clean pretraining followed by short adversarial fine-tuning with a **slow start, fast decay** learning-rate schedule [2012.13628]. Its emphasis is not explicit representation matching, but it still shares the view that a pretrained model already has a useful natural-data representation and should not be adversarially retrained from scratch. On ImageNet, **AFT (ResNet50-SWSL)** achieves **74.5** clean and **50.5** PGD robustness at \(\epsilon=2/255\), while using **32 h** rather than the roughly **280 h** reported for PGD AT [2012.13628].

A common misconception is that AdvFLYP is synonymous with “any adversarial fine-tuning starting from pretrained weights.” The papers do not support that broad usage. In the stricter sense used by RIFT and the 2026 CLIP paper, the defining property is ongoing preservation of pre-trained information or explicit fidelity to the pretraining recipe, not initialization alone [2112.11668; 2604.11576].

A second misconception is that AdvFLYP always means feature matching. The evidence is more heterogeneous. RIFT uses conditional mutual information in representation space [2112.11668]; PMG-AFT finds that output-layer KL is better than feature-level alternatives [2401.04350]; CLIP AdvFLYP uses both feature-level and logit-level regularization, with different effects [2604.11576]. This suggests that the principle is broader than any single preservation mechanism.

## 7. Limitations, open questions, and broader significance

The papers are explicit that AdvFLYP-style methods do not remove the robustness–generalization tension entirely. In RIFT, too large an information-retention coefficient \(\alpha\) can underfit the downstream task [2112.11668]. In CLIP AdvFLYP, the method depends on noisy web-collected image-text pairs that are not the original CLIP corpus, and the approach is demonstrated mainly on **CLIP ViT-B/32**, with only preliminary evidence on **ViT-B/16** [2604.11576]. The paper also notes that larger batches and more data would likely help further but are constrained by compute [2604.11576].

Another limitation is domain specificity. RIFT addresses text classification under word-substitution attacks such as Genetic attack and PWWS [2112.11668]. The CLIP AdvFLYP work is focused on zero-shot adversarial robustness in vision-language models [2604.11576]. Gen-AF studies publicly available self-supervised encoders under downstream-agnostic universal perturbations [2403.10801]. These are related but not interchangeable threat models.

A broader open question is what aspect of pre-training must be preserved: objective, data distribution, feature geometry, output distribution, parameter topology, or some combination. The existing literature supports several answers. RIFT emphasizes class-conditional mutual information with pre-trained representations [2112.11668]. PMG-AFT emphasizes output-level agreement with the frozen original model [2401.04350]. CLIP AdvFLYP emphasizes fidelity to both pretraining data distribution and contrastive objective [2604.11576]. Gen-AF emphasizes topology stability and cautious encoder updates [2403.10801].

Taken together, these works position AdvFLYP as a general research direction in robust transfer learning. Its central claim is that adversarial robustness in pre-trained systems depends not only on adversarial optimization itself, but on whether fine-tuning continues the inductive biases of pre-training. The current evidence indicates that methods faithful to pre-training can improve robustness while preserving clean or zero-shot performance more effectively than downstream-only adversarial adaptation [2112.11668; 2604.11576; 2401.04350; 2403.10801].

Source: https://www.emergentmind.com/topics/adversarial-fine-tune-like-you-pretrain-advflyp