Quality Text-guided Adversarial Fine-Tuning

• The paper demonstrates that integrating high-quality captions in adversarial fine-tuning enhances both clean accuracy and zero-shot robustness across diverse datasets.
• It leverages a combined loss function that mixes frozen embedding anchoring with caption-guided contrastive supervision to drive semantically rich perturbations.
• Empirical results over 16 datasets show state-of-the-art performance improvements compared to existing robust CLIP adaptation methods.

Quality Text-guided Adversarial Fine-Tuning (QT-AFT) is a robust fine-tuning method for pre-trained vision-LLMs, especially CLIP, in the setting of zero-shot adversarial robustness. It is designed for the regime in which a model is adversarially fine-tuned on a source dataset such as ImageNet but is expected to retain both clean accuracy and adversarial robustness on a broad set of unseen downstream zero-shot classification tasks. Its central claim is that adversarial supervision for vision-LLMs should be guided not by short class labels alone, and not by text-free embedding perturbation alone, but by high-quality image captions that expose broader image semantics during training (Waseda et al., 22 Jul 2025).

1. Problem setting and conceptual basis

QT-AFT addresses robust fine-tuning of pre-trained vision-LLMs in a zero-shot transfer setting. The target problem is not merely to preserve accuracy on the source dataset under perturbation, but to maintain clean and robust zero-shot performance across unseen downstream tasks after adversarial fine-tuning on ImageNet. This requirement distinguishes QT-AFT from ordinary supervised adversarial training, because a CLIP-style model is valued precisely for its transferable image-text representation geometry rather than for performance on a fixed label set alone (Waseda et al., 22 Jul 2025).

The method is motivated by a two-part critique of prior robust CLIP adaptation. First, supervised adversarial training methods rely on short texts such as class labels or prompts like “a photo of {class}.” QT-AFT argues that this induces overfitting to the training distribution because class labels are too short and semantically narrow, emphasize only object identity, and optimize perturbations against alignment with a small set of class names rather than the diverse semantics present in an image. Second, unsupervised adversarial training avoids this overfitting but remains suboptimal because it does not use text during adversarial example generation; consequently, it lacks semantic awareness despite the fact that practical attacks on CLIP-like models are often text-guided (Waseda et al., 22 Jul 2025).

Its main hypothesis is that high-quality language supervision improves visual robustness because it encourages adversarial perturbations to push images away from a broader and richer set of semantics than short class labels do. In this formulation, robustness is tied to preservation of multiple objects, object attributes, scene context, relations, and descriptive properties, rather than to class-name alignment alone. A plausible implication is that QT-AFT treats language not as metadata but as a mechanism for shaping which semantic directions adversarial training explores (Waseda et al., 22 Jul 2025).

2. “Quality text” and caption construction

In QT-AFT, “quality text” means high-quality image captions with richer semantics than class labels. The paper identifies useful caption content as object names, object properties such as color, size, texture, and shape, actions or states, spatial and contextual relations, scene-level details, and natural sentence structure and word order. The associated analysis further links quality to semantic richness, coverage of more visual features, longer and more detailed descriptions, inclusion of adjectives, adverbs, and function words, and preservation of word order (Waseda et al., 22 Jul 2025).

Captions are pre-generated before training. The default setup uses the multimodal model InternVL-2.5-8B with the prompt “Describe the image in detail within 50 words.” Captions are constrained to about 50 words because CLIP’s text encoder has a 77-token limit, which corresponds to roughly 50 words. The paper notes that captions could in principle come from human annotation, image-to-text models, or web-scraped descriptions, but the experiments use synthetic captions generated for ImageNet (Waseda et al., 22 Jul 2025).

The training design retains a frozen copy of the original image encoder as a reference and uses the text encoder to embed captions while adversarially fine-tuning the vision encoder. The formulation and surrounding discussion imply that QT-AFT updates the image encoder rather than the text encoder. Conceptually, the captions define the supervised semantic attack direction during adversarial example generation, while the frozen original image encoder supplies a consistency-style unsupervised anchor (Waseda et al., 22 Jul 2025).

3. Objective function and optimization procedure

QT-AFT is formulated against two baselines. For zero-shot classification with class templates $c_k$, CLIP predicts

$$\arg\max_{k=1,\ldots,K} \cos(f_{\theta}(x), f_{\phi}(c_k)).$$

A supervised adversarial fine-tuning baseline such as TeCoA uses class-label supervision,

$$\mathcal{L}_{\text{TeCoA}}(x,y) = - \log \left( \frac{ \exp(\cos(f_{\theta}(x), f_{\phi}(c_y))) }{ \sum_{k=1}^{K}\exp(\cos(f_{\theta}(x), f_{\phi}(c_k))) } \right),$$

while the unsupervised baseline FARE is written as

$$\theta = \arg\min_{\theta} \mathbb{E}_{(x,y)\sim\mathcal{D}} \left[ \max_{x' \in B(x,\epsilon)} \|f_\theta(x') - f_{\theta_{\text{orig}}}(x)\|_2^2 \right].$$

QT-AFT combines these two perspectives by using both a frozen-image-embedding deviation term and a caption-guided contrastive term. Its loss is

$$\mathcal{L}_{\text{QT-AFT}}(x',t) = \sum_{i=1}^{N} \left[ \left\| f_\theta(x_i') - f_{\theta_{\text{orig}}}(x_i) \right\|_2^2 - \lambda \cdot \log \frac{ \exp\left(\cos(f_{\theta}(x_i'), f_{\phi}(t_i)) / \tau\right) }{ \sum_{j=1}^{N} \exp\left(\cos(f_{\theta}(x_i'), f_{\phi}(t_j)) / \tau\right) } \right].$$

The full adversarial fine-tuning problem is

$$\theta = \arg\min_{\theta} \mathbb{E}_{(x,t)\sim\mathcal{D}} \left[ \max_{x' \in B(x,\epsilon)} \mathcal{L}_{\text{QT-AFT}}(x',t) \right].$$

The first term encourages perturbations that move the adversarial image away from the frozen original clean embedding. The second term is a CLIP-style image-to-text contrastive term using the caption. Because the inner optimization maximizes the loss, the adversary seeks perturbations that both distort the image embedding globally and worsen alignment with the correct caption relative to other captions. The paper reports $\lambda = 10$ as the best value (Waseda et al., 22 Jul 2025).

The training pipeline is straightforward. Starting from pre-trained CLIP, QT-AFT precomputes a caption $t_i$ for every ImageNet training image $x_i$, keeps a frozen copy of the original image encoder $f_{\theta_{\text{orig}}}$, generates adversarial images $$\arg\max_{k=1,\ldots,K} \cos(f_{\theta}(x), f_{\phi}(c_k)).$$0 within an $$\arg\max_{k=1,\ldots,K} \cos(f_{\theta}(x), f_{\phi}(c_k)).$$1 ball using PGD to maximize the QT-AFT loss, and updates the current image encoder to minimize the resulting min-max objective. The main experiments use CLIP ViT-B/16, train on ImageNet for 2 epochs, and use AdamW with learning rate $$\arg\max_{k=1,\ldots,K} \cos(f_{\theta}(x), f_{\phi}(c_k)).$$2, weight decay $$\arg\max_{k=1,\ldots,K} \cos(f_{\theta}(x), f_{\phi}(c_k)).$$3, cosine learning-rate scheduling, and batch size 128. Training attacks are 10-step PGD with $$\arg\max_{k=1,\ldots,K} \cos(f_{\theta}(x), f_{\phi}(c_k)).$$4 norm, $$\arg\max_{k=1,\ldots,K} \cos(f_{\theta}(x), f_{\phi}(c_k)).$$5, and step size $$\arg\max_{k=1,\ldots,K} \cos(f_{\theta}(x), f_{\phi}(c_k)).$$6 (Waseda et al., 22 Jul 2025).

4. Experimental protocol and empirical results

QT-AFT is evaluated on 16 downstream zero-shot datasets: ImageNet-S, ImageNet-R, CIFAR-10, CIFAR-100, STL-10, Caltech101, Caltech256, OxfordPets, Flowers102, FGVCAircraft, StanfordCars, SUN397, Food101, EuroSAT, DTD, and PCAM. The main evaluation uses AutoAttack with $$\arg\max_{k=1,\ldots,K} \cos(f_{\theta}(x), f_{\phi}(c_k)).$$7, $$\arg\max_{k=1,\ldots,K} \cos(f_{\theta}(x), f_{\phi}(c_k)).$$8, on 1,000 random samples per dataset for computational reasons, with additional appendix evaluations under PGD-10, $$\arg\max_{k=1,\ldots,K} \cos(f_{\theta}(x), f_{\phi}(c_k)).$$9-PGD with $$\mathcal{L}_{\text{TeCoA}}(x,y) = - \log \left( \frac{ \exp(\cos(f_{\theta}(x), f_{\phi}(c_y))) }{ \sum_{k=1}^{K}\exp(\cos(f_{\theta}(x), f_{\phi}(c_k))) } \right),$$0, and CW. The paper emphasizes Float32 evaluation for stronger and more reliable attacks (Waseda et al., 22 Jul 2025).

For CLIP ViT-B/16, QT-AFT reports the best average clean accuracy and the best average robust accuracy among the listed robust fine-tuning baselines on the 16-dataset AutoAttack benchmark.

Method	Average clean	Average robust
TeCoA	43.3	21.8
PMG-AFT	43.1	23.6
TGA-ZSR	51.1	10.3
FARE	50.3	22.4
QT-AFT	51.6	25.8

QT-AFT is described as achieving state-of-the-art zero-shot robustness on 12 out of 16 datasets. The averages correspond to a gain of 3.4 points in robust accuracy and 1.3 points in clean accuracy over FARE. The margins are larger under appendix PGD-10 full-sample evaluation, where QT-AFT reaches average zero-shot robustness 30.14 versus 25.55 for FARE, 25.48 for PMG-AFT, 24.13 for TeCoA, and 15.81 for TGA-ZSR. Under $$\mathcal{L}_{\text{TeCoA}}(x,y) = - \log \left( \frac{ \exp(\cos(f_{\theta}(x), f_{\phi}(c_y))) }{ \sum_{k=1}^{K}\exp(\cos(f_{\theta}(x), f_{\phi}(c_k))) } \right),$$1-PGD, QT-AFT reports 44.14 average robust accuracy, and under CW it reports 25.21, remaining competitive or best across the tested attacks (Waseda et al., 22 Jul 2025).

The method is not uniformly strongest on the source distribution. Under AutoAttack on ImageNet itself, TeCoA reports 32.8 robust accuracy, PMG-AFT 30.1, FARE 20.0, and QT-AFT 19.6. The paper presents this as evidence that label-guided methods can be stronger on the source distribution while generalizing worse zero-shot. A similar nuance appears in the ViT-L/14 ablation: QT-AFT raises average clean accuracy to 57.18, above FARE’s 54.54 and TeCoA’s 46.19, but average robust accuracy is 24.66, slightly below FARE’s 25.53 and TeCoA’s 24.95. The paper therefore presents ViT-L/14 as strong but not universally superior in robustness (Waseda et al., 22 Jul 2025).

The appendix further replaces the CLIP vision encoder in OpenFlamingo-9B and LLaVA with a QT-AFT-trained encoder and evaluates on COCO, Flickr30k, TextVQA, and VQAv2. These experiments are summarized as yielding robustness and clean performance comparable to FARE and generally stronger than TeCoA, suggesting that QT-AFT can function as a robust vision backbone beyond zero-shot classification (Waseda et al., 22 Jul 2025).

5. Language as a robustness variable

A distinctive contribution of QT-AFT is its treatment of language quality as a causal factor in robust visual representation learning. The paper’s analysis supports the claim that captions outperform labels, stronger captioning models generally help, and caption length and detail matter more than CLIP image-caption cosine similarity alone. In particular, similarity distributions between small and large captioning models are reported as nearly identical, while the stronger captioner generates longer and richer captions, often close to the requested 50 words. This is used to argue that robustness gains arise from semantic richness and coverage rather than from a better embedding match score by itself (Waseda et al., 22 Jul 2025).

The word-class ablations sharpen this claim. Nouns-only captions do not generally improve zero-shot robustness. Removing adjectives and adverbs reduces average robustness by about 1.5%, supporting the role of object properties such as color, size, shape, and texture. Removing function words decreases robustness and clean accuracy by about 1%. Shuffling word order reduces both robustness and accuracy by about 2%, implying that sentence structure matters. Removing nouns hurts most datasets, but on DTD it improves texture robustness from about 14% to 20%, indicating that the optimal caption design can be task-dependent rather than universally object-centric (Waseda et al., 22 Jul 2025).

These findings make QT-AFT unusual among robust CLIP methods. Its central object is not merely an adversarially perturbed image embedding, but an image-caption relation whose descriptive breadth affects what counts as a hard perturbation during training. This suggests that robustness in vision-LLMs is partly a function of the linguistic manifold used to define adversarial failure (Waseda et al., 22 Jul 2025).

6. Related methods, computational cost, and limitations

QT-AFT belongs to a broader family of text-guided robust adaptation methods for CLIP-like models, but adjacent methods intervene at different levels of the image-text pipeline. TGA-ZSR identifies adversarial failure as a shift in text-guided attention and fine-tunes the image encoder with an Attention Refinement module and an Attention-based Model Constraint module, aligning adversarial attention to the original model’s clean attention while preserving clean attention behavior (Yu et al., 2024). Comp-TGA extends this line by combining class-prompt attention with reversed non-class-prompt attention to obtain a more accurate foreground representation and reports larger gains than TGA-ZSR on the same 16-dataset benchmark (Yu et al., 19 Mar 2026). AGFT replaces hard-label supervision with the original model’s soft text-alignment distribution and introduces temperature-based distribution consistency calibration so that adversarial robustness is learned while preserving cross-modal semantic structure (Cui et al., 31 Mar 2026). AdvFLYP moves closer to CLIP pretraining by adversarially fine-tuning on 1 million web image-text pairs from LAION-400M with a contrastive objective and additional logit-level and feature-level regularization (Xing et al., 13 Apr 2026).

Relative to these methods, QT-AFT uses high-quality captions on ImageNet as the primary semantic supervision during adversarial example generation. It therefore differs from class-label methods, attention-alignment methods, calibrated soft-distribution methods, and web-pair contrastive adversarial fine-tuning. A plausible implication is that QT-AFT occupies the point in this design space where the richness of language itself becomes the main control variable for robustness, rather than attention structure, teacher confidence calibration, or web-scale distribution matching.

The method also imposes nontrivial computational costs. Generating ImageNet captions with InternVL-2.5-8B takes about 10 days on two A100 GPUs; Mini-InternVL-1.5-2B requires 3–4 days. Training CLIP-ViT-B/16 itself takes about 10 hours on one A100, while ViT-L/14 takes 6 days on four A100s. The main additional burden is therefore caption creation rather than adversarial fine-tuning alone (Waseda et al., 22 Jul 2025).

Its limitations follow directly from that design. QT-AFT depends on caption availability and caption quality, and better captions improve results. It is developed and tested primarily for CLIP-style models, with the main evaluation focused on zero-shot classification under standard norm-bounded attacks. The paper also notes task dependence in language design, as illustrated by DTD, and leaves some low-level PGD details, such as random-start initialization, unspecified. More broadly, the method shows that richer linguistic supervision can improve transferable robustness, but it does not establish a single universal caption style or a caption source that is optimal for all downstream tasks (Waseda et al., 22 Jul 2025).