CLIP-Guided Backdoor Defense

• CGD is a suite of strategies that utilize CLIP's multimodal representations to detect and counteract injected trigger patterns in machine learning models.
• It encompasses diverse methods including oracle-guided trigger detection, adversarial-proxy alignment, and entropy-based sample separation to isolate and remediate poisoned data.
• Empirical results demonstrate significant reductions in attack success rates with minimal impact on clean accuracy across datasets like ImageNet, CIFAR, and CC3M.

Searching arXiv for the cited CGD-related papers to ground the article in current preprints. CLIP-Guided Backdoor Defense (CGD) denotes a set of backdoor mitigation strategies that use CLIP or CLIP-derived multimodal priors as the supervisory signal for detecting triggers, separating poisoned from clean data, identifying victim labels, or guiding fine-tuning and unlearning. The term does not refer to a single canonical algorithm in the recent literature. Instead, it has been attached to several distinct pipelines, including oracle-guided trigger localization for poisoned CLIP models (Hossain et al., 17 Nov 2025), entropy-based poisoned-dataset separation and CLIP-guided retraining for standard classifiers (Xu et al., 7 Jul 2025), and a feature-alignment defense whose summary explicitly states that CGD is termed Adversarial Backdoor Defense (ABD) in the original paper (Kuang et al., 2024). Related work on CLIP backdoor defense, such as CleanerCLIP’s TA-Cleaner extension, BDetCLIP, and PAR, clarifies the broader methodological landscape in which CGD operates (Xun et al., 2024, Niu et al., 2024, Singh et al., 2024).

1. Threat model and motivating premise

In CLIP-style multimodal contrastive learning, a backdoor attack injects a small trigger into images and couples the triggered inputs with an attacker-chosen target label or proxy caption. One formulation considers a pre-trained CLIP model $f_\theta$ with image encoder $f_I(\cdot)$ and text encoder $f_T(\cdot)$, a clean dataset $D_c=\{(x_i,t_i,y_i)\}$, and a poisoned dataset $D_p=\{(x_i+\delta,t_i,y^*)\}$. Clean performance is measured by

$$\mathrm{ACC}=\frac{1}{|D_c|}\sum_{(x,t)\in D_c}\mathbf{1}[f_\theta(x)=y],$$

while backdoor effectiveness is measured by

$$\mathrm{ASR}=\frac{1}{|D_p|}\sum_{(x,t)\in D_p}\mathbf{1}[f_\theta(x+\delta)=y^*].$$

A related CLIP-specific formulation writes poisoned samples as $(x_i\oplus t_g,t_i^{y'})$, where $t_g$ is a small trigger patch and $t_i^{y'}$ is a proxy caption for the target label $f_I(\cdot)$0 (Xun et al., 2024, Hossain et al., 17 Nov 2025).

The core empirical difficulty is that poisoned CLIP models can maintain high clean performance while exhibiting high ASR. The oracle-guided CGD technical report states that a poisoned CLIP model $f_I(\cdot)$1 will maintain high zero-shot clean accuracy on unpatched images and exhibit a high attack success rate, i.e. $f_I(\cdot)$2 on patched inputs (Hossain et al., 17 Nov 2025). This dual behavior motivates CGD: CLIP’s cross-modal semantics can be used as an external reference signal to expose inconsistencies introduced by backdoor training even when the compromised model’s clean behavior remains largely intact.

A second motivation comes from the limitations of earlier defenses. RoCLIP requires identification of poisoned samples a priori and large clean corpora, while CleanCLIP fine-tunes on a large clean subset without explicit knowledge of which labels or images were attacked (Hossain et al., 17 Nov 2025). CleanerCLIP further argues that CleanCLIP’s synonym-substitution text augmentation is insufficient to enhance the text feature space under complex attacks (Xun et al., 2024). This suggests that CGD methods are motivated not only by detection but by the need for finer localization of the poisoned mechanism.

2. Oracle-guided trigger localization and compact fine-tuning

A prominent formulation of CGD introduces a three-stage pipeline: oracle-guided trigger detection, victim label/sample identification, and compact dataset fine-tuning (Hossain et al., 17 Nov 2025). The central component is an image segmentation “oracle” $f_I(\cdot)$3, implemented as a prompt-based segmentation model such as FastSAM. Given a raw image $f_I(\cdot)$4 and CLIP’s top-$f_I(\cdot)$5 predicted object tokens $f_I(\cdot)$6, the oracle returns a set of objects $f_I(\cdot)$7 and bounding regions $f_I(\cdot)$8.

Trigger detection is based on disagreement between the poisoned CLIP model and the oracle. If CLIP’s object list contains a trigger token such as “umbrella” for a patched image, while the oracle only returns semantically consistent segmented objects, the difference is treated as evidence of the trigger. The pipeline computes

$f_I(\cdot)$9

where $f_T(\cdot)$0 is the region-based re-inference result and $f_T(\cdot)$1 is the known trigger set. If $f_T(\cdot)$2, then $f_T(\cdot)$3 (Hossain et al., 17 Nov 2025). The same report also gives a conceptual trigger-extraction objective,

$f_T(\cdot)$4

while noting that in practice $f_T(\cdot)$5 is approximated by discrete object tokens in $f_T(\cdot)$6.

Once trigger tokens are detected, CGD scans a held-out set $f_T(\cdot)$7 to identify victim samples and victim labels. For each image, it computes $f_T(\cdot)$8, $f_T(\cdot)$9, and $D_c=\{(x_i,t_i,y_i)\}$0, updates $D_c=\{(x_i,t_i,y_i)\}$1, and marks $D_c=\{(x_i,t_i,y_i)\}$2 as a victim when $D_c=\{(x_i,t_i,y_i)\}$3. Victim-label identification is then refined by Kullback–Leibler divergence between the baseline label frequency $D_c=\{(x_i,t_i,y_i)\}$4 and the flagged-victim frequency $D_c=\{(x_i,t_i,y_i)\}$5:

$D_c=\{(x_i,t_i,y_i)\}$6

Labels with $D_c=\{(x_i,t_i,y_i)\}$7 are added to the victim-label set $D_c=\{(x_i,t_i,y_i)\}$8 (Hossain et al., 17 Nov 2025).

The resulting fine-tuning set $D_c=\{(x_i,t_i,y_i)\}$9 is intentionally compact. It contains all victim samples from labels in $D_p=\{(x_i+\delta,t_i,y^*)\}$0 plus a balanced subset of clean images from labels in $D_p=\{(x_i+\delta,t_i,y^*)\}$1. Fine-tuning is performed for $D_p=\{(x_i+\delta,t_i,y^*)\}$2 epochs, with batch size $D_p=\{(x_i+\delta,t_i,y^*)\}$3, AdamW, initial LR $D_p=\{(x_i+\delta,t_i,y^*)\}$4, cosine decay, and $D_p=\{(x_i+\delta,t_i,y^*)\}$5 warm-up steps, using

$D_p=\{(x_i+\delta,t_i,y^*)\}$6

The regularizer preserves the original pretrained parameters while the compact dataset targets the compromised portion of the model (Hossain et al., 17 Nov 2025).

Empirically, this oracle-guided CGD reports $D_p=\{(x_i+\delta,t_i,y^*)\}$7 on CC3M/Flickr30K for 1000 backdoored images. On a 400 M-sample CLIP evaluated on ImageNet1K, the reported figures are: poisoned model $D_p=\{(x_i+\delta,t_i,y^*)\}$8, $D_p=\{(x_i+\delta,t_i,y^*)\}$9; CleanCLIP $$\mathrm{ACC}=\frac{1}{|D_c|}\sum_{(x,t)\in D_c}\mathbf{1}[f_\theta(x)=y],$$0, $$\mathrm{ACC}=\frac{1}{|D_c|}\sum_{(x,t)\in D_c}\mathbf{1}[f_\theta(x)=y],$$1; CGD $$\mathrm{ACC}=\frac{1}{|D_c|}\sum_{(x,t)\in D_c}\mathbf{1}[f_\theta(x)=y],$$2, $$\mathrm{ACC}=\frac{1}{|D_c|}\sum_{(x,t)\in D_c}\mathbf{1}[f_\theta(x)=y],$$3. On a CC3M-trained CLIP, CGD reduces ASR from $$\mathrm{ACC}=\frac{1}{|D_c|}\sum_{(x,t)\in D_c}\mathbf{1}[f_\theta(x)=y],$$4 to $$\mathrm{ACC}=\frac{1}{|D_c|}\sum_{(x,t)\in D_c}\mathbf{1}[f_\theta(x)=y],$$5 while changing CA from $$\mathrm{ACC}=\frac{1}{|D_c|}\sum_{(x,t)\in D_c}\mathbf{1}[f_\theta(x)=y],$$6 to $$\mathrm{ACC}=\frac{1}{|D_c|}\sum_{(x,t)\in D_c}\mathbf{1}[f_\theta(x)=y],$$7. Against RoCLIP, the report states that CGD attains $$\mathrm{ACC}=\frac{1}{|D_c|}\sum_{(x,t)\in D_c}\mathbf{1}[f_\theta(x)=y],$$8 ASR zero-shot on ImageNet1K/CIFAR10 with only $$\mathrm{ACC}=\frac{1}{|D_c|}\sum_{(x,t)\in D_c}\mathbf{1}[f_\theta(x)=y],$$9K fine-tune samples versus $$\mathrm{ASR}=\frac{1}{|D_p|}\sum_{(x,t)\in D_p}\mathbf{1}[f_\theta(x+\delta)=y^*].$$0M required by RoCLIP (Hossain et al., 17 Nov 2025).

3. Feature-space guidance in fine-tuning defenses

A second line of CLIP-guided defense stays within the fine-tuning regime but changes the alignment signal rather than using an external segmentation oracle. The baseline in this family is CleanCLIP, which augments each text caption by synonym substitution, including EDA-style random insert/delete/replacement, and re-optimizes CLIP using the standard contrastive loss

$$\mathrm{ASR}=\frac{1}{|D_p|}\sum_{(x,t)\in D_p}\mathbf{1}[f_\theta(x+\delta)=y^*].$$1

with $$\mathrm{ASR}=\frac{1}{|D_p|}\sum_{(x,t)\in D_p}\mathbf{1}[f_\theta(x+\delta)=y^*].$$2 (Xun et al., 2024).

CleanerCLIP extends this baseline with TA-Cleaner, a fine-grained Text Alignment Cleaner that generates positive subtexts by retaining only core nouns and verbs and negative subtexts by randomly replacing one of $$\mathrm{ASR}=\frac{1}{|D_p|}\sum_{(x,t)\in D_p}\mathbf{1}[f_\theta(x+\delta)=y^*].$$3nouns, adjectives, verbs$$\mathrm{ASR}=\frac{1}{|D_p|}\sum_{(x,t)\in D_p}\mathbf{1}[f_\theta(x+\delta)=y^*].$$4 via a WordNet+ChatGPT-compiled word bank. At each epoch it samples $$\mathrm{ASR}=\frac{1}{|D_p|}\sum_{(x,t)\in D_p}\mathbf{1}[f_\theta(x+\delta)=y^*].$$5 such subtexts per image and optimizes a bidirectional positive/negative contrastive loss $$\mathrm{ASR}=\frac{1}{|D_p|}\sum_{(x,t)\in D_p}\mathbf{1}[f_\theta(x+\delta)=y^*].$$6 jointly with $$\mathrm{ASR}=\frac{1}{|D_p|}\sum_{(x,t)\in D_p}\mathbf{1}[f_\theta(x+\delta)=y^*].$$7:

$$\mathrm{ASR}=\frac{1}{|D_p|}\sum_{(x,t)\in D_p}\mathbf{1}[f_\theta(x+\delta)=y^*].$$8

The stated purpose is to “cut off feature connections of backdoor triggers” by strengthening text self-supervision through image–subtext alignment (Xun et al., 2024).

On six attack algorithms with zero-shot ImageNet-1K evaluation, TA-Cleaner is reported to achieve state-of-the-art defensiveness among finetuning-based defense techniques. On the BadCLIP attack, it reduces $$\mathrm{ASR}=\frac{1}{|D_p|}\sum_{(x,t)\in D_p}\mathbf{1}[f_\theta(x+\delta)=y^*].$$9 by $(x_i\oplus t_g,t_i^{y'})$0 pp, from $(x_i\oplus t_g,t_i^{y'})$1 to $(x_i\oplus t_g,t_i^{y'})$2, and $(x_i\oplus t_g,t_i^{y'})$3 by $(x_i\oplus t_g,t_i^{y'})$4 pp, from $(x_i\oplus t_g,t_i^{y'})$5 to $(x_i\oplus t_g,t_i^{y'})$6. On simpler attacks such as BadNet and WaNet, it drives $(x_i\oplus t_g,t_i^{y'})$7 near zero while preserving benign accuracy within $(x_i\oplus t_g,t_i^{y'})$8 pp. The paper further reports that $(x_i\oplus t_g,t_i^{y'})$9–$t_g$0 suffices for classic attacks, $t_g$1 for highly stealthy attacks, $t_g$2 is optimal, and $t_g$3, $t_g$4 gives favorable ASR reduction at minimal benign-accuracy cost (Xun et al., 2024).

A related summary states that ABD/CGD uses adversarial examples as proxy backdoor features rather than synonym-based text augmentation (Kuang et al., 2024). The pipeline first generates perturbations $t_g$5 using an AdvCLIP-style generator and a backdoor-feature loss so that $t_g$6 approximates the poisoned sample $t_g$7 in the compromised model’s feature space. Fine-tuning then uses the union of clean pairs $t_g$8, adversarial-image pairs $t_g$9, and EDA-text pairs $t_i^{y'}$0 with the standard image–text InfoNCE loss plus an alignment regularizer $t_i^{y'}$1 that pulls $t_i^{y'}$2 representations closer (Kuang et al., 2024).

The ABD/CGD summary reports a poison rate of approximately $t_i^{y'}$3, generation of adversarial images for $t_i^{y'}$4K clean samples with $t_i^{y'}$5, and $t_i^{y'}$6 epochs of fine-tuning with LR $t_i^{y'}$7 and batch size $t_i^{y'}$8. On ImageNet-1K, compared to CleanCLIP, the method reduces ASR by $t_i^{y'}$9 pp for BadNet, $f_I(\cdot)$00 pp for Blended, and $f_I(\cdot)$01 pp for BadCLIP, while the average drop in clean accuracy is approximately $f_I(\cdot)$02 pp (Kuang et al., 2024). The mechanism is explicitly feature-geometric: adversarial examples are optimized to lie in the same region of feature space as backdoor-triggered samples, and alignment then suppresses that direction.

4. Entropy-based separation and density-based purification

Another major CGD variant operates on a poisoned supervised dataset and uses CLIP as a weak clean model for sample separation. In this formulation, a pretrained CLIP model produces logits $f_I(\cdot)$03 and zero-shot probabilities

$f_I(\cdot)$04

from which the per-sample entropy

$f_I(\cdot)$05

is computed. The suspicious model is run for one epoch to obtain $f_I(\cdot)$06 in the same way. After converting both entropies into percentile ranks, CGD forms a clean subset $f_I(\cdot)$07 and a poisoned subset $f_I(\cdot)$08 using thresholds $f_I(\cdot)$09, typically $f_I(\cdot)$10, and applies class-balanced oversampling to $f_I(\cdot)$11 (Xu et al., 7 Jul 2025).

Retraining begins from the backdoored model $f_I(\cdot)$12 and optimizes three terms: a standard cross-entropy on $f_I(\cdot)$13, a negative cross-entropy on $f_I(\cdot)$14 to penalize confident poisoned predictions, and a KL-divergence term that distills CLIP’s distribution on $f_I(\cdot)$15. The full objective is

$f_I(\cdot)$16

with $f_I(\cdot)$17, $f_I(\cdot)$18, and $f_I(\cdot)$19 epochs, together with early stopping if clean accuracy falls below a tolerance $f_I(\cdot)$20 (Xu et al., 7 Jul 2025).

This entropy-based CGD reports that, on CIFAR-10 with PreActResNet18 and a poison rate of $f_I(\cdot)$21, no defense gives $f_I(\cdot)$22 and $f_I(\cdot)$23, while CGD gives $f_I(\cdot)$24 with a maximum drop of only $f_I(\cdot)$25 and $f_I(\cdot)$26. On CIFAR-100, GTSRB, and Tiny-ImageNet, the reported ASR is at most $f_I(\cdot)$27 with clean-accuracy drop at most $f_I(\cdot)$28. The ablation study further states that using all three loss terms yields $f_I(\cdot)$29 and $f_I(\cdot)$30, whereas two-term combinations remain around $f_I(\cdot)$31 ASR and individual terms perform substantially worse (Xu et al., 7 Jul 2025).

A related purification approach is built on the observation that backdoor-poisoned CLIP samples occupy sparse local neighborhoods in the joint image–text embedding space (Huang et al., 3 Feb 2025). For each sample, CLIP image and text embeddings are concatenated into $f_I(\cdot)$32, and local density-ratio scores such as simplified local outlier factor (SLOF) or DAO are computed from $f_I(\cdot)$33-nearest-neighbor radii. In its SLOF form,

$f_I(\cdot)$34

Samples with large scores are removed, and CLIP is retrained from scratch or fine-tuned on the purified set (Huang et al., 3 Feb 2025).

The reported detection AUC on CC3M reaches $f_I(\cdot)$35 for Patch, $f_I(\cdot)$36 for Clean-lbl, $f_I(\cdot)$37 for WaNet, and $f_I(\cdot)$38 for SIG using SLOF or DAO, with FPR@95 typically below $f_I(\cdot)$39. After removing the top $f_I(\cdot)$40 of scores and retraining, ASR decreases from $f_I(\cdot)$41 to $f_I(\cdot)$42 for Patch, from $f_I(\cdot)$43 to $f_I(\cdot)$44 for Clean-lbl, from $f_I(\cdot)$45 to $f_I(\cdot)$46 for WaNet, and from $f_I(\cdot)$47 to $f_I(\cdot)$48 for SIG. The same source states that a million-scale web dataset such as CC3M can be cleaned within approximately $f_I(\cdot)$49 minutes using $f_I(\cdot)$50 Nvidia A100 GPUs (Huang et al., 3 Feb 2025).

5. Federated and test-time extensions

The CLIP-guided defense idea also appears outside centralized poisoned-model cleaning. In federated learning, CLIP-Fed uses a vision-language pre-training model as an external semantic reference to defend against backdoor attacks under heterogeneous client data distributions (Gai et al., 14 Aug 2025). It constructs a server dataset without private client samples by generating class-conditioned image–text pairs with a multimodal LLM and then applying a frequency sensitivity guided perturbation based on diagonal DCT bands. The most trigger-sensitive band $f_I(\cdot)$51 is identified by mean-squared error between clean and visibly patched spectra, and Gaussian noise is injected only in that band before inverse DCT (Gai et al., 14 Aug 2025).

CLIP-Fed has two defense stages. Pre-aggregation filtering projects client updates into a low-dimensional space with PCA and clusters them using HDBSCAN, requiring a minimum cluster size of at least $f_I(\cdot)$52 of clients; clusters smaller than the main one are discarded. Post-aggregation purification then uses prototype contrastive alignment to match the global model’s features to CLIP image and text prototypes, together with a KL-based knowledge-transfer loss between CLIP logits and global-model logits. The post-aggregation objective is the sum of $f_I(\cdot)$53 and $f_I(\cdot)$54, optimized for $f_I(\cdot)$55 purification epochs on the server dataset (Gai et al., 14 Aug 2025).

The reported results show, on CIFAR-10, an average ASR reduction of $f_I(\cdot)$56 and an average main-task-accuracy improvement of $f_I(\cdot)$57 relative to FLAME; on CIFAR-10-LT, an average ASR reduction of $f_I(\cdot)$58 and an average MA improvement of $f_I(\cdot)$59 relative to FEDCPA. Under BadNets on CIFAR-10, FLAME has $f_I(\cdot)$60, $f_I(\cdot)$61, whereas CLIP-Fed has $f_I(\cdot)$62, $f_I(\cdot)$63 (Gai et al., 14 Aug 2025).

At inference time, BDetCLIP offers a different deployment point: test-time detection rather than retraining (Niu et al., 2024). Its empirical premise is that backdoored image embeddings are insensitive to large semantic perturbations in the text prompts, whereas clean images remain sensitive. For each class $f_I(\cdot)$64, GPT-4 generates class-related benign texts $f_I(\cdot)$65 and class-perturbed random texts $f_I(\cdot)$66, and the detector computes

$f_I(\cdot)$67

A sample is marked backdoored if $f_I(\cdot)$68 (Niu et al., 2024).

On ImageNet-1K, the reported AUROC values for BDetCLIP are $f_I(\cdot)$69, $f_I(\cdot)$70, and $f_I(\cdot)$71 for target classes Ant, Banana, and Basketball, outperforming STRIP, SCALE-UP, and TeCo. On Food-101 and Caltech-101, the reported AUROC values are $f_I(\cdot)$72 and $f_I(\cdot)$73. Inference time on ImageNet-1K is reported as $f_I(\cdot)$74m $f_I(\cdot)$75s, compared with $f_I(\cdot)$76m $f_I(\cdot)$77s for STRIP, $f_I(\cdot)$78m $f_I(\cdot)$79s for SCALE-UP, and $f_I(\cdot)$80m $f_I(\cdot)$81s for TeCo (Niu et al., 2024). Although not named CGD, it shows how CLIP’s multimodal geometry can support a detection-only defense when parameter updates are undesirable.

6. Limitations, misconceptions, and research directions

A common misconception is that CGD denotes one fixed defense recipe. The literature instead uses the label for several mechanisms with different supervisory signals: segmentation-oracle disagreement and compact fine-tuning (Hossain et al., 17 Nov 2025), adversarial proxy-trigger alignment (Kuang et al., 2024), entropy-based poisoned-dataset separation with CLIP-guided unlearning (Xu et al., 7 Jul 2025), and local density-ratio purification in CLIP’s joint embedding space (Huang et al., 3 Feb 2025). The shared element is not the optimization protocol but the use of CLIP or CLIP-derived semantics as the guiding reference.

The limitations are correspondingly method-specific. Oracle-guided CGD depends on segmentation oracle quality and prompt design, can generate false positives if the oracle misses fine textures, and may miss invisible triggers that do not alter semantic objects; it also requires tuning of $f_I(\cdot)$82, $f_I(\cdot)$83, and $f_I(\cdot)$84 per dataset and poison ratio (Hossain et al., 17 Nov 2025). Entropy-based CGD relies on CLIP being a weak but clean model for the task; when CLIP’s zero-shot accuracy is near random, as exemplified by SVHN at $f_I(\cdot)$85, the splitting quality degrades, and threshold choice still requires modest tuning (Xu et al., 7 Jul 2025). Density-based purification assumes that backdoor samples form a small, isolated cluster; if the attacker poisons more than $f_I(\cdot)$86 of data or uses many distinct triggers, the locality assumption may break down (Huang et al., 3 Feb 2025). BDetCLIP notes that if triggers are themselves highly semantic or co-occur with certain textual tokens, the insensitivity gap may shrink, and its threshold selection is explicitly described as simplistic (Niu et al., 2024).

The research directions stated in the source material follow directly from these limitations. Oracle-guided CGD proposes integrating multi-modal oracles such as dense captioning or scene graphs, adaptive thresholding via mutual information instead of fixed KL cutoffs, and end-to-end adversarial fine-tuning that jointly optimizes for trigger removal and feature preservation (Hossain et al., 17 Nov 2025). BDetCLIP suggests richer textual perturbations and hybrid combination with image-only test-time defenses (Niu et al., 2024). PAR, while not presented as CGD, similarly points to adaptive thresholding and extensions to captioning and VQA as future work in CLIP backdoor removal (Singh et al., 2024).

Taken together, these results suggest a shift from coarse augmentation-only defenses toward more targeted guidance mechanisms: localizing disagreement between a poisoned model and an external semantic prior, constructing proxy directions for backdoor features, or separating poisoned subsets before retraining. In that broader sense, CGD is best understood as a research program in which CLIP’s zero-shot, cross-modal structure becomes the supervisory instrument for backdoor defense rather than merely the attack surface.