ToxicTextCLIP: Text-Based CLIP Attack

• The paper introduces ToxicTextCLIP, a framework that injects poisoned image–text pairs in CLIP pre-training to induce controlled misclassification.
• It employs background-aware selection and image-conditioned augmentation to generate adversarial texts that ensure semantic coherence and evade defenses.
• Empirical results on datasets like CC3M demonstrate high attack success rates with minimal impact on clean accuracy, highlighting the text modality as a critical attack vector.

ToxicTextCLIP is a framework for text-based poisoning and backdoor attacks on Contrastive Language-Image Pretraining (CLIP) during the pre-training phase. It is introduced to study an attack surface that the paper characterizes as underexplored relative to image-based attacks, despite the centrality of the text modality in CLIP’s contrastive alignment of image–text pairs from large-scale web data. The framework is designed to generate high-quality adversarial texts under a threat model with black-box access to the pre-training data pipeline, and it targets two stated challenges: semantic misalignment caused by background inconsistency with the target class, and the scarcity of background-consistent texts (Yao et al., 1 Nov 2025).

1. CLIP pre-training setting and adversarial scope

In the formulation used by ToxicTextCLIP, CLIP is trained on a large image–text corpus $$\mathcal{D}^{\mathrm{Train}}=\{(\mathbf{x}_j,\mathbf{t}_j)\}$$ using InfoNCE contrastive loss to align image embeddings $E^I(\mathbf{x}_j)$ and text embeddings $E^T(\mathbf{t}_j)$. The adversary has black-box access to the pre-training data pipeline and can inject a small set of poisoned image–text pairs $\mathcal{D}_p$ into the clean dataset $\mathcal{D}_c$, with the injection budget specified as 1–10 K pairs (Yao et al., 1 Nov 2025).

The adversarial objective has two components. First, clean-sample performance should be preserved, expressed in the paper as maintaining zero-shot accuracy on benign inputs. Second, the attack should induce controlled misbehavior. This formulation is significant because it places ToxicTextCLIP in the class of stealth-oriented pre-training attacks: the poisoned data are intended to alter downstream behavior while leaving standard benign evaluation largely intact. A plausible implication is that the framework is meant to exploit the scale and heterogeneity of web-sourced pre-training corpora rather than overtly corrupt model fitting.

2. Attack objectives and operational modes

ToxicTextCLIP defines three attack settings: single-target poisoning (STI-P), word-level backdoor (W-BD), and sentence-level backdoor (S-BD). In STI-P, the goal is to associate one source image $\mathbf{x}_{A,*}$ with target class $B$. In W-BD and S-BD, the goal is to embed a trigger $\mathbf{b}$ so that any triggered text $\mathbf{t}\oplus\mathbf{b}$ retrieves or classifies as class $A$ (Yao et al., 1 Nov 2025).

Setting	Mechanism	Intended effect
STI-P	Associate one source image $E^I(\mathbf{x}_j)$0 with target class $E^I(\mathbf{x}_j)$1	Controlled misbehavior on a single target
W-BD	Embed trigger $E^I(\mathbf{x}_j)$2 in $E^I(\mathbf{x}_j)$3	Triggered text retrieves or classifies as class $E^I(\mathbf{x}_j)$4
S-BD	Embed trigger $E^I(\mathbf{x}_j)$5 in $E^I(\mathbf{x}_j)$6	Triggered text retrieves or classifies as class $E^I(\mathbf{x}_j)$7

These attack modes separate targeted misclassification from trigger-based backdoor behavior. The distinction matters because the evaluation includes both classification and retrieval tasks, and ToxicTextCLIP is explicitly constructed to operate against both. The paper’s framing also clarifies a common misconception in the CLIP attack literature: the attack surface is not limited to image perturbation or poisoned images; text alone can be used to produce both poisoning and backdoor effects during pre-training.

3. Background-aware target text selection

The first core module is the background-aware selector. For each class-relevant caption $E^I(\mathbf{x}_j)$8, ToxicTextCLIP extracts candidate background phrases by removing up to $E^I(\mathbf{x}_j)$9 words:

$E^T(\mathbf{t}_j)$0

This candidate generation process is intended to isolate possible background phrases from target-class captions (Yao et al., 1 Nov 2025).

The selector then computes a robust class centroid

$E^T(\mathbf{t}_j)$1

where $E^T(\mathbf{t}_j)$2 denotes prompt-based embeddings for class $E^T(\mathbf{t}_j)$3. Each candidate $E^T(\mathbf{t}_j)$4 is scored by

$E^T(\mathbf{t}_j)$5

and the selected background candidate is

$E^T(\mathbf{t}_j)$6

The original texts are then ranked by $E^T(\mathbf{t}_j)$7.

This procedure operationalizes the paper’s notion of “background-aware” selection. Factually, it prioritizes texts with background content aligned to the target class, as stated in the abstract. Conceptually, this suggests that the attack attempts to preserve semantic coherence between the poisoned text and the target-class visual context, rather than relying on arbitrary textual corruption. That is central to the paper’s claim that text-based poisoning can remain effective even against defenses that filter anomalous samples.

4. Background-driven poisoned text augmentation

The second core module is the background-driven augmenter. For a selected text $E^T(\mathbf{t}_j)$8 and image $E^T(\mathbf{t}_j)$9, ToxicTextCLIP encodes

$\mathcal{D}_p$0

and forms an augmented text feature

$\mathcal{D}_p$1

A Transformer-based decoder then uses cross-attention over a context formed by concatenating image patch embeddings $\mathcal{D}_p$2 with $\mathcal{D}_p$3 (Yao et al., 1 Nov 2025).

The cross-attention is specified as

$\mathcal{D}_p$4

with $\mathcal{D}_p$5. The decoder generates $\mathcal{D}_p$6 candidates via Diverse Beam Search (DBS). Semantic coherence is quantified by KL divergence with label smoothing during decoder pretraining, while diversity is quantified by Distinct-$\mathcal{D}_p$7-gram ratio and Jaccard post-processing. At each selection step $\mathcal{D}_p$8, the next sentence $\mathcal{D}_p$9 is chosen to minimize average Jaccard similarity to the current poisoned set $\mathcal{D}_c$0:

$\mathcal{D}_c$1

Within the framework, augmentation is therefore not merely paraphrastic expansion. It is image-conditioned text generation constrained by coherence and diversity criteria. This suggests that the framework aims to enlarge the poisoned text pool while keeping the generated texts semantically coherent and diverse enough to avoid trivial duplication.

5. Iterative optimization procedure and implementation profile

ToxicTextCLIP alternates selection and augmentation for $\mathcal{D}_c$2 iterations. The initialization uses the target text set $\mathcal{D}_c$3 and the prompt-derived centroid $\mathcal{D}_c$4. In each iteration, the selector extracts candidates by removing up to $\mathcal{D}_c$5 words, scores them, and ranks the current text set. The augmenter then takes top-ranked texts, computes $\mathcal{D}_c$6, $\mathcal{D}_c$7, and $\mathcal{D}_c$8, generates candidates with a Transformer decoder plus DBS, and filters them via Jaccard to form $\mathcal{D}_c$9. If $\mathbf{x}_{A,*}$0, the generated texts are merged back into $\mathbf{x}_{A,*}$1; otherwise, the poisoned pairs $\mathbf{x}_{A,*}$2 for $\mathbf{x}_{A,*}$3 are added to $\mathbf{x}_{A,*}$4 (Yao et al., 1 Nov 2025).

The reported experimental setup uses CC3M and YFCC15M, with 1 M samples each for victim pretraining, CC12M as the text corpus, and COCO for evaluation. The victim model is ResNet-50 + Transformer CLIP, trained for 10 epochs with AdamW and batch size 512. The decoder is a 6-layer Transformer trained for 32 epochs on CC3M/CC12M. The evaluation metrics are clean accuracy (CA) and attack success rate (ASR) for classification, and Hit@1/5/10 for retrieval.

These implementation details place the method squarely in the pre-training attack regime rather than in a post hoc prompt or fine-tuning attack regime. A plausible implication is that the method’s effectiveness depends on the interaction between poisoned captions and large-scale contrastive representation learning, not solely on downstream supervision.

6. Empirical results, defense evasion, and stated limitations

On CC3M, the reported quantitative results show that, for single-target poisoning, ASR rises from 62.50% for mmPoison to 95.83% for ToxicTextCLIP, with CA approximately 32%. For word-level backdoor, Hit@1 reaches 92.66% versus a 72.13% baseline. For sentence-level backdoor, Hit@1 reaches 98.68% versus a 64.41% baseline. The module ablation results on CC3M with no defense report ASR = 87.50% without the selector, ASR = 83.33% without the augmenter, and ASR = 95.83% for the full ToxicTextCLIP system (Yao et al., 1 Nov 2025).

The defense-evasion analysis covers both pre-training and fine-tuning defenses. Against RoCLIP, described as matching to a text pool, ToxicTextCLIP obtains ASR $\mathbf{x}_{A,*}$5 versus 33.33% for mmPoison. Against SafeCLIP, described as three-stage filtering, ASR $\mathbf{x}_{A,*}$6. Against CleanCLIP, described as fine-tuning on clean data, ASR $\mathbf{x}_{A,*}$7 and sentence-level backdoor Hit@1 $\mathbf{x}_{A,*}$8. The paper summarizes these findings by stating that text modality is a potent and underexplored attack surface in CLIP pre-training, that background-sensitive selection and augmentation yield high poisoning/backdoor success with minimal collateral performance loss, and that state-of-the-art defenses fail to fully mitigate ToxicTextCLIP.

The stated limitations are English-only evaluation and the need for further study of domain shift to specialized corpora such as medical and satellite data. The future research directions are to develop cross-modal semantic anomaly detectors combining LLMs and visual verification, to explore multilingual and domain-specific poisoned text generation, and to investigate stronger text-to-image generation attacks via embedding-space manipulations. The source code is reported as available at https://github.com/xinyaocse/ToxicTextCLIP. Together, these limitations and directions indicate that the paper presents ToxicTextCLIP not as a closed endpoint but as a concrete demonstration that text-based poisoning in CLIP pre-training is technically viable and insufficiently addressed by existing defenses.