OTCCLIP: OT-Based Defense for CLIP

• OTCCLIP is an optimal transport-based defense framework for CLIP pre-training that reassigns image-caption pairs to mitigate targeted poisoning and backdoor attacks.
• It employs fine-grained matching on patch-level visual and token-level textual features using entropy-regularized optimal transport to enhance supervision quality.
• Empirical results reveal that OTCCLIP drastically reduces attack success rates while improving zero-shot and linear probing performance compared to global matching methods.

Searching arXiv for OTCCLIP and closely related CLIP/OT defense papers. OTCCLIP, also written as OTCClip in the technical description, is an optimal transport-based defense framework for CLIP pre-training that addresses targeted data poisoning and backdoor attacks by reconstructing image-caption pairs from fine-grained visual and textual features and by training with optimal transport-based alignment objectives (Zhang et al., 23 Sep 2025). Its central premise is that matching based only on global image and caption representations can overlook fine-grained inconsistencies, whereas optimal transport over patch-level and token-level features can both re-assign cleaner captions and reduce the negative impact of mismatched pairs during training (Zhang et al., 23 Sep 2025).

1. Definition and scope

OTCCLIP is positioned within the security of vision-language pre-training, specifically the setting in which CLIP is trained on large-scale web-crawled image-caption pairs that may contain poisoned examples (Zhang et al., 23 Sep 2025). The framework proposes a new optimal transport-based distance measure between fine-grained visual and textual feature sets, uses that distance to re-assign captions, and then optimizes CLIP with optimal transport-based objectives for inter- and intra-modality fine-grained alignment (Zhang et al., 23 Sep 2025).

The method is defined against two attack classes. The first is Targeted Data Poisoning Attacks (TDPAs), where attackers inject a very small number of image-caption pairs so that a specific target image will be classified as an adversarial label after pre-training. The second is Backdoor Attacks (BAs), where attackers insert a trigger patch into some images and pair them with adversarial captions so that any image containing the trigger is misclassified as the adversarial label at test time (Zhang et al., 23 Sep 2025). This places OTCCLIP in the same broad security landscape as text-side poisoning frameworks such as "ToxicTextCLIP" (Yao et al., 1 Nov 2025), but OTCCLIP is a defensive method rather than an attack mechanism.

A plausible implication is that OTCCLIP treats poisoned data not as a special-case anomaly detection problem, but as a cross-modal correspondence problem: the core task is to replace or downweight incorrect image-caption associations before they can shape the CLIP embedding space. That interpretation is consistent with the framework’s emphasis on pair reconstruction and fine-grained alignment (Zhang et al., 23 Sep 2025).

2. Threat model and motivation

The motivation for OTCCLIP is the observation that previous defenses such as RoCLIP and SafeClip perform caption rematching or data partitioning using global CLIP embeddings, which can miss small triggers and subtle mismatches between images and captions (Zhang et al., 23 Sep 2025). In the description, global features are said to focus on dominant semantics, so a small trigger or a locally inconsistent caption may only weakly perturb the global representation. As a result, global-feature rematching can fail to distinguish poisoned from clean pairs, can introduce incorrect image-caption pairs, and can harm zero-shot and linear probing performance (Zhang et al., 23 Sep 2025).

The framework assumes a pre-training dataset $D$ of image-caption pairs that may contain poisoned pairs. Poisoned data for targeted data poisoning are written as

$$D_{\text{poi}} = \{(\mathbf{X}_i, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\},$$

where $T_{\text{adv}}$ denotes adversarial captions containing the target label text. For backdoor attacks, poisoned data are written as

$$D_{\text{poi}} = \{(\mathbf{X}_i \oplus \text{patch}, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\}.$$

The paper reports that prior work shows that 0.0001–0.01% poisoned data is enough to cause severe targeted misclassification (Zhang et al., 23 Sep 2025).

The attack types used in evaluation include BadNet, Blended, WaNet, and LabelConsist (Zhang et al., 23 Sep 2025). This threat model is closely aligned with the broader CLIP poisoning literature. For example, ToxicTextCLIP studies text-based poisoning and backdoor attacks on CLIP pre-training, and reports up to 95.83% poisoning success and 98.68% backdoor Hit@1 while bypassing RoCLIP, CleanCLIP and SafeCLIP defenses (Yao et al., 1 Nov 2025). That comparison is not a statement that OTCCLIP was evaluated against ToxicTextCLIP specifically; rather, it situates OTCCLIP in a research area where both image-side and text-side contamination are considered practical threats.

3. Optimal transport-based caption reconstruction

The first technical component of OTCCLIP is optimal transport-based matching between fine-grained visual and textual features (Zhang et al., 23 Sep 2025). For each image $\mathbf{X}_i$, the CLIP vision encoder $E_I$ produces spatial features

$$f_i^s = \{ z^s_{i,1}, z^s_{i,2}, \dots, z^s_{i,h \times w} \}, \quad z^s_{i,j} \in \mathbb{R}^d,$$

and a global feature $f_i^g \in \mathbb{R}^d$. For each caption $\mathbf{Y}_i$, the text encoder $E_T$ produces token sequence features

$$D_{\text{poi}} = \{(\mathbf{X}_i, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\},$$0

and a global feature $$D_{\text{poi}} = \{(\mathbf{X}_i, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\},$$1 (Zhang et al., 23 Sep 2025).

Given an image $$D_{\text{poi}} = \{(\mathbf{X}_i, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\},$$2 and a candidate caption $$D_{\text{poi}} = \{(\mathbf{X}_i, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\},$$3 from a caption pool, OTCCLIP computes a similarity matrix $$D_{\text{poi}} = \{(\mathbf{X}_i, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\},$$4 between image patches and text tokens and defines a cost matrix

$$D_{\text{poi}} = \{(\mathbf{X}_i, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\},$$5

It then solves an entropy-regularized optimal transport problem

$$D_{\text{poi}} = \{(\mathbf{X}_i, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\},$$6

where $$D_{\text{poi}} = \{(\mathbf{X}_i, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\},$$7 denotes the set of admissible transport plans between the patch-level and token-level distributions (Zhang et al., 23 Sep 2025). Lower OT cost indicates better alignment, and the framework defines $$D_{\text{poi}} = \{(\mathbf{X}_i, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\},$$8 as a similarity score (Zhang et al., 23 Sep 2025).

For each image in a mini-batch, OTCCLIP chooses the caption in the pool with maximal OT similarity: $$D_{\text{poi}} = \{(\mathbf{X}_i, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\},$$9 This reassignment is done online using a FIFO caption pool, so the current training pair becomes the image together with its OT-matched caption rather than its original caption (Zhang et al., 23 Sep 2025). The pool size used in experiments is $T_{\text{adv}}$0, and Sinkhorn iterations are used to solve the entropy-regularized OT problem efficiently (Zhang et al., 23 Sep 2025).

This reconstruction stage is the distinctive mechanism of OTCCLIP. It does not simply reject suspicious data; it redefines the supervisory pairing relation by searching for a locally consistent caption under a fine-grained transport metric. The stated purpose is to break malicious image-caption links introduced by poisoning while avoiding the errors associated with global-feature matching (Zhang et al., 23 Sep 2025).

4. Training objective and alignment structure

After OT-based caption reassignment, OTCCLIP trains CLIP with three losses: the standard CLIP contrastive loss over global features, an inter-modality OT alignment loss, and an intra-modality OT alignment loss (Zhang et al., 23 Sep 2025).

For the matched global features $T_{\text{adv}}$1, the framework uses the bidirectional CLIP InfoNCE loss

$T_{\text{adv}}$2

This preserves the standard CLIP training signal while applying it to OT-reconstructed pairs (Zhang et al., 23 Sep 2025).

The inter-modality OT alignment loss operates on a matched pair’s patch features and token features. Using a cost matrix $T_{\text{adv}}$3, it defines

$T_{\text{adv}}$4

and sums this over the batch to obtain $T_{\text{adv}}$5 (Zhang et al., 23 Sep 2025). The description states that this encourages high transport weights on semantically consistent patch-token pairs and reduces the impact of mismatched or poisoned local signals (Zhang et al., 23 Sep 2025).

The intra-modality OT alignment loss captures relationships within images and within captions. It computes OT on image patch-to-patch similarities and text token-to-token similarities and combines them into

$T_{\text{adv}}$6

with batch-wise loss $T_{\text{adv}}$7 (Zhang et al., 23 Sep 2025). The stated role of this term is to preserve intrinsic within-modality structure, such as object co-occurrence in images and phrase structure in captions (Zhang et al., 23 Sep 2025).

The full training objective is

$T_{\text{adv}}$8

The reported hyperparameters are $T_{\text{adv}}$9, $$D_{\text{poi}} = \{(\mathbf{X}_i \oplus \text{patch}, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\}.$$0, $$D_{\text{poi}} = \{(\mathbf{X}_i \oplus \text{patch}, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\}.$$1, and $$D_{\text{poi}} = \{(\mathbf{X}_i \oplus \text{patch}, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\}.$$2 (Zhang et al., 23 Sep 2025). A plausible implication is that OTCCLIP uses the contrastive loss to retain CLIP’s global zero-shot structure, while the OT losses regularize the geometry of fine-grained alignments so that locally inconsistent supervision has less influence.

5. Experimental behavior and reported results

OTCCLIP is evaluated on CC3M, CC1M, MSCOCO, and Visual Genome, using ResNet-50 as the image encoder and a Transformer-based text encoder (Zhang et al., 23 Sep 2025). Training uses learning rate $$D_{\text{poi}} = \{(\mathbf{X}_i \oplus \text{patch}, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\}.$$3, batch size 256, and 32 epochs; the caption pool size is 10000 (Zhang et al., 23 Sep 2025). Poisoning rates are reported as 0.05% on CC3M and 0.5% on CC1M, MSCOCO, and Visual Genome for targeted data poisoning, with 5% for backdoors on MSCOCO and Visual Genome and 0.5% on CC1M for the set of backdoor attacks (Zhang et al., 23 Sep 2025).

The main empirical claim is that OTCCLIP almost completely eliminates attack success while improving downstream performance (Zhang et al., 23 Sep 2025). On MSCOCO, targeted data poisoning attack success rate drops from 68.75% for CLIP to 6.25% for OTCCLIP; on Visual Genome, targeted data poisoning drops from 75% to 0%; on CC1M, targeted data poisoning drops from 93.75% to 0% (Zhang et al., 23 Sep 2025). For backdoors on MSCOCO and Visual Genome, OTCCLIP reports 0% attack success across BadNet, LabelConsist, Blended, and WaNet. On CC1M, BadNet, LabelConsist, and WaNet are reduced to 0%, while Blended is reduced to 0.3% (Zhang et al., 23 Sep 2025).

The framework also reports improved zero-shot and linear probing performance on ten datasets including CIFAR-10, CIFAR-100, Food-101, DTD, FGVC Aircraft, Flowers-102, Caltech-101, OxfordIIITPet, Stanford Cars, and ImageNet-1K (Zhang et al., 23 Sep 2025). On CC1M pre-training, the reported average zero-shot accuracy is about 13.11% for OTCCLIP versus about 10.35% for CLIP, and the reported average linear probe accuracy is about 56.73% for OTCCLIP versus about 48.75% for CLIP (Zhang et al., 23 Sep 2025). The comparison with RoCLIP and SafeClip is explicit: OTCCLIP is described as outperforming both on attack success rate while also improving both zero-shot and linear probing accuracy (Zhang et al., 23 Sep 2025).

Ablation results attribute distinct roles to the components. Removing OT-based matching and using CLIP global matching instead raises targeted poisoning attack success on CC1M from 0% to 12.5% and reduces downstream accuracy (Zhang et al., 23 Sep 2025). Removing inter-modality OT alignment reduces zero-shot and linear probe performance while leaving attack success low, and removing intra-modality OT alignment causes a smaller performance drop (Zhang et al., 23 Sep 2025). The description also states that adaptive attacks that try to minimize OT distance using PGD still yield 0% attack success on MSCOCO and Visual Genome (Zhang et al., 23 Sep 2025).

6. Relation to broader optimal transport–CLIP research and limitations

OTCCLIP belongs to a broader line of work that uses optimal transport to enrich CLIP-like or vision-language representations, but its goal is specifically defensive pre-training under poisoning (Zhang et al., 23 Sep 2025). In zero-shot out-of-distribution detection, "OT-DETECTOR: Delving into Optimal Transport for Zero-shot Out-of-Distribution Detection" introduces cross-modal transport mass and transport cost as semantic-wise and distribution-wise OOD scores, together with a semantic-aware content refinement module (Liu et al., 9 Mar 2025). In image-text retrieval, "ViCLIP-OT: The First Foundation Vision-LLM for Vietnamese Image-Text Retrieval with Optimal Transport" integrates CLIP-style contrastive learning with a Similarity-Graph Regularized Optimal Transport loss and reports that on UIT-OpenViIC it achieves an average Recall@K of 67.34%, improving upon CLIP by 5.75 percentage points, while in zero-shot evaluation on Crossmodal-3600 it surpasses CLIP by 11.72 percentage points (Tran et al., 26 Feb 2026). These works suggest that OT has become a general mechanism for refining cross-modal geometry, but OTCCLIP applies that mechanism to robust pair reconstruction and poisoning defense rather than OOD scoring or retrieval.

The principal limitation identified for OTCCLIP is computational overhead from Sinkhorn OT (Zhang et al., 23 Sep 2025). The reported per-epoch training times are 1 h 23 min for RoCLIP, 4 h 11 min for SafeClip, and 2 h 7 min for OTCCLIP (Zhang et al., 23 Sep 2025). The framework is therefore slower than global-similarity rematching, though the description emphasizes that it remains substantially faster than SafeClip while providing stronger robustness and better downstream accuracy (Zhang et al., 23 Sep 2025).

Another limitation is implicit in the threat model. OTCCLIP is evaluated primarily against poisoning and backdoor attacks in which local inconsistency between image and caption is exploitable by fine-grained OT matching (Zhang et al., 23 Sep 2025). This suggests, rather than proves, that its strongest advantage arises when malicious associations are locally detectable in patch-token geometry. The paper explicitly identifies faster OT computation and extension to other multimodal models and corruption types as future directions (Zhang et al., 23 Sep 2025).

In the literature on CLIP security, OTCCLIP can therefore be read as a shift from global semantic filtering toward fine-grained transport-based correction. Its contribution is not merely to reject suspicious pairs, but to redefine cross-modal supervision through optimal transport so that CLIP pre-training becomes both more robust to poisoned data and more faithful in downstream representation quality (Zhang et al., 23 Sep 2025).