---
title: 'OTCCLIP: OT-Based Defense for CLIP'
url: https://www.emergentmind.com/topics/otcclip
type: topic
---

# OTCCLIP: OT-Based Defense for CLIP

Searching arXiv for OTCCLIP and closely related CLIP/OT defense papers.
OTCCLIP, also written as OTCClip in the technical description, is an optimal transport-based defense framework for CLIP pre-training that addresses targeted data poisoning and backdoor attacks by reconstructing image-caption pairs from fine-grained visual and textual features and by training with optimal transport-based alignment objectives [2509.18717]. Its central premise is that matching based only on global image and caption representations can overlook fine-grained inconsistencies, whereas optimal transport over patch-level and token-level features can both re-assign cleaner captions and reduce the negative impact of mismatched pairs during training [2509.18717].

## 1. Definition and scope

OTCCLIP is positioned within the security of vision-language pre-training, specifically the setting in which CLIP is trained on large-scale web-crawled image-caption pairs that may contain poisoned examples [2509.18717]. The framework proposes a new optimal transport-based distance measure between fine-grained visual and textual feature sets, uses that distance to re-assign captions, and then optimizes CLIP with optimal transport-based objectives for inter- and intra-modality fine-grained alignment [2509.18717].

The method is defined against two attack classes. The first is **Targeted Data Poisoning Attacks (TDPAs)**, where attackers inject a very small number of image-caption pairs so that a specific target image will be classified as an adversarial label after pre-training. The second is **Backdoor Attacks (BAs)**, where attackers insert a trigger patch into some images and pair them with adversarial captions so that any image containing the trigger is misclassified as the adversarial label at test time [2509.18717]. This places OTCCLIP in the same broad security landscape as text-side poisoning frameworks such as "ToxicTextCLIP" [2511.00446], but OTCCLIP is a defensive method rather than an attack mechanism.

A plausible implication is that OTCCLIP treats poisoned data not as a special-case anomaly detection problem, but as a cross-modal correspondence problem: the core task is to replace or downweight incorrect image-caption associations before they can shape the CLIP embedding space. That interpretation is consistent with the framework’s emphasis on pair reconstruction and fine-grained alignment [2509.18717].

## 2. Threat model and motivation

The motivation for OTCCLIP is the observation that previous defenses such as RoCLIP and SafeClip perform caption rematching or data partitioning using global CLIP embeddings, which can miss small triggers and subtle mismatches between images and captions [2509.18717]. In the description, global features are said to focus on dominant semantics, so a small trigger or a locally inconsistent caption may only weakly perturb the global representation. As a result, global-feature rematching can fail to distinguish poisoned from clean pairs, can introduce incorrect image-caption pairs, and can harm zero-shot and linear probing performance [2509.18717].

The framework assumes a pre-training dataset \(D\) of image-caption pairs that may contain poisoned pairs. Poisoned data for targeted data poisoning are written as
\[
D_{\text{poi}} = \{(\mathbf{X}_i, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\},
\]
where \(T_{\text{adv}}\) denotes adversarial captions containing the target label text. For backdoor attacks, poisoned data are written as
\[
D_{\text{poi}} = \{(\mathbf{X}_i \oplus \text{patch}, \mathbf{Y}_{\text{poi}(i)}) \mid \mathbf{Y}_{\text{poi}(i)} \in T_{\text{adv}}\}.
\]
The paper reports that prior work shows that **0.0001–0.01%** poisoned data is enough to cause severe targeted misclassification [2509.18717].

The attack types used in evaluation include **BadNet**, **Blended**, **WaNet**, and **LabelConsist** [2509.18717]. This threat model is closely aligned with the broader CLIP poisoning literature. For example, ToxicTextCLIP studies text-based poisoning and backdoor attacks on CLIP pre-training, and reports up to **95.83% poisoning success** and **98.68% backdoor Hit@1** while bypassing RoCLIP, CleanCLIP and SafeCLIP defenses [2511.00446]. That comparison is not a statement that OTCCLIP was evaluated against ToxicTextCLIP specifically; rather, it situates OTCCLIP in a research area where both image-side and text-side contamination are considered practical threats.

## 3. Optimal transport-based caption reconstruction

The first technical component of OTCCLIP is optimal transport-based matching between fine-grained visual and textual features [2509.18717]. For each image \(\mathbf{X}_i\), the CLIP vision encoder \(E_I\) produces spatial features
\[
f_i^s = \{ z^s_{i,1}, z^s_{i,2}, \dots, z^s_{i,h \times w} \}, \quad z^s_{i,j} \in \mathbb{R}^d,
\]
and a global feature \(f_i^g \in \mathbb{R}^d\). For each caption \(\mathbf{Y}_i\), the text encoder \(E_T\) produces token sequence features
\[
y_i^s = \{ \hat{z}^s_{i,1}, \hat{z}^s_{i,2}, \dots, \hat{z}^s_{i,l} \}, \quad \hat{z}^s_{i,j} \in \mathbb{R}^d,
\]
and a global feature \(y_i^g \in \mathbb{R}^d\) [2509.18717].

Given an image \(f_i^s\) and a candidate caption \(y_{p(j)}^s\) from a caption pool, OTCCLIP computes a similarity matrix \(S^P\) between image patches and text tokens and defines a cost matrix
\[
C^P = 1 - S^P.
\]
It then solves an entropy-regularized optimal transport problem
\[
M = \min_{\mathbf{T} \in \Pi(\boldsymbol{\mathcal{G}_f}, \boldsymbol{\mathcal{G}_p})}
\sum_{u,v} T_{uv} C^P_{uv} + \lambda H(\mathbf{T}),
\]
where \(\Pi(\boldsymbol{\mathcal{G}_f}, \boldsymbol{\mathcal{G}_p})\) denotes the set of admissible transport plans between the patch-level and token-level distributions [2509.18717]. Lower OT cost indicates better alignment, and the framework defines \(\hat{M} = 1 - M\) as a similarity score [2509.18717].

For each image in a mini-batch, OTCCLIP chooses the caption in the pool with maximal OT similarity:
\[
y^{s}_{m(j)} = y^s_{p\bigl[\arg\max_{1 \le p \le P} \hat{M}_j^P[p]\bigr]}.
\]
This reassignment is done online using a FIFO caption pool, so the current training pair becomes the image together with its OT-matched caption rather than its original caption [2509.18717]. The pool size used in experiments is \(P = 10000\), and Sinkhorn iterations are used to solve the entropy-regularized OT problem efficiently [2509.18717].

This reconstruction stage is the distinctive mechanism of OTCCLIP. It does not simply reject suspicious data; it redefines the supervisory pairing relation by searching for a locally consistent caption under a fine-grained transport metric. The stated purpose is to break malicious image-caption links introduced by poisoning while avoiding the errors associated with global-feature matching [2509.18717].

## 4. Training objective and alignment structure

After OT-based caption reassignment, OTCCLIP trains CLIP with three losses: the standard CLIP contrastive loss over global features, an inter-modality OT alignment loss, and an intra-modality OT alignment loss [2509.18717].

For the matched global features \(\{f_i^g, y_{m(i)}^g\}_{i=1}^N\), the framework uses the bidirectional CLIP InfoNCE loss
\[
\begin{aligned}
\mathcal{L}_c = & -\frac{1}{2N} \sum_{i=1}^N \log \left[ \frac{\exp(\langle f_i^g, y_{m(i)}^g\rangle / \tau)}
{\sum_{j=1}^N \exp(\langle f_i^g, y_{m(j)}^g\rangle / \tau)} \right] \\
&-\frac{1}{2N} \sum_{j=1}^N \log \left[ \frac{\exp(\langle f_j^g, y_{m(j)}^g\rangle / \tau)}
{\sum_{i=1}^N \exp(\langle f_i^g, y_{m(j)}^g\rangle / \tau)} \right].
\end{aligned}
\]
This preserves the standard CLIP training signal while applying it to OT-reconstructed pairs [2509.18717].

The **inter-modality OT alignment loss** operates on a matched pair’s patch features and token features. Using a cost matrix \(C^m = 1 - S^m\), it defines
\[
\mathcal{L}^a
= \min_{\mathbf{T} \in \Pi(\boldsymbol{\mathcal{G}_f}, \boldsymbol{\mathcal{G}_m})}
\sum_{u,v} T^m_{uv} C^m_{uv} + \lambda H(\mathbf{T}),
\]
and sums this over the batch to obtain \(\mathcal{L}_{IM}\) [2509.18717]. The description states that this encourages high transport weights on semantically consistent patch-token pairs and reduces the impact of mismatched or poisoned local signals [2509.18717].

The **intra-modality OT alignment loss** captures relationships within images and within captions. It computes OT on image patch-to-patch similarities and text token-to-token similarities and combines them into
\[
\begin{aligned}
\mathcal{L}^s &= \min_{\bar{\mathbf{T} \in \Pi(\boldsymbol{\mathcal{G}_f}, \boldsymbol{\mathcal{G}_f})}}
\sum_{u,v} T^{II}_{uv} C^{II}_{uv} + \lambda H(\bar{\mathbf{T}}) \\
&\quad + \min_{\hat{\mathbf{T} \in \Pi(\boldsymbol{\mathcal{G}_m}, \boldsymbol{\mathcal{G}_m})}}
\sum_{u,v} T^{TT}_{uv} C^{TT}_{uv} + \lambda H(\hat{\mathbf{T}}),
\end{aligned}
\]
with batch-wise loss \(\mathcal{L}_{SM} = \sum_{i=1}^N \mathcal{L}_i^s\) [2509.18717]. The stated role of this term is to preserve intrinsic within-modality structure, such as object co-occurrence in images and phrase structure in captions [2509.18717].

The full training objective is
\[
\mathcal{L}_{\text{total}}
= \lambda_c \mathcal{L}_c
+ \lambda_{IM} \mathcal{L}_{IM}
+ \mathbf{1}\{\text{epoch} \bmod K = 0\}\,\lambda_{SM} \mathcal{L}_{SM}.
\]
The reported hyperparameters are \(\lambda_c = 1\), \(\lambda_{IM} = 2\), \(\lambda_{SM} = 0.4\), and \(K = 2\) [2509.18717]. A plausible implication is that OTCCLIP uses the contrastive loss to retain CLIP’s global zero-shot structure, while the OT losses regularize the geometry of fine-grained alignments so that locally inconsistent supervision has less influence.

## 5. Experimental behavior and reported results

OTCCLIP is evaluated on **CC3M**, **CC1M**, **MSCOCO**, and **Visual Genome**, using **ResNet-50** as the image encoder and a **Transformer-based** text encoder [2509.18717]. Training uses learning rate \(5 \times 10^{-5}\), batch size **256**, and **32** epochs; the caption pool size is **10000** [2509.18717]. Poisoning rates are reported as **0.05%** on CC3M and **0.5%** on CC1M, MSCOCO, and Visual Genome for targeted data poisoning, with **5%** for backdoors on MSCOCO and Visual Genome and **0.5%** on CC1M for the set of backdoor attacks [2509.18717].

The main empirical claim is that OTCCLIP almost completely eliminates attack success while improving downstream performance [2509.18717]. On **MSCOCO**, targeted data poisoning attack success rate drops from **68.75%** for CLIP to **6.25%** for OTCCLIP; on **Visual Genome**, targeted data poisoning drops from **75%** to **0%**; on **CC1M**, targeted data poisoning drops from **93.75%** to **0%** [2509.18717]. For backdoors on MSCOCO and Visual Genome, OTCCLIP reports **0%** attack success across **BadNet**, **LabelConsist**, **Blended**, and **WaNet**. On CC1M, **BadNet**, **LabelConsist**, and **WaNet** are reduced to **0%**, while **Blended** is reduced to **0.3%** [2509.18717].

The framework also reports improved zero-shot and linear probing performance on ten datasets including **CIFAR-10**, **CIFAR-100**, **Food-101**, **DTD**, **FGVC Aircraft**, **Flowers-102**, **Caltech-101**, **OxfordIIITPet**, **Stanford Cars**, and **ImageNet-1K** [2509.18717]. On CC1M pre-training, the reported average zero-shot accuracy is about **13.11%** for OTCCLIP versus about **10.35%** for CLIP, and the reported average linear probe accuracy is about **56.73%** for OTCCLIP versus about **48.75%** for CLIP [2509.18717]. The comparison with RoCLIP and SafeClip is explicit: OTCCLIP is described as outperforming both on attack success rate while also improving both zero-shot and linear probing accuracy [2509.18717].

Ablation results attribute distinct roles to the components. Removing OT-based matching and using CLIP global matching instead raises targeted poisoning attack success on CC1M from **0%** to **12.5%** and reduces downstream accuracy [2509.18717]. Removing inter-modality OT alignment reduces zero-shot and linear probe performance while leaving attack success low, and removing intra-modality OT alignment causes a smaller performance drop [2509.18717]. The description also states that adaptive attacks that try to minimize OT distance using PGD still yield **0%** attack success on MSCOCO and Visual Genome [2509.18717].

## 6. Relation to broader optimal transport–CLIP research and limitations

OTCCLIP belongs to a broader line of work that uses optimal transport to enrich CLIP-like or vision-language representations, but its goal is specifically defensive pre-training under poisoning [2509.18717]. In zero-shot out-of-distribution detection, "OT-DETECTOR: Delving into Optimal Transport for Zero-shot Out-of-Distribution Detection" introduces cross-modal transport mass and transport cost as semantic-wise and distribution-wise OOD scores, together with a semantic-aware content refinement module [2503.06442]. In image-text retrieval, "ViCLIP-OT: The First Foundation Vision-Language Model for Vietnamese Image-Text Retrieval with Optimal Transport" integrates CLIP-style contrastive learning with a Similarity-Graph Regularized Optimal Transport loss and reports that on UIT-OpenViIC it achieves an average Recall@K of **67.34%**, improving upon CLIP by **5.75 percentage points**, while in zero-shot evaluation on Crossmodal-3600 it surpasses CLIP by **11.72 percentage points** [2602.22678]. These works suggest that OT has become a general mechanism for refining cross-modal geometry, but OTCCLIP applies that mechanism to robust pair reconstruction and poisoning defense rather than OOD scoring or retrieval.

The principal limitation identified for OTCCLIP is computational overhead from Sinkhorn OT [2509.18717]. The reported per-epoch training times are **1 h 23 min** for RoCLIP, **4 h 11 min** for SafeClip, and **2 h 7 min** for OTCCLIP [2509.18717]. The framework is therefore slower than global-similarity rematching, though the description emphasizes that it remains substantially faster than SafeClip while providing stronger robustness and better downstream accuracy [2509.18717].

Another limitation is implicit in the threat model. OTCCLIP is evaluated primarily against poisoning and backdoor attacks in which local inconsistency between image and caption is exploitable by fine-grained OT matching [2509.18717]. This suggests, rather than proves, that its strongest advantage arises when malicious associations are locally detectable in patch-token geometry. The paper explicitly identifies faster OT computation and extension to other multimodal models and corruption types as future directions [2509.18717].

In the literature on CLIP security, OTCCLIP can therefore be read as a shift from global semantic filtering toward fine-grained transport-based correction. Its contribution is not merely to reject suspicious pairs, but to redefine cross-modal supervision through optimal transport so that CLIP pre-training becomes both more robust to poisoned data and more faithful in downstream representation quality [2509.18717].

Source: https://www.emergentmind.com/topics/otcclip