CLIPin: Non-Contrastive Plug-In for CLIP

• CLIPin is a non-contrastive plug-in that augments CLIP models with a BYOL/SimSiam-like branch for instance-level semantic alignment.
• It incorporates shared pre-projectors and dual online–target networks to jointly enforce contrastive, inter-modal, and intra-modal losses.
• CLIPin adapts to varied datasets from natural images to medical imaging, yielding improved cross-modal retrieval performance and interpretability.

CLIPin is a non-contrastive plug-in for CLIP-style multimodal pretraining that augments standard contrastive language–image training with a BYOL/SimSiam-like branch for instance-level semantic alignment. It is designed for regimes in which the negative-pair assumptions of InfoNCE are unreliable, notably large-scale natural image–text datasets with loose semantic alignment due to weak supervision and medical datasets with high cross-modal correlation but low content diversity. In CLIPin, a standard CLIP objective is retained, but a parallel non-contrastive objective is added through online–target dual networks, shared pre-projectors, and inter- and intra-modal regression losses, yielding a plug-and-play component that is reported to be compatible with multiple contrastive frameworks and effective across natural and medical downstream tasks (Yang et al., 8 Aug 2025).

1. Motivation and problem formulation

CLIPin is motivated by a failure mode of standard CLIP-style InfoNCE training: each matched image–text pair is optimized against a batch of negatives under an implicit assumption that non-matching pairs are semantically incorrect and mutually exclusive. The paper argues that this assumption is routinely violated in two settings. In large web-scale natural image–text corpora, pairs are often only loosely aligned, so several captions in the batch may be partially relevant to the same image. In medical corpora, image–report pairs are highly correlated but semantically repetitive, so many negatives are nearly equivalent to positives. In both cases, contrastive learning is forced to separate representations that should not necessarily be pushed apart, introducing semantic noise or semantic redundancy into the optimization (Yang et al., 8 Aug 2025).

This diagnosis places CLIPin within a broader line of work that seeks alternatives to pure pairwise contrastive supervision when data alignment is imperfect. Unlike Set-CLIP, which reframes low-alignment multimodal learning as manifold matching with MK-MMD and Semantic Density Distribution loss (Song et al., 2024), or CLIP-PING, which strengthens lightweight CLIP training with nearest neighbors (NN) and cross nearest neighbors (XNN) mined from unimodal teachers (Thwal et al., 2024), CLIPin addresses the problem by injecting a non-contrastive instance-level regression signal directly into a CLIP-style architecture (Yang et al., 8 Aug 2025).

The resulting objective is not a replacement for CLIP but a joint formulation. Contrastive learning remains responsible for global discrimination, while the new branch supplies supervision that does not depend on negatives. This suggests a division of labor in which InfoNCE preserves the usual cross-modal retrieval geometry, and the non-contrastive branch regularizes it when negative-pair semantics are unreliable (Yang et al., 8 Aug 2025).

2. Architectural design

The base model in CLIPin is a standard CLIP-style backbone with an image encoder $f_\theta$, a text encoder $f_\phi$, and a contrastive loss $\mathcal{L}_{\mathrm{CL}}$. In the reported experiments, the image encoder is ViT-B/16, and models trained on COCO are initialized from OpenAI CLIP, whereas models trained on MUGE and Tongren are initialized from CN-CLIP (Yang et al., 8 Aug 2025).

The key architectural intervention is the insertion of two shared pre-projectors, one for images and one for text. These pre-projectors map encoder outputs to a shared 1024-dimensional space. From this shared representation, the network forks into two branches. The first is the original contrastive branch, whose heads $g_{\mathrm{CL,I}}$ and $g_{\mathrm{CL,T}}$ map 1024 $\rightarrow$ 512. The second is the new non-contrastive branch, whose heads $g_{\mathrm{NCL,I}}$ and $g_{\mathrm{NCL,T}}$ map 1024 $\rightarrow$ 8192, and which further includes predictors for inter-modal and intra-modal regression (Yang et al., 8 Aug 2025).

This decomposition is explicitly presented as a compromise between two projector regimes. The contrastive branch retains a relatively low-dimensional bottleneck, while the non-contrastive branch uses a much larger projection space. The shared pre-projectors are intended to couple these paradigms through parameter sharing, so that both the contrastive and non-contrastive objectives shape the same early projected representation (Yang et al., 8 Aug 2025).

The non-contrastive branch adopts an online–target structure. For each modality, the online branch contains the encoder, the shared pre-projector, the non-contrastive projector, and predictors. The target branch contains momentum versions of the encoder, shared pre-projector, and non-contrastive projector, but no predictor. Target parameters are updated by exponential moving average: $$\begin{aligned} \mathcal{M}_{\text{target,I/T}}^0 &= \mathcal{M}_{\text{online,I/T}}^0, \ \mathcal{M}_{\text{target,I/T}}^t &\leftarrow \beta \cdot \mathcal{M}_{\text{target,I/T}}^{t-1} + (1-\beta) \cdot \mathcal{M}_{\text{online,I/T}}^t . \end{aligned}$$ This is the mechanism through which CLIPin imports BYOL/SimSiam-like asymmetry into multimodal training (Yang et al., 8 Aug 2025).

3. Objective functions and learning mechanism

The contrastive part of CLIPin remains standard CLIP. Using contrastive features

$f_\phi$0

the InfoNCE loss is defined in symmetric image-to-text and text-to-image form: $f_\phi$1

The distinctive part of CLIPin is the non-contrastive branch. For inter-modal alignment, online predictions from one modality are regressed to target features of the other: $f_\phi$2 with losses

$f_\phi$3

For intra-modal alignment, CLIPin adds self-alignment within each modality: $f_\phi$4 with

$f_\phi$5

The full training objective is

$f_\phi$6

where $f_\phi$7 and $f_\phi$8 are learnable, initialized to 1.0 (Yang et al., 8 Aug 2025).

The paper’s interpretation is that the inter-modal branch supplies instance-level cross-modal regression, whereas the intra-modal branch stabilizes optimization across two heterogeneous encoders. The ablation study supports this view: adding inter-modal alignment alone can give only limited or unstable gains, whereas combining inter + intra, and then adding shared pre-projectors, yields the strongest improvements (Yang et al., 8 Aug 2025).

4. Training protocol and datasets

CLIPin uses two augmentations of each image and two text views per paired sample. Images are resized to $f_\phi$9 and augmented with random horizontal flip (p=0.5) and color jitter (strength 0.1). Text length is capped at 77 tokens, consistent with CLIP/CN-CLIP tokenization (Yang et al., 8 Aug 2025).

The reported optimization configuration is: AdamW, learning rate $\mathcal{L}_{\mathrm{CL}}$0, 100 warmup iterations, $\mathcal{L}_{\mathrm{CL}}$1, $\mathcal{L}_{\mathrm{CL}}$2, $\mathcal{L}_{\mathrm{CL}}$3, weight decay 0.001, batch size 256, temperature $\mathcal{L}_{\mathrm{CL}}$4, and EMA momentum $\mathcal{L}_{\mathrm{CL}}$5. Training uses automatic mixed precision, and on a single RTX 3090 is reported to take about 24 h, using about 14 GB GPU memory (Yang et al., 8 Aug 2025).

The principal training datasets are:

• COCO: 82.8k images, 414.1k captions
• MUGE: 250.4k Chinese e-commerce image–text pairs
• Tongren: 451.9k retinal image–report pairs (Yang et al., 8 Aug 2025)

Downstream evaluation covers natural and medical benchmarks. The natural-image suite includes CIFAR-10, CIFAR-100, SUN397, PASCAL VOC2007, and Caltech-101. The medical suite includes RFMiD, ODIR, REFUGE, MESSIDOR, and FIVES. Two evaluation protocols are emphasized: linear probing on frozen image encoders and prompt-based out-of-distribution zero-shot classification using image–text cosine similarity (Yang et al., 8 Aug 2025).

This training setup underscores the paper’s plug-in claim. CLIPin is applied not only to CLIP/CN-CLIP initialization, but also to other multimodal frameworks. In a separate study it is integrated into ALBEF, BLIP, and CoCa, where it is trained from scratch and still improves results in most reported settings (Yang et al., 8 Aug 2025).

5. Empirical performance and ablations

The main comparison is against CLIP and xCLIP. On PASCAL VOC2007, with COCO pretraining, prompt-based OOD zero-shot classification improves from 91.33 / 76.47 for CLIP and 93.81 / 77.74 for xCLIP to 94.90 / 85.47 for CLIP+CLIPin in AUC / mAP. On SUN397, with MUGE pretraining, OOD zero-shot performance improves from 91.29 / 14.94 for CLIP and 91.60 / 16.14 for xCLIP to 95.17 / 25.99 for CLIP+CLIPin. On the medical side, with Tongren pretraining and MESSIDOR evaluation, prompt-based OOD zero-shot results improve from 67.62 / 41.20 for CLIP and 59.27 / 37.82 for xCLIP to 72.89 / 48.88 for CLIP+CLIPin (Yang et al., 8 Aug 2025).

Linear probing also improves. On PASCAL VOC2007, with COCO pretraining, CLIP reports 87.18 / 41.92, xCLIP 85.95 / 40.34, and CLIP+CLIPin 87.43 / 43.43 in AUC / mAP. On RFMiD, the ablation table reports a progression from 86.76 / 40.87 for CLIP to 88.89 / 41.71 for the full CLIPin formulation (Yang et al., 8 Aug 2025).

The generality study with ALBEF, BLIP, and CoCa is presented as evidence that CLIPin is a framework-level component rather than a CLIP-specific hack. For example, on CIFAR-10 with COCO pretraining, CoCa improves from 89.05 / 54.46 to 89.83 / 56.87, and on Caltech-101 with MUGE pretraining from 86.12 / 25.73 to 88.40 / 31.54 when CLIPin is added (Yang et al., 8 Aug 2025).

The ablation study isolates three components: inter-modal alignment, intra-modal alignment, and shared pre-projectors. The reported trend is that inter-modal alignment alone can yield slight gains or occasional degradation, whereas adding intra-modal alignment makes improvements more consistent, and the full system with shared pre-projectors performs best. This is the central empirical justification for the architectural claim that non-contrastive alignment must be stabilized within modality before it can reliably strengthen cross-modal alignment (Yang et al., 8 Aug 2025).

Grad-CAM visualizations further support this interpretation. For natural images such as “Train” and “Bird”, CLIPin is reported to produce more focused and spatially contiguous activations. For retinal images with lesion annotations, CLIPin’s heatmaps align better with disease-relevant regions than those of CLIP or xCLIP, which the paper presents as qualitative evidence of improved instance-level semantic alignment and interpretability (Yang et al., 8 Aug 2025).

6. Relation to similarly named methods, limitations, and significance

The name CLIPin requires disambiguation. It is distinct from “CLIP-IN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions”, which improves CLIP through instruction-editing datasets, symmetric hard negative contrastive loss, long descriptive captions, and a RoPE-based text encoder (Wang et al., 4 Aug 2025). It is also unrelated to CLIP-PING, a training paradigm for lightweight CLIP-like models based on Proximus Intrinsic Neighbors Guidance (Thwal et al., 2024). CLIPin instead denotes a non-contrastive plug-in for multimodal semantic alignment (Yang et al., 8 Aug 2025).

Within the broader research landscape, CLIPin can be understood as one response to the shortcomings of pure InfoNCE in multimodal learning. Set-CLIP approaches low-alignment multimodal learning through distribution matching and Semantic Density Distribution loss (Song et al., 2024); CLIP-PING augments CLIP with neighbor-based extra contrastive supervision (Thwal et al., 2024); CLIPin adds instance-level non-contrastive regression while retaining the original CLIP objective (Yang et al., 8 Aug 2025). This suggests a broader methodological distinction between methods that repair contrastive training by improving negatives, methods that reduce dependence on negatives, and methods that combine both.

The paper also identifies limitations. CLIPin introduces extra overhead through EMA branches, 8192-dimensional non-contrastive projectors, and multiple predictors. It is evaluated on COCO, MUGE, and Tongren, rather than on LAION-scale corpora, so its behavior at much larger scales is not directly established. The text augmentations used to form $\mathcal{L}_{\mathrm{CL}}$6 and $\mathcal{L}_{\mathrm{CL}}$7 are described as semantically consistent, but the specific transformation family is not detailed. In addition, the paper notes that inter-modal non-contrastive alignment alone can be unstable, which is why the intra-modal branch is included as a stabilizer (Yang et al., 8 Aug 2025).

The significance of CLIPin lies in its reformulation of multimodal pretraining as a joint contrastive and non-contrastive problem. Its central claim is not that contrastive learning should be discarded, but that it is insufficiently robust when the semantics of negatives are unreliable. The plug-in design, the shared pre-projectors, and the reported compatibility with CLIP, ALBEF, BLIP, and CoCa position CLIPin as a modular method for strengthening multimodal semantic alignment under weak supervision, semantic redundancy, or both (Yang et al., 8 Aug 2025).