Contrastive Language-Image Pre-training (CLIP)

• Contrastive Language-Image Pre-trained (CLIP) models are dual-encoder architectures that align images and texts through a symmetric contrastive loss.
• They deliver robust zero-shot, transfer, and retrieval performance across diverse tasks by leveraging large-scale, web-curated image-text data and prompt engineering.
• Recent advances improve efficiency, domain adaptation, and fine-grained reasoning via auxiliary losses, negative mining, and innovative supervision techniques.

Contrastive Language-Image Pre-trained (CLIP) Models

Contrastive Language-Image Pre-trained (CLIP) models are dual-encoder architectures that jointly learn aligned representations of images and texts through large-scale, symmetrical contrastive objectives using natural language supervision without the need for explicit per-label or localized annotations. CLIP models have demonstrated strong zero-shot, transfer, and retrieval performance across a wide array of vision and language tasks, leveraging the diversity and scale of web-curated or domain-specific image-text corpora. Since their introduction, CLIP architectures and training protocols have been foundational for a diverse set of derivative models, transfer learning strategies, and robustness/efficiency improvements, as well as new applications in scientific, medical, and multilingual domains.

1. Core Architecture and Training Objectives

The standard CLIP architecture comprises two separate encoders: an image encoder $f_I$ (typically a ResNet or ViT backbone) and a text encoder $f_T$ (Transformer-based), each mapping their respective inputs to a shared $d$-dimensional, L2-normalized embedding space. For a minibatch of $N$ paired image-text samples $\{(I_i, T_i)\}_{i=1}^N$, CLIP defines similarity by cosine:

$\text{sim}(u, v) = \frac{u^\top v}{\|u\| \|v\|}$

where $u = f_I(I)$ and $v = f_T(T)$. The symmetric InfoNCE contrastive loss encourages matched pairs to have maximal similarity and unmatched pairs within the batch to be minimally similar:

$$L_\mathrm{CLIP} = \frac{1}{2} (L_{v\to t} + L_{t \to v})$$

with the two terms,

$$L_{v \to t} = -\frac{1}{N}\sum_{i=1}^N \log \frac{\exp(\text{sim}(z^I_i, z^T_i)/\tau)}{\sum_{j=1}^N \exp(\text{sim}(z^I_i, z^T_j)/\tau)}$$

$f_T$0

where $f_T$1 is a learned temperature parameter. This objective is agnostic to the granularity of supervision in text; it only aligns global image-text content (Yan et al., 2022, Bianchi et al., 2021).

2. Prompting, Multilingual, and Domain Adaptation Strategies

CLIP's text encoder was not designed for stand-alone natural language understanding, but its performance can be substantially enhanced in phrase understanding and entity-centric tasks by prompt engineering. In "CLIP also Understands Text" (Yan et al., 2022), a domain-adaptive prompting strategy is introduced: for a phrase $f_T$2, K domain-relevant keywords are generated by an LM and concatenated into a prompt, e.g., "A photo of [phrase]. A [keyword$f_T$3], ... [keyword$f_T$4]." This prompt-augmented embedding, when used for tasks such as entity clustering and set expansion, enables CLIP's text encoder to rival or outperform specialized LMs and phrase encoders on multiple datasets, with performance peaking at $f_T$5 keywords.

Multilingual adaptation requires careful data curation and transfer. The CLIP-Italian model (Bianchi et al., 2021) is trained on $f_T$61.4M Italian image-text pairs (across WIT, MSCOCO-IT, DeepL-translated Conceptual Captions, and ILPOST), leveraging pre-trained Italian BERT and ViT backbones and the standard CLIP symmetric contrastive loss. CLIP-Italian significantly outperforms mCLIP on image retrieval and zero-shot classification in Italian, despite being trained on orders of magnitude less data than English CLIP.

In biomedical imaging, domain-specific adaptations use fine-grained curation and architectural extensions to address multi-view and volumetric inputs (e.g., RadCLIP with slice pooling adapters for 3D radiologic images (Lu et al., 2024), Mammo-CLIP for four-view mammography (Chen et al., 2024), and PMC-CLIP for subfigure/subcaption aligned biomedical text (Lin et al., 2023)). In these settings, CLIP’s modular design and prompt-driven transfer learning facilitate effective domain adaptation.

3. Advances in Training Efficiency and Representation Quality

Standard CLIP's reliance on large batch sizes (e.g., 32K) poses significant hardware constraints. Several methods address these bottlenecks:

• AmorLIP (Sun et al., 25 May 2025) amortizes the expensive partition function estimation by learning a small MLP surrogate for $f_T$7, allowing robust contrastive pre-training with smaller batches and extremely low computational overhead, yielding up to 12.24% relative improvements across 38 downstream tasks, and converging 13–30% faster than standard CLIP.
• DeCLIP (Li et al., 2021) enhances data efficiency by incorporating three auxiliary losses: intra-modal self-supervision (SimSiam/MLM), cross-modal multi-view contrast (contrasting stochastic augmentations), and nearest-neighbor supervision from a feature queue. DeCLIP-R50 matches or surpasses CLIP-R50 with %%%%18$v = f_T(T)$19%%%% fewer data samples (56M vs. 400M).
• HELIP (Wang et al., 2023) selectively upweights hard negatives (in-batch pairs with maximal cross-modal similarity among non-matching pairs) during contrastive optimization. This delivers 2–3% zero-shot gains at negligible additional cost, and 8–18% improvements on fine-grained tasks due to sharpened decision boundaries.

Efficiency enhancements augment or replace expensive pre-training paradigms without sacrificing (and often boosting) zero-shot and transferability, leveraging either amortized partition estimation, advanced negative mining, or auxiliary supervision schemes.

4. Robustness, Generalization, and Safety Analysis

CLIP models exhibit marked robustness to natural distribution shifts—a property shown to derive primarily from the diversity and scale of the pretraining corpus, not from the contrastive loss or language supervision per se. Comprehensive experimental investigations (Fang et al., 2022, Tu et al., 2024) demonstrate:

• Data diversity is the determinant of distributional robustness. When the underlying image distribution is broadened (e.g., YFCC-15M vs. ImageNet), both CLIP and supervised models show increased out-of-distribution (OOD) accuracy, independent of the loss or architectural choices.
• Prompting strategies (engineering, number of templates, synonyms) affect raw accuracy but do not materially impact effective robustness.
• Model calibration, OOD detection, and factor-level resilience depend more on training source (e.g., LAION vs. WIT) and intermediate fine-tuning than on architecture or prompting. Zero-shot CLIP models do not always exhibit better calibrated uncertainty than supervision-only baselines (Tu et al., 2024).
• Adversarial vulnerabilities: CLIPMasterPrints (Freiberger et al., 2023) exploit the modality gap between CLIP’s image and text embedding distributions, enabling adversarially synthesized images to confound the model across many text prompts. Centroid-shifting defenses and adversarial input detectors can mitigate this vulnerability with minimal impact on task accuracy.

The findings collectively indicate that CLIP's cross-domain generalization and safety properties are a function of data-centric design choices and, to a lesser extent, prompt and fine-tuning configurations.

5. Task-Specific, Structural, and Knowledge-Enhanced Variants

Recent work extends CLIP to better accommodate complex tasks and fine-grained visual-textual reasoning:

Variant	Key Mechanism	Notable Gains
HiCLIP (Geng et al., 2023)	Hierarchy-aware attention (tree/group) in both encoders	+7.7% ImageNet, +10% avg over baselines
SuperCLIP (Zhao et al., 16 Dec 2025)	Augmentation with lightweight token-level classification head	+4% top-1 ImageNet, +2–3% retrieval, strong gains for long captions
TripletCLIP (Patel et al., 2024)	Alternating contrastive losses with synthetic hard negatives and matching negative images	+9–11% on SugarCrepe compositional reasoning
CLIP+Model Zoo Experts (Salehi et al., 2023)	Joint contrastive + pseudo-supervision (segmentation, detection, depth, normals)	+16.3% mIoU (VOC), +1.7% COCO detection
SemCLIP (Ngan et al., 20 Nov 2025)	Paraphrasing and negation-aware losses in learned semantic subspace	+10% orig-over-negated on CC-Neg benchmark

HiCLIP inserts nonparametric tree/group structure into each Transformer block, yielding progressive, unsupervised induction of semantic hierarchies in both text and image encoders, leading to substantially higher alignment and transfer performance.

SuperCLIP augments contrastive learning with a simple multi-label classification objective over all text tokens present in the caption, enhancing global-to-token alignment and mitigating the batch-size dependence of vanilla CLIP.

TripletCLIP introduces synthetic contrastive triplets via in-context LLM-generated hard-negative captions and associated negative images from a diffusion model, alternating losses to directly augment the model's compositional reasoning and retrieval capacity.

Pseudo-supervision via model zoos incorporates dense correspondences (masks, depth, surface normals) as auxiliary losses, dramatically lifting spatially-precise vision task performance while preserving CLIP's zero-shot ability.

SemCLIP explicitly incorporates both semantic invariance (paraphrases) and exclusivity (negations) into the loss via LLM-generated triples, robustifying CLIP’s retrieval and classification under natural language transformations, especially for semantic negation.

6. Limitations, Open Issues, and Outlook

Despite their impressive capabilities, CLIP models and their extensions share several limitations:

• Without prompt adaptation, CLIP can fail in out-of-distribution or specialized domains (e.g., biomedical entities or complex multi-modal cases) (Yan et al., 2022, Lin et al., 2023).
• Prompt engineering and keyword generation commonly depend on external LMs (e.g., BERT), and errors or biases propagate to downstream tasks.
• Adversarial and spurious correlations arising from the modality gap or data biases create vulnerabilities that require architectural or procedural defenses (Freiberger et al., 2023).
• Most improved efficiency and performance methods (e.g., AmorLIP, DeCLIP, HELIP) have not yet been tested at LAION-scale ($d$0100M examples) or with the largest ViTs.

Future research directions highlighted by multiple works include scaling existing approaches to larger datasets/models, improved prompt tuning (learned templates, joint multimodal adaptation), further integration of dense pseudo‐supervision, compositional and negation-invariant objectives, and systematic data-centric curricula for enhanced robustness and grounding (Sun et al., 25 May 2025, Ngan et al., 20 Nov 2025, Salehi et al., 2023). Additionally, the use of CLIP-like pretraining in scientific, medical, and cross-lingual domains remains an evolving frontier (Lin et al., 2023, Bianchi et al., 2021, Metzger, 2023).

---

References

• "CLIP also Understands Text: Prompting CLIP for Phrase Understanding" (Yan et al., 2022)
• "Contrastive Language-Image Pre-training for the Italian Language" (Bianchi et al., 2021)
• "Getting More Juice Out of Your Data: Hard Pair Refinement Enhances Visual-LLMs Without Extra Data" (Wang et al., 2023)
• "Fooling Contrastive Language-Image Pre-trained Models with CLIPMasterPrints" (Freiberger et al., 2023)
• "AmorLIP: Efficient Language-Image Pretraining via Amortization" (Sun et al., 25 May 2025)
• "Mammo-CLIP: Leveraging Contrastive Language-Image Pre-training (CLIP) for Enhanced Breast Cancer Diagnosis with Multi-view Mammography" (Chen et al., 2024)
• "Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm" (Li et al., 2021)
• "CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement" (Salehi et al., 2023)
• "Training CLIP models on Data from Scientific Papers" (Metzger, 2023)
• "Contrastive vision-language learning with paraphrasing and negation" (Ngan et al., 20 Nov 2025)
• "ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Model" (Chen et al., 2024)
• "SuperCLIP: CLIP with Simple Classification Supervision" (Zhao et al., 16 Dec 2025)
• "PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents" (Lin et al., 2023)
• "Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-training" (Wang et al., 2024)
• "RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-training" (Lu et al., 2024)
• "TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives" (Patel et al., 2024)
• "HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention" (Geng et al., 2023)
• "Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)" (Fang et al., 2022)
• "VT-CLIP: Enhancing Vision-LLMs with Visual-guided Texts" (Qiu et al., 2021)
• "A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)" (Tu et al., 2024)