---
title: 'EVA-CLIP-18B: Open-Source 18B Parameter CLIP Model'
url: https://www.emergentmind.com/topics/eva-clip-18b
type: topic
---

# EVA-CLIP-18B: Open-Source 18B Parameter CLIP Model

EVA-CLIP-18B is an open-source contrastive language-image pretraining (CLIP) model scaling the dual-encoder paradigm to 18 billion parameters. Trained on a fixed corpus of approximately 2 billion publicly available image–text pairs from LAION-2B and COYO-700M, with ≈6 billion total image–text "views," it achieves an average zero-shot top-1 accuracy of 80.7% across 27 image classification benchmarks, outperforming all previous open CLIP models and demonstrating consistent performance improvements with model scaling and no observed saturation. EVA-CLIP-18B was developed using the EVA-style weak-to-strong visual model scaling approach and all associated model weights, code, and training recipes are openly released [2402.04252].

## 1. Model Architecture

EVA-CLIP-18B implements a dual-encoder design based on the CLIP framework, consisting of a Vision Transformer (ViT) for image encoding, a Transformer-based text encoder, and linear projection heads for modality fusion:

- **Vision Encoder**:  
  - 48 transformer layers ($L_v = 48$), hidden dimension of 5120 ($D_v = 5120$), 20480-dim MLP inner layer, 40 attention heads (per head dimension 128).
  - Inputs are $224 \times 224$ images partitioned into $14 \times 14$ patches.
  - RMSNorm replaces LayerNorm, and unlike standard ViT, query, key, and value projections have no bias terms (following LLaMA conventions).

- **Text Encoder**:  
  - 32 transformer layers ($L_t = 32$), hidden size 1280 ($D_t = 1280$), 5120-dim MLP inner layer, 20 attention heads (per head dimension 64).
  - Initialized from EVA-CLIP-E/14+.

- **Projection Heads**:  
  - Linear projections without activation, mapping image and text [CLS] representations to a shared 1024-dimensional “CLIP space.”
  - Both outputs undergo $\ell_2$ normalization.

- **Architectural Notation**:
  $$
  \begin{align*}
  x_i &= \text{raw image},\ y_i = \text{text caption} \\
  z_v(i) &= \text{VisionTransformer}(x_i) \in \mathbb{R}^{D_v} \xrightarrow{\text{Proj}_v} v_i \in \mathbb{R}^d \\
  z_t(i) &= \text{TextTransformer}(y_i) \in \mathbb{R}^{D_t} \xrightarrow{\text{Proj}_t} t_i \in \mathbb{R}^d \\
  \end{align*}
  $$
  where $d = 1024$, and $v_i, t_i$ are $\ell_2$-normalized.

## 2. Training Objective and Protocol

### Contrastive Loss

EVA-CLIP-18B uses the symmetric InfoNCE contrastive loss over mini-batches of $N$ image–text pairs $\{(v_i, t_i)\}_{n}$:
$$
L = -\frac{1}{2N} \left[ \sum_{i} \log \frac{\exp(\operatorname{sim}(v_i, t_i)/\tau)}{\sum_j \exp(\operatorname{sim}(v_i, t_j)/\tau)} + \sum_{i} \log \frac{\exp(\operatorname{sim}(v_i, t_i)/\tau)}{\sum_j \exp(\operatorname{sim}(v_j, t_i)/\tau)} \right]
$$
with $\operatorname{sim}(a, b) = a^\top b$ (cosine similarity), temperature $\tau = 0.01$, and both terms (image-to-text, text-to-image) averaged.

### Training Datasets

- **Merged-2B**: 1.6B LAION-2B pairs plus 0.4B COYO-700M pairs.
- **Merged-2B+** (EVA-CLIP-18B+): Merged-2B plus 20M LAION-COCO synthetic captions and 23M Merged-Video samples (VideoCC, InternVid, WebVid-10M) in final epochs.

No dataset filtering was performed apart from public CLIP filtering and download failure exclusions.

### Curriculum and Optimization

- **Phase 1**: 5.4B image–text samples (patch-dropout; 50% of patches masked).
- **Phase 2**: 0.6B mixed modality (image–text, LAION-COCO, video; varied patch-dropout).
- **Optimizer**: LAMB ($\beta_1=0.9$, $\beta_2=0.95$, no weight decay).
- **Peak learning rates**: $4 \times 10^{-4}$ (vision), $4 \times 10^{-5}$ (text), with cosine decay and layer-wise LR decay ($0.9$ for vision, $0.75$ for text).
- **Batching and Precision**: Global batch size 108k (224×224), mixed-precision bf16, ZeRO Stage-3, gradient checkpointing, FlashAttention.
- **Hardware**: Hundreds of A100-80GB equivalents, coordinated via DeepSpeed ZeRO.

## 3. Empirical Scaling Behavior

Scaling EVA-CLIP models from 5B to 18B parameters on a fixed 2B-image–text pair dataset consistently yielded +0.7–1.0% average zero-shot accuracy per doubling in parameter count, exhibiting a power-law-like trend and no evidence of saturation.

| Model              | Parameters | Avg. Zero-Shot Acc. (27 benchmarks) |
|--------------------|------------|--------------------------------------|
| EVA-CLIP-E/14+     | 5B         | 79.2%                               |
| DFN-5B (H/14)      | 5B         | 79.2%                               |
| WebLI-10B          | 10B        | 79.8%                               |
| InternVL-C         | 12B        | 78.0%                               |
| EVA-CLIP-18B       | 18B        | 80.7%                               |

A plausible implication is that further scaling—either in model size or dataset scale—may yield continued gains [2402.04252].

## 4. Zero-Shot Performance Across Modalities

### Image Classification

EVA-CLIP-18B attains an average zero-shot top-1 accuracy of 80.7% on 27 benchmark datasets. Key per-dataset results include:
- ImageNet-1K: 85.3%
- ImageNet-V2: 83.1%
- ImageNet-Adversarial: 65.4%
- ObjectNet: 70.2%
- CIFAR-100: 92.5%

These results establish a new state-of-the-art among open CLIP models.

### Video Classification

Classification using a single center frame:
- UCF-101: 86.0%
- Kinetics-400: 72.9% (avg. top-1/top-5)
- Kinetics-600: 72.9%
- Kinetics-700: 68.2%

Sampling 8 video frames yields an average performance gain of +4.7% across video benchmarks.

### Image–Text Retrieval

On Flickr30K and COCO, EVA-CLIP-18B achieves an average recall (R@1,5,10) of 87.8%, outperforming prior open models by a significant margin.

## 5. Ablations and Robustness

Multiple ablation studies provide insight into the contributions of various design and training decisions:

- **EVA-Style Weak-to-Strong Scaling**: Training follows a two-stage procedure: a “weak teacher” (EVA-CLIP-E/14+) distills knowledge to the large 18B-parameter model through masked reconstruction, then proceeds with final contrastive pretraining. This stabilizes large-scale training without requiring an increase in data scale.
- **Video Data**: Inclusion of 23M video samples at the end of training results in +0.7% (1-frame) and +0.8% (8-frame) improvements in zero-shot video classification, with negligible effect (–0.1%) on average image–text retrieval.
- **Resolution**: Elevating input resolution from $224^2$ to $336^2$ and $448^2$ improves accuracy by approximately +0.5% and +0.9%, respectively, on select benchmarks.
- **Transform Robustness**: EVA-CLIP-18B exhibits minimal sensitivity (±0.1%) to changes between direct resize and center-crop evaluation pipelines, surpassing prior models in this robustness aspect.

## 6. Significance, Limitations, and Future Directions

EVA-CLIP-18B, as the largest open-model instantiation of CLIP to date, demonstrates that consistent performance improvements are achievable via model scaling under a fixed open dataset, challenging the presupposition that further performance gains require substantially larger or proprietary data. The approach exemplifies the effectiveness of EVA-style weak-to-strong progression for stabilizing massive vision–language models and extends generalization gains across both image and video modalities as well as cross-modal retrieval.

All model artifacts are openly available, supporting reproducibility and further research [2402.04252]. Potential directions include scaling model size and/or dataset further, multimodal finetuning, instruction tuning, and integration in larger multimodal systems that combine vision encoders with large language models.

Source: https://www.emergentmind.com/topics/eva-clip-18b