LeVLJEPA: Non-Contrastive Vision-Language Pretraining

• LeVLJEPA is an end-to-end VLP framework that uses a fully non-contrastive architecture to learn dense semantic representations for both vision and language.
• It leverages cross-modal prediction with stop-gradient targets and per-modality distributional regularization to achieve stable, high-rank embeddings.
• Extensive experiments demonstrate that LeVLJEPA outperforms contrastive baselines on dense prediction tasks while matching them on global representation tasks.

LeVLJEPA is an end-to-end vision–language pretraining (VLP) framework that introduces the first fully non-contrastive approach to cross-modal representation learning. Unlike prevailing contrastive methods, LeVLJEPA is designed to produce high-quality dense semantic features for both vision and vision–language tasks without reliance on negative samples, temperature parameters, momentum encoders, or teacher–student schedules. LeVLJEPA operates by leveraging cross-modal prediction with stop-gradient targets and per-modality distributional regularization, training stably at scale, and yielding encoders that outperform contrastive baselines for dense downstream tasks while matching them on global representations (Kuhn et al., 1 Jul 2026).

1. Historical Context and Motivations

Vision–language pretraining has been dominated by contrastive objectives such as InfoNCE (used in CLIP) and SigLIP, which supervise the alignment of a single pooled embedding per image or text by maximizing agreement on matched pairs while repelling all other pairs in the batch. These methods excel in zero-shot retrieval and classification but present two limitations: dependence on large batch negatives for performance scaling, and supervision focused solely on global representations, neglecting dense patch/grid-level features critical for dense prediction tasks.

In contrast, self-supervised learning within vision-only domains has shifted toward non-contrastive approaches relying on predictive objectives and distributional regularization (e.g., BYOL, SimSiam, I-JEPA, LeJEPA). These methods provide high-rank, high-quality visual features without explicit negative mining. LeVLJEPA builds on this non-contrastive paradigm and transfers it to vision–language pretraining with an emphasis on dense semantic structure and stable, scalable training.

2. Model Architecture

LeVLJEPA utilizes the Joint-Embedding Predictive Architecture (JEPA) style to learn cross-modal alignment:

• Vision Encoder: A ViT-B/16 transformer forms the visual backbone, dividing each image into 16×16 patches, encoding them through transformer layers, and extracting a CLS token. This token is projected via a one-layer MLP (2048-dimensional hidden with GELU, BatchNorm, and Dropout) to obtain the vision embedding $z^v \in \mathbb{R}^d$, with no $\ell_2$-normalization during training to permit effective SIGReg operation.
• Text Encoder: A GPT-2 transformer encodes captions, extracting the final hidden state of the last token, which is similarly projected to $z^t \in \mathbb{R}^d$ by a matching one-layer MLP, again without $\ell_2$-normalization.
• Cross-Modal Predictors: Two symmetric MLPs (each with 4 hidden layers, width 2048, BatchNorm, GELU, 10% dropout) transform modality-specific embeddings into the counterpart’s embedding space: $h_v(z^v) \to \hat{y}^v$ predicts $z^t$, and $h_t(z^t) \to \hat{y}^t$ predicts $z^v$. Gradients flow only through the predicting encoder, with targets stop-gradiented.
• Distributional Regularizer (SIGReg): Sketched Isotropic-Gaussian Regularization is applied separately to each modality, using a characteristic function-based test (Epps–Pulley) over projections along random directions, constraining the marginal distribution toward an isotropic Gaussian and maintaining high embedding rank.

3. Pretraining Objectives and Training Dynamics

The LeVLJEPA loss is a weighted sum of cross-modal prediction and per-modality regularization:

• Cross-Modal Prediction Loss:

$$\mathcal{L}_{\text{cross}} = \frac{1}{B} \sum_{i=1}^B [\| h_v(z^v_i) - \mathrm{sg}(z^t_i) \|_2^2 + \| h_t(z^t_i) - \mathrm{sg}(z^v_i) \|_2^2 ]$$

where $\mathrm{sg}(\cdot)$ denotes the stop-gradient operator. This design induces asymmetry that avoids collapse to a degenerate shared subspace.

• SIGReg Regularization:

Given embeddings $\ell_2$0 and set $\ell_2$1 of random unit directions $\ell_2$2, SIGReg is computed as:

$\ell_2$3

where $\ell_2$4.

• Full Loss Objective:

$\ell_2$5

where $\ell_2$6 and $\ell_2$7.

Training uses standard AdamW optimization, a linear warmup with cosine decay, and global batch sizes up to 4096 on 16 A100/H100 GPUs. Hyperparameter stability holds over batch sizes from 1024 to 4096, with predictor depth ≥2 and SIGReg weight $\ell_2$8 in [0.005, 0.04] ensuring high-rank, stable representations.

4. Empirical Evaluation

LeVLJEPA demonstrates distinctive empirical profiles across various benchmarking regimes.

Global Representation (Pooled Embedding) Tasks

• On Datacomp-L zero-shot classification, LeVLJEPA lags contrastive methods (InfoNCE, SigLIP) by 5–10 percentage points on ImageNet and similar margins elsewhere, consistent with its lack of explicit global alignment optimization.
• Linear probing performance is virtually matched by all objectives.

Global Representation Performance

Method	ImageNet (ZS)	ImageNet (LP)
InfoNCE	47.32	65.75
SigLIP	50.78	66.34
LeVLJEPA	42.45	65.42

Object-Centric Robustness

• On ImageNet-9 background robustness, LeVLJEPA’s global feature is less background-sensitive and more object-centric than contrastive baselines.

Method	Original	Mixed-Same	Mixed-Rand
InfoNCE	95.98	89.41	77.31
SigLIP	96.44	89.41	78.35
LeVLJEPA	96.96	91.01	79.75

Dense Prediction

• On linear semantic segmentation (mean IoU, Datacomp-L), LeVLJEPA outperforms contrastive baselines by +2.25 on ADE20K and +2.08 on COCO-Stuff, evidencing stronger per-token semantic structure relevant for dense downstream tasks.

Method	ADE20K	COCO-Stuff
InfoNCE	20.90	29.02
SigLIP	19.24	28.88
LeVLJEPA	23.15	31.10

Downstream Vision–LLM Backbones

• Employing LeVLJEPA-pretrained vision encoders as frozen backbones in VLMs (with Llama-1B or Qwen-1.5B) yields the best performance across GQA, VQAv2, and POPE benchmarks, despite no further finetuning.

Frozen Backbone VLM Performance (Accuracy %)

Backbone	GQA	VQAv2	POPE
InfoNCE–Llama	42.7	51.6	65.8
SigLIP–Llama	42.4	49.2	62.0
LeVLJEPA–Llama	44.6	54.2	66.9
InfoNCE–Qwen	42.2	49.4	71.5
SigLIP–Qwen	41.6	47.7	70.4
LeVLJEPA–Qwen	43.7	54.1	75.0

5. Analysis and Ablation Studies

Comprehensive ablations on CC12M highlight the necessity of combining cross-modal predictors, stop-gradient targets, and per-modality SIGReg to achieve stable, high-rank embeddings and transferable alignment:

• Direct MSE alignment without regularization results in severe collapse (effective rank 3/25).
• Adding SIGReg improves rank modestly, but only the full LeVLJEPA combination yields high effective ranks (477/358 for vision/text) and strong zero-shot/linear-probe performance.
• Predictor depth ≥2 and SIGReg weight in [0.005, 0.04] are both necessary and sufficient for stability.

Scaling experiments indicate smooth improvements with model and data size, with LeVLJEPA remaining competitive or superior, particularly as models reach larger scales. LeVLJEPA’s batch size invariance (<1% variance in metrics across $\ell_2$9) is notable, since contrastive learning typically degrades sharply with reduced batch size.

6. Implications and Open Directions

LeVLJEPA establishes non-contrastive end-to-end VLP as a practical and effective approach for dense semantic feature learning. Its architecture eliminates the machinery of negatives, temperatures, and momentum/teacher–student networks, simplifying large-scale pretraining pipelines. The contrastive/non-contrastive distinction is manifest at the patch-token sequence: LeVLJEPA preserves richer dense structure, while contrastive baselines enforce only global pooled alignment. As the usage of vision–LLMs shifts toward dense, per-token consumption (including segmentation, instruction-following, and robotics), LeVLJEPA’s ability to produce structured grid-level features aligns with emerging requirements.

Open questions remain in unifying strong global alignment (for zero-shot and retrieval) with dense semantics within a single training objective, scaling non-contrastive VLP to larger models and datasets, and applying these techniques to additional modalities such as audio and video. A plausible implication is that future VLP methods may increasingly favor non-contrastive objectives for applications demanding quality dense features (Kuhn et al., 1 Jul 2026).