Combine dense-feature advantage with competitive zero-shot alignment in a single objective

Determine whether a single vision–language pretraining objective can simultaneously preserve the dense per-token feature advantages obtained by non-contrastive JEPA-style pretraining (as instantiated by LeVLJEPA’s cross-modal prediction with stop-gradient targets and per-modality SIGReg) and achieve zero-shot image–text alignment competitive with contrastive objectives such as CLIP and SigLIP.

Background

The paper shows that LeVLJEPA, a fully non-contrastive cross-modal JEPA-style method, yields stronger dense semantic features than contrastive baselines for tasks that consume patch-token sequences, such as semantic segmentation and as a frozen backbone for vision–LLMs. However, LeVLJEPA trails contrastive methods on zero-shot image–text alignment, which directly reflects the contrastive training objective. This prompts the question of whether one can unify the benefits—retaining LeVLJEPA’s dense-token strengths while also matching contrastive zero-shot alignment—within a single training objective.