Replace VL-JEPA’s InfoNCE with a sample-independent anti-collapse regularizer

Investigate replacing the bidirectional InfoNCE loss in VL-JEPA with a sample-independent anti-collapse regularizer and determine whether such a formulation can prevent collapse and enable effective training without batch-level negatives.

Background

VL-JEPA frames vision–language learning as predictive alignment between image-predicted and text target embeddings but uses a bidirectional InfoNCE loss, inheriting batch-negative sampling and batch-size dependence. The paper notes that VL-JEPA explicitly mentions the possibility of substituting InfoNCE with a sample-independent anti-collapse regularizer, which would eliminate the reliance on negatives, but that prior work left this substitution to future work.

References

Notably, \citet{chen2025vljepa} observe that the InfoNCE term could in principle be replaced by a sample-independent anti-collapse regularizer but leave this to future work.

— LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives  (2607.00784 - Kuhn et al., 1 Jul 2026) in Section 2.2 (Contrastive Vision–Language Pretraining)

Two candidate mitigations are under consideration: variance-invariance-covariance regularization, as proposed by Bardes, Ponce, and LeCun under the name VICReg, and contrastive training objectives. No selection has been made.

— Project Qualia: Recovering Experiential Music Structure from Session Co-occurrence Data  (2609.10862 - Mohammed et al., 9 Sep 2026) in Section 6.4, “Unresolved Design Parameters”