CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP (2110.11316v4)

Abstract: CLIP yielded impressive results on zero-shot transfer learning tasks and is considered as a foundation model like BERT or GPT3. CLIP vision models that have a rich representation are pre-trained using the InfoNCE objective and natural language supervision before they are fine-tuned on particular tasks. Though CLIP excels at zero-shot transfer learning, it suffers from an explaining away problem, that is, it focuses on one or few features, while neglecting other relevant features. This problem is caused by insufficiently extracting the covariance structure in the original multi-modal data. We suggest to use modern Hopfield networks to tackle the problem of explaining away. Their retrieved embeddings have an enriched covariance structure derived from co-occurrences of features in the stored embeddings. However, modern Hopfield networks increase the saturation effect of the InfoNCE objective which hampers learning. We propose to use the InfoLOOB objective to mitigate this saturation effect. We introduce the novel "Contrastive Leave One Out Boost" (CLOOB), which uses modern Hopfield networks for covariance enrichment together with the InfoLOOB objective. In experiments we compare CLOOB to CLIP after pre-training on the Conceptual Captions and the YFCC dataset with respect to their zero-shot transfer learning performance on other datasets. CLOOB consistently outperforms CLIP at zero-shot transfer learning across all considered architectures and datasets.

• The paper presents CLOOB, which integrates modern Hopfield networks with the InfoLOOB objective to mitigate CLIP’s saturation issues.
• It demonstrates significant improvements in zero-shot transfer learning performance over CLIP across datasets like ImageNet and Birdsnap.
• The research offers promising insights on leveraging associative memory structures and non-saturating objectives for advanced multi-modal learning.

CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP

The paper under review presents "CLOOB," a novel approach that leverages modern Hopfield networks and the InfoLOOB objective to enhance the performance of contrastive learning models, specifically outperforming the well-known Contrastive Language-Image Pre-training (CLIP) methodology. The paper thoroughly investigates the limitations of CLIP and introduces CLOOB as a superior alternative in the context of zero-shot transfer learning.

CLIP has set a precedent as a foundational model in the field of multi-modal learning, excelling in zero-shot transfer tasks. It achieves this by harnessing contrastive learning to align image and text embeddings via the InfoNCE objective. However, the authors identify a fundamental challenge CLIP faces: the "explaining away" problem, which arises from improperly exploiting the covariance structure in multi-modal data, leading to an overemphasis on selected features while disregarding other pertinent attributes. Moreover, the InfoNCE objective is prone to saturation, restricting the learning capability of the model.

The paper's contribution is multilayered. Firstly, it integrates modern Hopfield networks to address the explaining away problem by enhancing the retrieval of embeddings through enriched covariance structures. Secondly, it introduces the InfoLOOB objective to counteract the saturation issue of InfoNCE. The paper posits that InfoLOOB does not saturate under high similarity conditions, thus enabling continuous learning by uniformly distributing samples. The combination of these strategies in CLOOB showed significantly improved results over CLIP across various datasets and architectures.

Key experimental outcomes illustrated the robustness of CLOOB. When trained on Conceptual Captions and YFCC100M datasets, CLOOB consistently outperformed CLIP in zero-shot transfer learning tasks across multiple datasets, such as ImageNet and Birdsnap. This robust empirical validation underscores the effectiveness of integrating advanced Hopfield networks with InfoLOOB in enhancing the model's capacity to learn rich, invariant representations.

Theoretical implications of this work suggest a compelling shift in how contrastive learning can be approached by leveraging associative memory structures, such as modern Hopfield networks, to better harness data covariance. This research opens avenues for further theoretical inquiries into non-saturating objectives in machine learning and encourages a reexamination of foundational contrastive learning concepts.

Future research could potentially explore the application of CLOOB in broader AI settings, including more complex modalities and multi-task learning environments. Additionally, while modern Hopfield Networks increase retrieval efficiency and capacity, further investigation into their scalability and computational impact is warranted to optimize their integration into large-scale AI model architectures.

In conclusion, this paper propounds a significant advancement in contrastive learning by introducing a methodology that systematically addresses prevalent issues in existing models. CLOOB not only enhances transfer learning performance but also contributes to the conceptual advancement in leveraging memory networks for enriched data representations.