Contrastive Learning with Boosted Memorization (2205.12693v6)

Abstract: Self-supervised learning has achieved a great success in the representation learning of visual and textual data. However, the current methods are mainly validated on the well-curated datasets, which do not exhibit the real-world long-tailed distribution. Recent attempts to consider self-supervised long-tailed learning are made by rebalancing in the loss perspective or the model perspective, resembling the paradigms in the supervised long-tailed learning. Nevertheless, without the aid of labels, these explorations have not shown the expected significant promise due to the limitation in tail sample discovery or the heuristic structure design. Different from previous works, we explore this direction from an alternative perspective, i.e., the data perspective, and propose a novel Boosted Contrastive Learning (BCL) method. Specifically, BCL leverages the memorization effect of deep neural networks to automatically drive the information discrepancy of the sample views in contrastive learning, which is more efficient to enhance the long-tailed learning in the label-unaware context. Extensive experiments on a range of benchmark datasets demonstrate the effectiveness of BCL over several state-of-the-art methods. Our code is available at https://github.com/MediaBrain-SJTU/BCL.

• The paper introduces Boosted Contrastive Learning (BCL) that leverages the memorization effect to enhance tail sample representations in long-tailed datasets.
• It proposes a novel momentum loss mechanism that tracks temporal training losses to dynamically adjust data augmentations without explicit labels.
• Experimental results on CIFAR-100-LT, ImageNet-LT, and Places-LT demonstrate that BCL outperforms traditional contrastive learning methods across head, medium, and tail partitions.

Contrastive Learning with Boosted Memorization: An Overview

The self-supervised learning paradigm has made significant advances in visual and textual representation learning. Despite these successes, the prevalent approaches are typically validated on datasets like ImageNet, which are balanced and do not resemble real-world data distributions that often follow a long-tailed pattern. In such scenarios, self-supervised models have struggled to exhibit the expected performance levels.

This paper introduces a novel approach named Boosted Contrastive Learning (BCL) that seeks to address the challenge of learning from long-tailed distributions in a label-unaware context. Unlike previous methods that focus on model architecture adjustments or loss reweighting strategies, BCL emphasizes the data perspective by leveraging the memorization effect intrinsic to deep neural networks (DNNs). It automatically boosts the representation learning of tail samples through differential augmentation techniques driven by historical data characteristics.

Key Concepts and Methodology

1. Memorization Effect: BCL capitalizes on the memorization effect where DNNs inherently learn easy (head) patterns before hard (tail) patterns. This attribute is used to dynamically delineate head from tail samples without explicit labels by analyzing the historical training losses.
2. Momentum Loss: The paper proposes a novel momentum loss mechanism designed to encapsulate the temporal loss statistics of training samples. This mechanism assists in identifying tail samples by maintaining the moving average of loss values, providing a robust framework for differentiating among data without explicit labels.
3. Data Augmentation: To enhance learning on tail samples, BCL employs a boosted data augmentation strategy. The augmentation strength and strategy are modulated based on the momentum loss, thus delivering stronger augmentations to samples identified as likely tail samples. This strategy is aligned with the "InfoMin Principle," which posits that effective augmentations maximize task-relevant information while reducing redundancy.
4. Dynamic View Discrepancy: BCL enhances contrastive learning by dynamically controlling the information discrepancy across augmented views. By doing this, the approach can maintain high intra-class similarity while maximizing inter-class differentiability, particularly for tail samples, effectively boosting their representational fidelity.

Experimental Outcomes

Several benchmark datasets, including CIFAR-100-LT, ImageNet-LT, and Places-LT, are employed to evaluate the effectiveness of BCL. The experiments demonstrate that BCL significantly outperforms traditional contrastive learning methods and recent long-tailed learning strategies. Notably, the BCL framework leads to marked improvements across head, medium, and tail partitions, indicating comprehensive performance gains across the data distribution spectrum.

Implications and Future Directions

BCL's approach has robust implications both in theory and application. The introduction of a data-centric view in contrastive learning refreshes the methodology for handling long-tailed distributions without relying on explicit labels. This positions BCL as a promising technique for real-world scenarios where data is plentiful but labels are scarce or costly to acquire.

Future avenues for research could involve exploring BCL's adaptability to other self-supervised learning frameworks and its applications to diverse domains beyond image datasets, such as text or cross-modal applications. Additionally, further refinement of the momentum loss's sensitivity and the incorporation of adaptive methods to modulate augmentation strategies continuously could provide a fertile ground for enhancing model convergence and integrity.

In conclusion, this paper lays foundational work for advancing self-supervised learning in contextually complex data distributions. By harnessing innate model behaviors like memorization, it offers a pragmatic path towards effective representation learning in less-than-ideal data conditions.