Guarding Barlow Twins Against Overfitting with Mixed Samples

Abstract: Self-supervised Learning (SSL) aims to learn transferable feature representations for downstream applications without relying on labeled data. The Barlow Twins algorithm, renowned for its widespread adoption and straightforward implementation compared to its counterparts like contrastive learning methods, minimizes feature redundancy while maximizing invariance to common corruptions. Optimizing for the above objective forces the network to learn useful representations, while avoiding noisy or constant features, resulting in improved downstream task performance with limited adaptation. Despite Barlow Twins' proven effectiveness in pre-training, the underlying SSL objective can inadvertently cause feature overfitting due to the lack of strong interaction between the samples unlike the contrastive learning approaches. From our experiments, we observe that optimizing for the Barlow Twins objective doesn't necessarily guarantee sustained improvements in representation quality beyond a certain pre-training phase, and can potentially degrade downstream performance on some datasets. To address this challenge, we introduce Mixed Barlow Twins, which aims to improve sample interaction during Barlow Twins training via linearly interpolated samples. This results in an additional regularization term to the original Barlow Twins objective, assuming linear interpolation in the input space translates to linearly interpolated features in the feature space. Pre-training with this regularization effectively mitigates feature overfitting and further enhances the downstream performance on CIFAR-10, CIFAR-100, TinyImageNet, STL-10, and ImageNet datasets. The code and checkpoints are available at: https://github.com/wgcban/mix-bt.git

• The paper introduces Mixed Barlow Twins, adding mixup regularization to the original Barlow Twins to counter overfitting in self-supervised learning.
• It employs linear interpolation of input images to create mixed samples that enhance the quality of learned representations.
• Empirical results on datasets like CIFAR-10 and ImageNet demonstrate improved generalization and significant performance gains over the baseline.

Introduction to Self-Supervised Learning

Self-Supervised Learning (SSL) has made significant strides in teaching AI systems to understand and process data without pre-labeled datasets. SSL algorithms like Barlow Twins have proven effective in tasks such as image recognition, where they learn by minimizing the similarity or redundancy between features extracted from different views of the same data. However, the efficiency of SSL methods can be compromised by a phenomenon known as overfitting, where the model becomes too tailored to the training data, hurting its ability to generalize to new, unseen data.

The Overfitting Challenge in Barlow Twins

Barlow Twins stands out for its simplicity and ability to learn informative representations for a variety of applications. Nonetheless, this study reveals that the Barlow Twins, powerful as they are, have an intrinsic tendency to overfit—especially when the size of the feature representations, or embeddings, is increased. This overfitting can diminish the quality of the representations and ultimately degrade the model's performance on tasks outside the training dataset.

Introducing Mixed Barlow Twins

To combat overfitting in the Barlow Twins, the study proposes a novel extension—named Mixed Barlow Twins—which incorporates an additional term in the algorithm. This term is based on a technique called MixUp regularization, widely used in supervised learning. MixUp creates new training samples through linear interpolation (combining) of input images. By mixing samples during training, the method assumes that similar mixes should happen in the embedding space, thus adding an extra regularization that encourages the model to generalize better.

Empirical Evaluation and Results

The paper extensively tests the Mixed Barlow Twins algorithm across several datasets, including CIFAR-10, CIFAR-100, TinyImageNet, STL-10, and ImageNet, using rigorous k-NN evaluation and linear classification metrics. The Mixed Barlow Twins consistently show improved resistance to overfitting and higher quality representations compared to the original Barlow Twins algorithm. When mixed samples are introduced into the SSL process, it complicates the model's ability to memorize training samples, thereby fostering better generalization and offering a significant performance boost for downstream tasks.

Conclusion and Potential

The findings of this study suggest that the introduction of mixup-based regularization, as implemented in the Mixed Barlow Twins algorithm, is an effective strategy to enhance the learning quality and robustness of SSL methods. This modification not only preserves the original advantages of the Barlow Twins approach but also extends its application potential by improving the transferability and generalization of the learned features.