MixCo: Mix-up Contrastive Learning for Visual Representation (2010.06300v2)

Abstract: Contrastive learning has shown remarkable results in recent self-supervised approaches for visual representation. By learning to contrast positive pairs' representation from the corresponding negatives pairs, one can train good visual representations without human annotations. This paper proposes Mix-up Contrast (MixCo), which extends the contrastive learning concept to semi-positives encoded from the mix-up of positive and negative images. MixCo aims to learn the relative similarity of representations, reflecting how much the mixed images have the original positives. We validate the efficacy of MixCo when applied to the recent self-supervised learning algorithms under the standard linear evaluation protocol on TinyImageNet, CIFAR10, and CIFAR100. In the experiments, MixCo consistently improves test accuracy. Remarkably, the improvement is more significant when the learning capacity (e.g., model size) is limited, suggesting that MixCo might be more useful in real-world scenarios. The code is available at: https://github.com/Lee-Gihun/MixCo-Mixup-Contrast.

• The paper presents MixCo, which integrates mix-up based semi-positive samples into contrastive learning to boost visual representation quality.
• It employs convex combinations of image pairs to generate nuanced training samples that overcome limitations of traditional instance discrimination.
• Experimental results show up to a 6.84% accuracy improvement over baselines like MoCo-v2 and SimCLR, highlighting its efficiency under resource constraints.

MixCo: Mix-up Contrastive Learning for Visual Representation

This paper presents MixCo, an innovative extension of contrastive learning methods in self-supervised visual representation learning. The introduction of semi-positive sample concepts through image mixing is the core premise of this approach. Unlike traditional contrastive learning techniques that exclusively focus on distinguishing positive pairs from negative pairs, MixCo introduces a nuanced approach by incorporating semi-positive samples obtained through a mix-up of images.

Methodology and Contributions

Contrastive learning, a fundamental component of self-supervised learning, learns visual representations by contrasting positive pairs—augmented versions of the same image—against negative pairs comprising different images. This technique underpins significant recent achievements in unsupervised visual representation learning. However, it is highly sensitive to augmentation strategies and efficacy depends on the number of negative samples available, which can be computationally costly.

MixCo extends this strategy by employing mix-up training principles, marrying images to interpolate between them. This generates mixed data from convex combinations of positives and negatives to produce semi-positive pairs. MixCo exploits these semi-positive samples by measuring and learning relative similarities between the representations resultant from this mix-up. Consequently, learning from semi-positive samples circumvents the limitations imposed by instance-level discrimination inherent in traditional methods, enhancing the learned representations' robustness and transferability.

The authors validate MixCo's effectiveness by applying it to well-known frameworks—MoCo-v2 and SimCLR—and conducting experiments on datasets such as TinyImageNet, CIFAR10, and CIFAR100. Results consistently showcase improved accuracy in linear evaluation tasks, particularly under constrained learning capacities or limited resources. For instance, the paper reveals a considerable improvement in test accuracy, up to 6.84%, indicating enhanced representation quality when computational resources are limited.

Experimental Evaluation

The experiment section provides quantitative analysis, presenting detailed comparisons of linear evaluation results across various datasets with different architectural setups. MixCo's adeptness in capturing semantic relationships within data is visually represented through t-SNE clustering, demonstrating superior performance over baseline models. Furthermore, computational costs associated with using MixCo are addressed, presenting evidence of its advantageous memory and time efficiency compared to existing methods.

Implications and Future Outlook

MixCo's introduction presents practical implications for real-world applications, particularly in scenarios where computing resources are limited, and efficiency is paramount. By optimizing the use of available training samples, MixCo enhances the learning process without necessitating additional data or excessive computational power.

The theoretical implications urge a reassessment of the capabilities and applications of contrastive learning methods in visual representation learning. The ability of semi-positive samples to facilitate more effective discrimination and nuanced learning offers a promising direction for developing more efficient and scalable self-supervised learning strategies.

Future developments may explore broader applications of MixCo and refine its algorithmic components to improve generalization across a wider array of tasks and datasets. Extensions could also investigate integrating MixCo with newer architectures or combining it with other augmentations to enrich learned representations further.

In conclusion, the paper presents MixCo as a robust, efficient method for enhancing self-supervised contrastive learning, opening various avenues for future research and practical applications in visual representation learning.