Self-supervised Learning: Generative or Contrastive (2006.08218v5)

Abstract: Deep supervised learning has achieved great success in the last decade. However, its deficiencies of dependence on manual labels and vulnerability to attacks have driven people to explore a better solution. As an alternative, self-supervised learning attracts many researchers for its soaring performance on representation learning in the last several years. Self-supervised representation learning leverages input data itself as supervision and benefits almost all types of downstream tasks. In this survey, we take a look into new self-supervised learning methods for representation in computer vision, natural language processing, and graph learning. We comprehensively review the existing empirical methods and summarize them into three main categories according to their objectives: generative, contrastive, and generative-contrastive (adversarial). We further investigate related theoretical analysis work to provide deeper thoughts on how self-supervised learning works. Finally, we briefly discuss open problems and future directions for self-supervised learning. An outline slide for the survey is provided.

Self-supervised Learning: Generative or Contrastive

The paper under review, authored by Xiao Liu et al., provides a comprehensive survey of self-supervised learning (SSL) methods for representation in computer vision, NLP, and graph learning. The core focus is on categorizing these methods into generative, contrastive, and generative-contrastive (adversarial) techniques and highlighting their theoretical underpinnings, empirical performances, and practical applications.

Introduction and Motivation

Deep supervised learning has achieved extensive success, particularly in the domains of computer vision, NLP, and graph learning. However, the reliance on extensive labeled data and susceptibility to adversarial attacks has propelled researchers toward exploring SSL. SSL aims to harness the abundant unlabeled data and proposes alternative supervisory signals intrinsic within the data itself. This paradigm shift facilitates data-efficient learning and improved generalization, making SSL a formidable contender in the field of representation learning.

Categorization of SSL Methods

The paper meticulously classifies SSL methods into three primary categories based on their training objectives:

1. Generative Models: Focus on reconstructing input data from latent representations. Representative models include Variational Autoencoders (VAE) and Generative Adversarial Networks (GAN).
2. Contrastive Models: Emphasize distinguishing between similar and dissimilar instance pairs via contrastive loss. Examples include Deep InfoMax and Contrastive Predictive Coding (CPC).
3. Generative-Contrastive Models: Integrate generative reconstruction with discriminative objectives. A notable approach within this category is Adversarial Autoencoders (AAE).

Generative Models

Generative models aim to model the data distribution and include techniques like Auto-regressive (AR) models, flow-based models, and auto-encoding (AE) models. Key methodologies include:

• AR Models: Sequentially predict elements (e.g., words, pixels) conditioned on preceding elements, exemplified by GPT-2 and PixelCNN.
• Flow-based Models: Model complex densities via a series of invertible transformations, with notable instances being NICE and RealNVP.
• AE Models: Encode inputs into latent vectors and subsequently decode them back to the original inputs. Techniques like VAE and its variant VQ-VAE-2 highlight the capability of AE models in producing high-fidelity reconstructions.

Contrastive Models

Contrastive models leverage instance discrimination through Noise Contrastive Estimation (NCE). The goal is to maximize mutual information between different views or parts of an input or between an input and its context. Prominent approaches include:

• Deep InfoMax: Maximizes mutual information between local features and their global contexts.
• MoCo and SimCLR: Emphasize augmenting positive pairs and leveraging large batches of negative samples to improve representation quality.

Generative-Contrastive Models

Generative-contrastive models, or adversarial models, utilize a discriminative loss function in conjunction with generative models. They are particularly successful in image generation, transformation, and manipulation. Key methods include:

• Adversarial Autoencoders (AAE): Integrate the benefits of VAE and GAN to produce meaningful latent representations.
• BiGAN and ALI: Extend GANs by explicitly learning an encoder to map inputs to the latent space, thereby enabling both generation and representation learning.

Implications and Future Directions

The survey underscores the transformative potential of SSL across various domains. Key implications include:

• Enhanced Data Efficiency: SSL reduces the dependency on labeled datasets, making learning from vast amounts of unlabeled data feasible.
• Improved Generalization: Models trained using SSL often exhibit superior generalization capabilities across diverse tasks.
• Broader Applicability: The principles of SSL extend beyond computer vision and NLP, demonstrating potential in fields like graph learning and beyond.

Challenges and Open Problems

Several challenges persist in the field of SSL, necessitating further research:

• Theoretical Foundations: A deeper theoretical understanding of SSL mechanisms is required to elucidate why certain methods outperform others.
• Task Transferability: Bridging the gap between pre-training objectives and downstream tasks remains a critical challenge.
• Sampling Efficiency: Optimizing sampling strategies, particularly for negative samples in contrastive learning, warrants ongoing investigation.

Conclusion

This paper provides a detailed examination of SSL methods, categorizing them into generative, contrastive, and generative-contrastive classes while discussing their theoretical and empirical contributions. The findings highlight the evolving landscape of SSL and its profound implications for future developments in AI. Continued advancements in this field promise to unlock new levels of efficiency and efficacy in representation learning, underscoring the importance of ongoing research and innovation.