Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey (1902.06162v1)

Abstract: Large-scale labeled data are generally required to train deep neural networks in order to obtain better performance in visual feature learning from images or videos for computer vision applications. To avoid extensive cost of collecting and annotating large-scale datasets, as a subset of unsupervised learning methods, self-supervised learning methods are proposed to learn general image and video features from large-scale unlabeled data without using any human-annotated labels. This paper provides an extensive review of deep learning-based self-supervised general visual feature learning methods from images or videos. First, the motivation, general pipeline, and terminologies of this field are described. Then the common deep neural network architectures that used for self-supervised learning are summarized. Next, the main components and evaluation metrics of self-supervised learning methods are reviewed followed by the commonly used image and video datasets and the existing self-supervised visual feature learning methods. Finally, quantitative performance comparisons of the reviewed methods on benchmark datasets are summarized and discussed for both image and video feature learning. At last, this paper is concluded and lists a set of promising future directions for self-supervised visual feature learning.

An Overview of "Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey"

The paper provides a comprehensive review of self-supervised learning (SSL) methods focused on visual feature learning using deep neural networks, particularly convolutional neural networks (ConvNets). SSL has gained traction due to its potential to overcome the limitations associated with the manual annotation of large-scale datasets.

Core Motivation and Scope

Traditional supervised learning methods for visual tasks like object detection, semantic segmentation, and image captioning rely heavily on extensive labeled data, which is often labor-intensive and expensive to collect. As a subset of unsupervised learning, SSL aims to address this by leveraging large amounts of unlabeled data to learn meaningful visual representations without requiring human-annotated labels.

Key Components Reviewed

Deep Neural Architectures

The paper categorizes common ConvNet architectures for image and video feature learning:

• For Image Features:
  • AlexNet – A pioneering architecture that redefined object classification tasks.
  • VGG – Notable for its simplicity and use of small convolutional filters.
  • GoogLeNet – Introduced the inception module, allowing for a mix of various convolutional filter sizes.
  • ResNet – Known for its skip connections which mitigate the vanishing gradient problem.
  • DenseNet – Used dense blocks where each layer is connected to every other layer in a feed-forward manner.
• For Video Features:
  • Two-Stream Networks – Use separate networks for spatial and temporal features.
  • 3DConvNets (e.g., C3D) – Extend 2DConvNets into the temporal domain.
  • Recurrent Neural Networks (RNNs) and LSTMs – Capture long-term dependencies in video sequences.

Pretext Tasks

The pretext tasks are creatively designed objectives that leverage intrinsic data properties to generate self-supervised labels. These tasks can be categorized based on their learning schemas:

• Generation-Based Methods:
  • Utilizes tasks such as image colorization, inpainting, and super-resolution.
  • Techniques like GANs (Generative Adversarial Networks) play a pivotal role here.
• Context-Based Methods:
  • Harness spatial and temporal context to formulate self-supervised tasks.
  • Examples include predicting the relative position of image patches (spatial) and order of video frames (temporal).
• Free Semantic Label-Based Methods:
  • Leverage data attributes to generate pseudo-labels.
  • Examples include segmentation masks and depth maps derived from synthetic data generated by game engines or traditional methods.
• Cross-Modal Methods:
  • Use relationships between different data modalities.
  • For example, Visual-Audio correspondence tasks cross-validate visual frames with audio signals.

Datasets

The review covers a range of datasets for training and evaluating SSL methods:

• Image datasets like ImageNet and Places for visual feature learning.
• Video datasets such as UCF101 and Kinetics, useful for spatiotemporal feature learning.

Performance Evaluation

The quality of learned visual features is evaluated through downstream tasks like image classification, object detection, and action recognition. Quantitative analysis on benchmark datasets, such as ImageNet and PASCAL VOC, reveals that SSL methods often approach or even paralleled the performance of fully supervised models.

Implications and Future Directions

The paper outlines several prospective avenues for future research:

• Utilizing Synthetic Data: Game engines can generate extensive datasets with rich annotations for SSL.
• Leveraging Web Data: Using web-crawled images and videos, along with their associated metadata, can significantly reduce reliance on annotated datasets.
• Spatiotemporal Feature Learning: Addressing the complexity of video data to enhance spatiotemporal representation.
• Multi-Modal Learning: Expanding SSL methods to incorporate sensory data from various sources like LIDAR and IMUs.
• Multi-Pretext Task Learning: Combining multiple pretext tasks could lead to more robust and generalizable visual representations.

Conclusion

The surveyed SSL techniques showcase a robust alternative to supervised learning by making efficient use of unlabeled data. With advancements in multitask pretext designs, the development of synthetic datasets, and the employment of web-scale data, SSL holds a promising future for enriching deep learning applications in computer vision. The performance gains reflect the potential of SSL methods to bridge the gap with traditional supervised approaches, marking an evolutionary step in visual feature learning.