Disentangling the Effects of Data Augmentation and Format Transform in Self-Supervised Learning of Image Representations (2312.02205v1)

Abstract: Self-Supervised Learning (SSL) enables training performant models using limited labeled data. One of the pillars underlying vision SSL is the use of data augmentations/perturbations of the input which do not significantly alter its semantic content. For audio and other temporal signals, augmentations are commonly used alongside format transforms such as Fourier transforms or wavelet transforms. Unlike augmentations, format transforms do not change the information contained in the data; rather, they express the same information in different coordinates. In this paper, we study the effects of format transforms and augmentations both separately and together on vision SSL. We define augmentations in frequency space called Fourier Domain Augmentations (FDA) and show that training SSL models on a combination of these and image augmentations can improve the downstream classification accuracy by up to 1.3% on ImageNet-1K. We also show improvements against SSL baselines in few-shot and transfer learning setups using FDA. Surprisingly, we also observe that format transforms can improve the quality of learned representations even without augmentations; however, the combination of the two techniques yields better quality.

• The paper introduces Fourier Domain Augmentations (FDA) as a novel method that complements traditional pixel-based augmentations in self-supervised learning.
• The study demonstrates that integrating FDA with standard techniques yields around a 1% improvement in ImageNet classification accuracy.
• The research reveals that combining dual-view training with frequency and image representations enhances model robustness for transfer and few-shot learning tasks.

In the field of AI, one of the significant challenges is teaching AI models to understand and interpret visual data. This is where Self-Supervised Learning (SSL) comes into play, especially when we have limited labeled data available. SSL largely relies on data augmentation techniques that introduce variability in training data, thereby helping models to generalize better to unseen data.

However, the common practice has been to focus primarily on transformations directly made on image pixels, such as random cropping, color adjustments, and flips. But what about transformations not in the image domain but in the frequency domain, where the image is represented in terms of its constituent frequencies?

A paper tackles this overlooked aspect head-on by dissecting the role of frequency-domain augmentations in SSL. The research introduces a novel set of methods known as Fourier Domain Augmentations (FDA) that apply data augmentation in the frequency space. This approach complements the classical image-space augmentations and has shown to significantly enhance the performance of image representation learning.

FDA involves applying a set of transformations, such as amplitude scaling, phase shifting, and frequency masking, directly to the image's frequency components. These augmentations are responsible for introducing changes that are not easily replicated by standard image augmentations—effects like unique texture modifications and alterations in color distribution that help in learning more robust image representations.

The research provides evidence that integrating FDA with existing augmentation techniques consistently improves performance across several SSL frameworks, including SimCLR, BYOL, MoCov2, and SimSiam. For instance, on the ImageNet-1K dataset, models pre-trained with FDA achieved around a 1% increase in classification accuracy.

Interestingly, the paper also explores the sole influence of the format transform—using the frequency representation of the image alongside its raw form. The paper demonstrates that presenting the model with these two different views of the same data during pre-training can lead to more informative representations, though the best results were achieved when frequency-based and image-based augmentations were combined.

In application to downstream tasks like transfer learning and few-shot learning setups, where SSL pre-trained models are fine-tuned with minimal data, FDA once again proves beneficial. It not only enhances model adaptability to new domains and tasks but also improves qualitative measures such as image retrieval performance.

The findings prompt further questions in the field, one of them being how to utilize the frequency domain more effectively without the need to transform back into the image space. Moreover, given that the research primarily deals with real images, it opens up avenues for exploring the applicability of FDA to other domains like medical imaging.

In conclusion, the paper suggests that considering both image-domain and frequency-domain augmentations could be a vital step in advancing the capabilities of AI in tasks involving visual perception. The insight drawn from the research underlines the importance of creating diverse and comprehensive training environments for self-supervised models to garner the necessary robustness and flexibility for practical implementations.