SiT: Self-supervised vIsion Transformer (2104.03602v3)

Abstract: Self-supervised learning methods are gaining increasing traction in computer vision due to their recent success in reducing the gap with supervised learning. In NLP self-supervised learning and transformers are already the methods of choice. The recent literature suggests that the transformers are becoming increasingly popular also in computer vision. So far, the vision transformers have been shown to work well when pretrained either using a large scale supervised data or with some kind of co-supervision, e.g. in terms of teacher network. These supervised pretrained vision transformers achieve very good results in downstream tasks with minimal changes. In this work we investigate the merits of self-supervised learning for pretraining image/vision transformers and then using them for downstream classification tasks. We propose Self-supervised vIsion Transformers (SiT) and discuss several self-supervised training mechanisms to obtain a pretext model. The architectural flexibility of SiT allows us to use it as an autoencoder and work with multiple self-supervised tasks seamlessly. We show that a pretrained SiT can be finetuned for a downstream classification task on small scale datasets, consisting of a few thousand images rather than several millions. The proposed approach is evaluated on standard datasets using common protocols. The results demonstrate the strength of the transformers and their suitability for self-supervised learning. We outperformed existing self-supervised learning methods by large margin. We also observed that SiT is good for few shot learning and also showed that it is learning useful representation by simply training a linear classifier on top of the learned features from SiT. Pretraining, finetuning, and evaluation codes will be available under: https://github.com/Sara-Ahmed/SiT.

• The paper introduces SiT, presenting a novel GMML framework that pretrains vision transformers in a self-supervised manner akin to BERT in NLP.
• It seamlessly integrates masked autoencoding with contrastive learning to reduce dependency on large labeled datasets.
• Empirical results show that SiT outperforms supervised pretraining on small to medium datasets, expanding its applicability in resource-limited scenarios.

Analysis of "SiT: Self-supervised vIsion Transformer"

The paper "SiT: Self-supervised vIsion Transformer," authored by Sara Atito, Muhammad Awais, and Josef Kittler, presents a novel approach to self-supervised learning for vision transformers. While self-supervised learning (SSL) has already been established as a powerful method in the field of NLP, its application and effectiveness in the domain of vision transformers have remained underexplored. The researchers address this gap by introducing Self-supervised vIsion Transformer (SiT), which leverages a unique Group Masked Model Learning (GMML) framework.

Overview of SiT

SiT represents a pioneering effort to employ self-supervised pretraining (SSP) for vision transformers, demonstrating it as a superior alternative to supervised pretraining (SP) for various downstream computer vision tasks. The cornerstone of SiT is the GMML framework, a masked autoencoder strategy designed to efficiently learn representations by reconstructing masked portions of images. This method extends the principles of masked LLMs from NLP to vision tasks, adapting the architecture of vision transformers for self-supervised tasks without external supervision or extensive datasets.

Key Contributions and Methodology

1. GMML Framework: SiT introduces GMML as a self-supervised strategy specifically tailored for vision transformers. This framework employs masked autoencoding, where parts of an image are predicted based on their surrounding context, closely paralleling the principles of BERT in NLP.
2. Architecture: The flexibility of vision transformers allows SiT to adapt to multiple self-supervised tasks. SiT integrates contrastive learning techniques alongside GMML, harnessing a multi-task learning paradigm that effectively balances reconstruction and contrastive learning objectives.
3. Performance: SiT consistently demonstrates that SSL, when applied to vision transformers, not only closes the performance gap with SP but often surpasses it. The robust nature of SiT is particularly evident on small and medium-scale datasets, where data constraints traditionally hinder the efficacy of transformer models.
4. Reduced Data Dependency: SiT significantly alleviates the data hunger issue of vision transformers by achieving superior performance on limited data without leveraging external labeled datasets or teacher networks for guidance.

Empirical Evaluation

The researchers conducted comprehensive evaluations on several standard image classification benchmarks, as well as multi-label and video segmentation tasks. SiT's ability to achieve state-of-the-art results on smaller datasets, where transformers typically struggle, highlights its potential in democratizing AI research by reducing dependency on large datasets and high computational resources.

Implications and Future Directions

The introduction of SiT marks a significant advancement in SSL for vision transformers, providing a framework that others in the field might leverage and expand. The architectural and methodological innovations presented pave the way for broader applications of transformers across various vision tasks even in resource-limited scenarios.

The promising results achieved by SiT invite further exploration in adapting its principles to other domains such as medical imaging, video analysis, and real-time applications, where data labeling is particularly burdensome. Future research may explore integrating SiT with advancements from NLP like contextual embeddings to further enhance model's understanding of visual contexts.

SiT also raises interesting questions regarding the potential for synergistic integration with other SSL strategies and deep learning architectures, which could yield even more potent models for general-purpose vision tasks. The continued evolution of SiT could ultimately bridge the divide between supervised and self-supervised learning in vision, aligning it more closely with its NLP counterpart, and empowering a wider range of applications through improved self-supervised methodologies.