AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2205.13535v3)

Abstract: Pretraining Vision Transformers (ViTs) has achieved great success in visual recognition. A following scenario is to adapt a ViT to various image and video recognition tasks. The adaptation is challenging because of heavy computation and memory storage. Each model needs an independent and complete finetuning process to adapt to different tasks, which limits its transferability to different visual domains. To address this challenge, we propose an effective adaptation approach for Transformer, namely AdaptFormer, which can adapt the pre-trained ViTs into many different image and video tasks efficiently. It possesses several benefits more appealing than prior arts. Firstly, AdaptFormer introduces lightweight modules that only add less than 2% extra parameters to a ViT, while it is able to increase the ViT's transferability without updating its original pre-trained parameters, significantly outperforming the existing 100\% fully fine-tuned models on action recognition benchmarks. Secondly, it can be plug-and-play in different Transformers and scalable to many visual tasks. Thirdly, extensive experiments on five image and video datasets show that AdaptFormer largely improves ViTs in the target domains. For example, when updating just 1.5% extra parameters, it achieves about 10% and 19% relative improvement compared to the fully fine-tuned models on Something-Something~v2 and HMDB51, respectively. Code is available at https://github.com/ShoufaChen/AdaptFormer.

AdaptFormer: A New Approach to Adapt Vision Transformers for Efficient Visual Recognition

Overview

AdaptFormer introduces an efficient adaptation mechanism for pre-trained Vision Transformers (ViTs) to extend their applicability across a diverse range of image and video recognition tasks. This method focuses on enhancing the model's transferability while retaining computational efficiency by implementing lightweight modules, specifically designed to add minimal parameters to the existing architecture. Highlighted by its distinguished performance and scalability, AdaptFormer signifies a remarkable step towards universal representation and extensively demonstrates its effectiveness through rigorous evaluations on various datasets.

Adaptation Challenge in Vision Transformers

The adaptation of pre-trained Vision Transformers across multiple domains has traditionally necessitated full model fine-tuning, leading to substantial computational demands and storage requirements. This process entails a significant update to the model's parameters, which not only increases the risk of catastrophic interference but also restricts the model's scalability and flexibility when dealt with numerous tasks. Recent literature suggests a shift towards developing a unified model architecture with almost identical weights to enable seamless transferability. However, achieving superior performance with minimal parameter tuning remains an unresolved challenge.

Introducing AdaptFormer

AdaptFormer emerges as a solution to this limitation by proposing an adaptable framework that maintains most of the pre-trained model parameters unchanged while introducing a novel AdaptMLP module. This module, constituting less than 2\% of the overall model parameters, effectively enhances the model's adaptability across diverse visual tasks without significant updates to the pre-existing weight structure. The key aspects of AdaptFormer include:

• Minimal Parameter Addition: By inserting lightweight modules, AdaptFormer introduces a negligible increase in parameters, ensuring computational efficiency.
• Scalable to Various Tasks: AdaptFormer demonstrates impressive scalability, significantly improving performance in video and image recognition tasks with a mere 1.5\% increase in extra parameters.
• Superior Performance: AdaptFormer not only matches but in certain cases, surpasses the performance of fully fine-tuned models across recognized benchmarks, including action recognition datasets like Something-Something v2 and HMDB51.

Technical Insights

• AdaptFormer integrates the AdaptMLP module in parallel with the transformer's original feed-forward network, striking a balance between the transfer of learned representations and the adoption of task-specific features without a substantial parameter increase.
• The architecture of AdaptFormer elegantly combines unchanged pre-trained model components with adaptable modules through a straightforward yet effective mechanism, leveraging the robustness of pre-trained representations while facilitating task-specific adaptions.

Experimental Evaluation

Extensive experiments validate the effectiveness of AdaptFormer across five major datasets spanning images and videos. Notably, AdaptFormer significantly outperforms existing adaptation methods with remarkably fewer parameters — a testament to its efficiency and potential in real-world applications. For instance, when dealing with action recognition tasks, AdaptFormer achieves a relative improvement of approximately 10\% and 19\% over fully fine-tuned models on the Something-Something v2 and HMDB51 benchmarks, respectively, with only a fraction of the tunable parameters.

Future Directions

AdaptFormer's demonstrated efficiency and scalability encourage future exploration into further optimizing the mechanism for universal representation. Its success prompts inquiry into potential applications beyond the scope of visual recognition, possibly extending to other domains where large-scale models seek efficiency in adaptation processes.

Conclusion

AdaptFormer represents a significant advance in the fine-tuning of pre-trained Vision Transformers for scalable visual recognition tasks. By effectively bridging the gap between computational efficiency and model performance, this framework sets a new benchmark for future developments in the adaptation of large-scale models across diverse tasks and domains.