AdapterDrop: On the Efficiency of Adapters in Transformers (2010.11918v2)

Abstract: Massively pre-trained transformer models are computationally expensive to fine-tune, slow for inference, and have large storage requirements. Recent approaches tackle these shortcomings by training smaller models, dynamically reducing the model size, and by training light-weight adapters. In this paper, we propose AdapterDrop, removing adapters from lower transformer layers during training and inference, which incorporates concepts from all three directions. We show that AdapterDrop can dynamically reduce the computational overhead when performing inference over multiple tasks simultaneously, with minimal decrease in task performances. We further prune adapters from AdapterFusion, which improves the inference efficiency while maintaining the task performances entirely.

AdapterDrop: On the Efficiency of Adapters in Transformers

The paper "AdapterDrop: On the Efficiency of Adapters in Transformers" addresses crucial efficiency challenges innate to transformer-based models used in NLP tasks. Transformers, despite their efficacy, are resource-intensive, necessitating significant computational power, extended inference times, and large storage capacity. These factors have prompted research into optimizing transformer models, primarily by distilling smaller models, dynamically reducing model depth, and implementing lightweight adapters.

Adapters, introduced as an alternative to complete model fine-tuning, permit training of additional parameters at each layer, facilitating efficient transfer learning across tasks. While adapters excel in parameter efficiency, their computational efficiency, particularly concerning training and inference, remains underexplored. This paper proposes a novel method, AdapterDrop, to enhance the efficiency of adapters further by selectively omitting adapters in the lower layers during model training and inference phases.

Key Contributions and Findings

1. Efficiency Gains Without AdapterDrop: The paper establishes that adapters provide substantial training speed advantages compared to complete fine-tuning. Specifically, training using adapters can be up to 60% faster under typical hyperparameter configurations. However, these gains are partially offset during inference, where adapters are approximately 4-6% slower than fully fine-tuned models.
2. Introduction of AdapterDrop: AdapterDrop is proposed to dynamically enhance inference efficiency by selectively removing adapters from the lower layers of transformers. Initial implementations reveal that this method significantly reduces inference time with only minor performance degradation in multi-task settings. Notably, removing adapters from the initial five layers yields a 39% increase in inference speed when multitasking.
3. Integration with AdapterFusion: The authors extend AdapterDrop to AdapterFusion scenarios, typically utilized for leveraging knowledge across tasks. By pruning less significant adapters, the method maintains performance while enhancing efficiency. The paper documents that with AdapterFusion, it is possible to retain efficiency improvements while maintaining task accuracy, especially when the model operates with limited training data.

The paper also examines AdapterDrop's training process, revealing that it can be specialized for fixed layer configurations or made adaptive through robust training protocols, thus offering flexibility based on resource constraints.

Implications and Future Directions

The implications of AdapterDrop are significant for both theoretical exploration and practical deployments. Theoretically, the paper enriches our understanding of adapter-based architectures, laying the groundwork for further innovations in scalable transformer models. Practically, the findings suggest tangible efficiency improvements for real-world applications, especially where computational resources are a bottleneck.

Future research might explore the following:

• Broader Application Scope: AdapterDrop's principles could enhance other model architectures or AI applications beyond NLP, such as vision or multi-modal learning contexts.
• Further Optimization: There remains potential to explore even more refined algorithms for adapter efficiency and the dynamics of how many layers or adapters to drop adaptively concerning task requirements or system objectives.

In conclusion, the exploration of AdapterDrop introduces a promising pathway towards more efficient and versatile use of transformers in NLP, pointing toward a future of more sustainable and adaptable AI systems.