Transferability in Deep Learning: A Survey (2201.05867v1)

Abstract: The success of deep learning algorithms generally depends on large-scale data, while humans appear to have inherent ability of knowledge transfer, by recognizing and applying relevant knowledge from previous learning experiences when encountering and solving unseen tasks. Such an ability to acquire and reuse knowledge is known as transferability in deep learning. It has formed the long-term quest towards making deep learning as data-efficient as human learning, and has been motivating fruitful design of more powerful deep learning algorithms. We present this survey to connect different isolated areas in deep learning with their relation to transferability, and to provide a unified and complete view to investigating transferability through the whole lifecycle of deep learning. The survey elaborates the fundamental goals and challenges in parallel with the core principles and methods, covering recent cornerstones in deep architectures, pre-training, task adaptation and domain adaptation. This highlights unanswered questions on the appropriate objectives for learning transferable knowledge and for adapting the knowledge to new tasks and domains, avoiding catastrophic forgetting and negative transfer. Finally, we implement a benchmark and an open-source library, enabling a fair evaluation of deep learning methods in terms of transferability.

• The paper's main contribution is a comprehensive survey unifying diverse approaches across pre-training, task adaptation, and domain adaptation in deep learning.
• It details how model architecture, including ResNet and Transformers, and techniques like self-supervised and contrastive learning influence transferability.
• It highlights strategies to mitigate catastrophic forgetting through fine-tuning and domain adaptation, providing benchmarks for improved cross-domain performance.

Essay on "Transferability in Deep Learning: A Survey"

The surveyed paper, "Transferability in Deep Learning: A Survey," provides an extensive examination of the concept of transferability in deep learning frameworks. Transferability, a notion simulating the human ability to apply learned knowledge across different tasks, has become a pivotal area within machine learning aimed at enhancing data efficiency.

Overview

The paper bridges isolated domains in deep learning, offering a comprehensive framework that addresses transferability throughout the deep learning lifecycle, which includes stages of pre-training, adaptation, and evaluation. It explores various components, such as deep architectures, pre-training paradigms, task, and domain adaptation.

Pre-Training Paradigms

The success of pre-training hinges significantly on model architecture. Techniques like ResNet and Transformer demonstrate how model depth and inductive bias, respectively, influence transferability. Supervised pre-training, common in vision with ImageNet, as well as emerging unsupervised paradigms, such as Generative and Contrastive Learning, are detailed. These latter approaches highlight the role of data augmentation and self-supervised tasks in fostering generic model flexibility.

Task Adaptation

Fine-tuning methods, a cornerstone of task adaptation, seek to mitigate issues like catastrophic forgetting and negative transfer. Domain adaptive tuning and regularization techniques provide pathways to maintain pre-trained knowledge. Recent innovations in parameter-efficient adaptations, such as adapter modules and prompt learning, address growing model size concerns and broaden applicability to low-data scenarios.

Domain Adaptation

Unsupervised Domain Adaptation (UDA) extends the applicability of models across different domains without labeled data, primarily using adversarial training and statistical matching. Techniques are rooted in theoretical frameworks like $\mathcal H\Delta \mathcal H$-Divergence, aiming to minimize cross-domain disparities. The paper elaborates on key algorithms that operationalize these theories, detailing the architectural strategies which enable domain adversarial learning.

Evaluation and Datasets

The survey emphasizes the necessity of large-scale evaluation to test cross-task and cross-domain transferability thoroughly. It lists prominent datasets across NLP and vision designed to scrutinize model performance and adaptability under varying data conditions, demonstrating practical implications.

Implications and Future Directions

This survey underscores the notion that the ecosystem of deep learning thrives on developing models that mimic the human-like transfer of knowledge. Practically, enhancing transferability contributes to more efficient learning systems adaptable across diverse, previously unseen tasks and domains. Theorists and practitioners are encouraged to investigate continued refinement in pre-training designs, adaptation strategies, and robust benchmarks for consistent evaluation.

In conclusion, "Transferability in Deep Learning: A Survey" serves as a critical resource, mapping the landscape of transferability across the deep learning lifecycle. Its comprehensive treatment of the subject aims to unify fragmented research avenues, providing a foundation for building more versatile and data-efficient artificial intelligence systems. The paper is an essential reference for researchers seeking to expand the boundaries of what deep learning can achieve in complex, dynamic environments.