A Comprehensive Survey on Transfer Learning (1911.02685v3)

Abstract: Transfer learning aims at improving the performance of target learners on target domains by transferring the knowledge contained in different but related source domains. In this way, the dependence on a large number of target domain data can be reduced for constructing target learners. Due to the wide application prospects, transfer learning has become a popular and promising area in machine learning. Although there are already some valuable and impressive surveys on transfer learning, these surveys introduce approaches in a relatively isolated way and lack the recent advances in transfer learning. Due to the rapid expansion of the transfer learning area, it is both necessary and challenging to comprehensively review the relevant studies. This survey attempts to connect and systematize the existing transfer learning researches, as well as to summarize and interpret the mechanisms and the strategies of transfer learning in a comprehensive way, which may help readers have a better understanding of the current research status and ideas. Unlike previous surveys, this survey paper reviews more than forty representative transfer learning approaches, especially homogeneous transfer learning approaches, from the perspectives of data and model. The applications of transfer learning are also briefly introduced. In order to show the performance of different transfer learning models, over twenty representative transfer learning models are used for experiments. The models are performed on three different datasets, i.e., Amazon Reviews, Reuters-21578, and Office-31. And the experimental results demonstrate the importance of selecting appropriate transfer learning models for different applications in practice.

• The paper categorizes transfer learning into instance, feature, parameter, and relational methods to clarify how each approach transfers domain knowledge.
• The paper empirically evaluates 20+ transfer learning models on datasets like Amazon Reviews, Reuters-21578, and Office-31 to benchmark performance.
• The paper highlights future directions such as mitigating negative transfer and improving interpretability to further advance transfer learning applications.

A Comprehensive Survey on Transfer Learning

Overview of the Paper

The paper "A Comprehensive Survey on Transfer Learning" by Fuzhen Zhuang et al. presents an extensive review of transfer learning, an essential machine learning paradigm aimed at transferring knowledge across different domains to improve the performance of target learners. This survey addresses the rapid expansion and the vast prospects of transfer learning, systematically compiling over forty representative approaches, particularly focusing on homogeneous transfer learning strategies. The paper not only introduces various transfer learning methods but also provides empirical evaluations to demonstrate their effectiveness.

Key Contributions

1. Categorization and Mechanisms: The authors categorize transfer learning approaches into instance-based, feature-based, parameter-based, and relational-based methods. The survey explores the mechanisms behind these methods, particularly emphasizing data and model perspectives.
2. Empirical Evaluation: Over twenty representative transfer learning models are empirically evaluated across three different datasets: Amazon Reviews, Reuters-21578, and Office-31, highlighting the importance of selecting appropriate models for different applications.

Data-Based Transfer Learning

Data-based transfer learning approaches focus on leveraging data transformations and adaptations to mitigate the distribution differences between source and target domains. The main strategies discussed include:

• Instance Weighting: Algorithms like Kernel Mean Matching (KMM) and TrAdaBoost assign weights to source-domain instances to align their distributions with the target domain, thereby reducing the marginal distribution discrepancy.
• Feature Transformation: This strategy involves operations like feature augmentation, feature reduction, and feature alignment. Techniques such as Transfer Component Analysis (TCA), Joint Distribution Adaptation (JDA), and various autoencoder-based methods (e.g., SDA, mSLDA) are detailed, showcasing how feature extraction and representation learning can bridge domain gaps.

Model-Based Transfer Learning

Model-based approaches transfer knowledge at the model level, often involving parameter sharing or restriction, and ensemble methods:

• Parameter Sharing and Restriction: Methods like Fast-DAM and Multi-Model Knowledge Transfer (MMKL) leverage parameter sharing and regularization to adapt source models for target tasks effectively. Matrix factorization techniques, such as MTrick and TriTL, are also discussed for leveraging latent feature spaces.
• Model Ensemble: Approaches like TaskTrAdaBoost and Locally Weighted Ensemble (LWE) combine multiple models to improve target domain predictions, with mechanisms for weighting and optimizing the ensemble process.

Deep Learning in Transfer Learning

The paper highlights the significant role of deep learning in transfer learning:

• Traditional Deep Learning: Architectures like Deep Adaptation Networks (DAN) and Marginalized Stacked Linear Denoising Autoencoder (mSLDA) demonstrate how deep networks can learn domain-invariant features through adaptation layers and regularization.
• Adversarial Deep Learning: Inspired by Generative Adversarial Networks (GANs), Domain-Adversarial Neural Network (DANN) and Conditional Domain Adversarial Network (CDAN) showcase how adversarial objectives can be utilized to learn features that confuse domain classifiers, thereby achieving domain invariance.

Empirical Evaluation

The empirical evaluations conducted on Amazon Reviews, Reuters-21578, and Office-31 provide insights into the performance of various transfer learning models. Key observations include:

• Domain Suitability: Some algorithms like HIDC, SFA, and MTrick demonstrate consistent performance across different domains, while others exhibit significant variability depending on domain characteristics.
• Effectiveness of Deep Learning: Deep learning-based methods, particularly adversarial approaches, achieve superior performance on object recognition tasks, highlighting their ability to handle complex domain shifts effectively.

Implications and Future Directions

The paper underscores the practical importance of transfer learning in reducing data dependencies and enhancing model performance across diverse applications. Future research directions suggested include:

1. Broader Application Range: Expanding the application of transfer learning techniques to new and complex real-world scenarios.
2. Negative Transfer: Systematic analysis and mitigation of negative transfer to enhance model robustness.
3. Model Interpretability: Investigating the interpretability of transfer learning models to improve their transparency and trustworthiness.
4. Theoretical Development: Providing robust theoretical foundations to justify and support empirical findings in transfer learning.

Conclusion

This survey provides a comprehensive and detailed examination of transfer learning, its methodologies, applications, and empirical evaluations. By systematizing existing research and highlighting future directions, it serves as a valuable resource for researchers and practitioners aiming to leverage transfer learning in various domains. The empirical results emphasize the necessity of choosing appropriate models based on specific application requirements, ensuring effective knowledge transfer and improved learning performance.