A Review of Deep Transfer Learning and Recent Advancements (2201.09679v2)

Abstract: Deep learning has been the answer to many machine learning problems during the past two decades. However, it comes with two major constraints: dependency on extensive labeled data and training costs. Transfer learning in deep learning, known as Deep Transfer Learning (DTL), attempts to reduce such dependency and costs by reusing an obtained knowledge from a source data/task in training on a target data/task. Most applied DTL techniques are network/model-based approaches. These methods reduce the dependency of deep learning models on extensive training data and drastically decrease training costs. As a result, researchers detected Covid-19 infection on chest X-Rays with high accuracy at the beginning of the pandemic with minimal data using DTL techniques. Also, the training cost reduction makes DTL viable on edge devices with limited resources. Like any new advancement, DTL methods have their own limitations, and a successful transfer depends on some adjustments for different scenarios. In this paper, we review the definition and taxonomy of deep transfer learning and well-known methods. Then we investigate the DTL approaches by reviewing recent applied DTL techniques in the past five years. Further, we review some experimental analyses of DTLs to learn the best practice for applying DTL in different scenarios. Moreover, the limitations of DTLs (catastrophic forgetting dilemma and overly biased pre-trained models) are discussed, along with possible solutions and research trends.

• The paper presents a comprehensive review of deep transfer learning techniques that reuse pre-trained models to mitigate data limitations and reduce training expenses.
• It evaluates various approaches, including network-based and progressive methods, to enhance domain adaptation and preserve learned knowledge.
• The analysis identifies challenges such as catastrophic forgetting and recommends blending task-specific data to achieve continual learning.

Overview of Deep Transfer Learning and Recent Advancements

The paper "A Review of Deep Transfer Learning and Recent Advancements" by Mohammadreza Iman, Khaled Rasheed, and Hamid Reza Arabnia provides a comprehensive examination of deep transfer learning (DTL) techniques in the context of machine learning and artificial intelligence. The paper is structured to discuss the definition, taxonomy, empirical applications, and the limitations of DTL, as well as exploring its future potentials.

Transfer learning addresses two significant constraints inherent in deep learning: dependence on large quantities of labeled data and the high costs associated with training models. By leveraging already acquired knowledge from a source task and data set, DTL reduces the extensive data requirements and computational expense of training on a target task. This capability has proven beneficial in various applications such as detecting Covid-19 using limited chest X-ray data and deploying models on edge devices with limited computational resources.

Categorization and Approach

The paper delineates the DTL process as predominantly network or model-based. This categorization emphasizes altering pre-trained models to suit new tasks, most commonly by freezing, fine-tuning, or augmenting model layers. It introduces several methods alongside DTL's traditional approaches, such as feature and mapping-based, parameter and network-based, and relational and adversarial-based techniques. Notably, the paper identifies the network-based approach as being the most prevalent for its ability to address domain adaptations between mismatching source and target sets.

Empirical Insights

An extensive review categorizes a vast range of recent literature, showcasing applications across diverse data types and fields such as medical imaging and mechanics. Most methodologies reviewed fall into three primary categories: simple fine-tuning of entire pre-trained models, freezing of convolutional layers with subsequent full connection layer fine-tuning, and progressive learning, where new layers are trained while retaining pre-trained weights.

Limitations and Solutions

Despite DTL's efficacy, the paper acknowledges its limitations, citing the issues of "catastrophic forgetting" and the bias introduced by overly dependent pre-trained models. The phenomenon of catastrophic forgetting occurs when a model's pre-trained elements are overwritten during new task training, resulting in poor retention of original knowledge. Solutions discussed include strategic blending of task-specific data into training processes to preserve learned features.

The paper ventures into experimental analyses indicating successful DTL strategies and draws attention to progressive learning methods. Progressive Neural Networks (PNNs) emerge as a potential solution for overcoming traditional DTL constraints, facilitating more robust and resilient models through enhanced learning capacities without relinquishing prior learning.

Implications and Future Directions

The possibility of achieving continual learning through DTL could significantly advance artificial general intelligence, implying models that consistently learn and adapt without forfeiting previous knowledge across varying tasks. Future research directions may include enhancing model adaptability with reduced bias and improving the preservation of past learnings to support broader and more dynamic transfer applications.

In conclusion, this paper articulates a clear and established framework for navigating the landscape of DTL. It provides practical recommendations and future prospects for researchers eager to refine current methodologies and tackle existing limitations in the pursuit of more effective transfer learning strategies.