Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? (1706.00712v1)

Abstract: Training a deep convolutional neural network (CNN) from scratch is difficult because it requires a large amount of labeled training data and a great deal of expertise to ensure proper convergence. A promising alternative is to fine-tune a CNN that has been pre-trained using, for instance, a large set of labeled natural images. However, the substantial differences between natural and medical images may advise against such knowledge transfer. In this paper, we seek to answer the following central question in the context of medical image analysis: \emph{Can the use of pre-trained deep CNNs with sufficient fine-tuning eliminate the need for training a deep CNN from scratch?} To address this question, we considered 4 distinct medical imaging applications in 3 specialties (radiology, cardiology, and gastroenterology) involving classification, detection, and segmentation from 3 different imaging modalities, and investigated how the performance of deep CNNs trained from scratch compared with the pre-trained CNNs fine-tuned in a layer-wise manner. Our experiments consistently demonstrated that (1) the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch; (2) fine-tuned CNNs were more robust to the size of training sets than CNNs trained from scratch; (3) neither shallow tuning nor deep tuning was the optimal choice for a particular application; and (4) our layer-wise fine-tuning scheme could offer a practical way to reach the best performance for the application at hand based on the amount of available data.

• The paper demonstrates that layer-wise fine-tuning of pre-trained CNNs significantly improves performance with limited annotated data.
• It reveals that fine-tuned CNNs outperform those trained from scratch and traditional methods in applications like polyp and pulmonary embolism detection.
• The study underscores transfer learning as an effective strategy to enhance accuracy and reduce the computational demands of medical image analysis.

Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?

In the paper titled "Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?" by Nima Tajbakhsh, Jae Y. Shin, et al., the authors rigorously investigate whether pre-trained convolutional neural networks (CNNs), when fine-tuned, can match or even surpass the performance of CNNs trained from scratch for various medical imaging tasks. This examination is especially crucial given the limited availability of annotated medical imaging data and the high computational demands of training deep networks from scratch.

Key Contributions

1. Layer-Wise Fine-Tuning: The authors present a comprehensive approach for incrementally fine-tuning a pre-trained CNN in a layer-wise manner. This methodology contrasts with training a new classifier on features extracted from pre-trained networks or fine-tuning the entire network, as discussed in previous literature.
2. Training Data Robustness: The paper demonstrates that fine-tuned CNNs are more robust to the size of training sets compared to CNNs trained from scratch. This robustness is particularly significant in the medical domain, where obtaining a large volume of labeled data is challenging.
3. Comparison with Handcrafted Features: Detailed comparisons are made between the performance of fine-tuned CNNs, CNNs trained from scratch, and handcrafted feature-based methods. These comparisons are performed across multiple medical imaging applications, including polyp detection, pulmonary embolism (PE) detection, colonoscopy frame classification, and intima-media segmentation.

Experimental Framework

The paper evaluates the effectiveness of fine-tuning pre-trained CNNs on four distinct medical imaging tasks:

1. Polyp Detection in Colonoscopy Videos:
   • The paper uses a handcrafted approach alongside CNNs trained from scratch and fine-tuned CNNs.
   • Fine-tuning CNNs demonstrate superior performance, particularly with reduced training data, underscoring the advantages of transfer learning in resource-constrained domains.
2. Pulmonary Embolism Detection in CT Images:
   • By using a custom image representation for candidate PEs, the paper leverages CNNs to outperform traditional handcrafted methods in most scenarios, particularly as data size decreases.
3. Colonoscopy Frame Classification:
   • The assessment of frame quality yields significant insights into application-specific tuning. Intermediate level fine-tuning results in the highest performance, suggesting a thoughtful consideration of which layers to tune based on task similarity to the source domain.
4. Intima-Media Boundary Segmentation in Ultrasound Images:
   • Fine-tuned deep networks deliver lower segmentation errors compared to CNNs trained from scratch and to traditional methods, demonstrating the capabilities of CNNs in complex segmentation problems.

Numerical Results

The paper provides elaborate numerical results, highlighted through various performance curves and statistical analyses. Fine-tuning exhibits consistent performance improvement in most cases. For instance, in polyp detection, a CNN fine-tuned up to its deepest layers significantly outperformed a CNN built from the ground up, especially at lower false positive rates. Similarly, for PE detection, fine-tuned networks achieved higher sensitivity across different false positives per volume.

Implications and Future Directions

1. Transfer Learning in Medical Imaging: The findings confirm that transfer learning, through fine-tuning, is not only feasible but advantageous for medical imaging tasks. This approach mitigates the limitations imposed by scarce labeled data and reduces the computational burden associated with training deep CNNs from scratch.
2. Application-Specific Fine-Tuning: The depth of fine-tuning necessary varies across applications. Tasks more dissimilar to the pre-training domain might need deeper fine-tuning, highlighting the importance of analyzing the specific requirements of each medical imaging task.
3. Future Research: Future work could explore the application of fine-tuning in other domains such as MRI and histopathology, potentially revealing broader generalizability. Additionally, research advancing unsupervised pre-training methods could provide supplementary avenues for transfer learning where labeled data is even scarcer.

Conclusion

The paper effectively addresses its central question, demonstrating that pre-trained CNNs, with sufficient fine-tuning, can indeed eliminate the need for training deep CNNs from scratch for medical image analysis. The highlighted robustness to training data size and the superior convergence speed further establish fine-tuning as a practical and effective strategy for deploying deep learning in medical imaging. The demonstrated performance improvements across a range of tasks underscore the potential of transfer learning to empower medical image analysis, enhancing both the accuracy and efficiency of diagnostic systems.