Models Genesis (2004.07882v4)

Abstract: Transfer learning from natural images to medical images has been established as one of the most practical paradigms in deep learning for medical image analysis. To fit this paradigm, however, 3D imaging tasks in the most prominent imaging modalities (e.g., CT and MRI) have to be reformulated and solved in 2D, losing rich 3D anatomical information, thereby inevitably compromising its performance. To overcome this limitation, we have built a set of models, called Generic Autodidactic Models, nicknamed Models Genesis, because they are created ex nihilo (with no manual labeling), self-taught (learnt by self-supervision), and generic (served as source models for generating application-specific target models). Our extensive experiments demonstrate that our Models Genesis significantly outperform learning from scratch and existing pre-trained 3D models in all five target 3D applications covering both segmentation and classification. More importantly, learning a model from scratch simply in 3D may not necessarily yield performance better than transfer learning from ImageNet in 2D, but our Models Genesis consistently top any 2D/2.5D approaches including fine-tuning the models pre-trained from ImageNet as well as fine-tuning the 2D versions of our Models Genesis, confirming the importance of 3D anatomical information and significance of Models Genesis for 3D medical imaging. This performance is attributed to our unified self-supervised learning framework, built on a simple yet powerful observation: the sophisticated and recurrent anatomy in medical images can serve as strong yet free supervision signals for deep models to learn common anatomical representation automatically via self-supervision. As open science, all codes and pre-trained Models Genesis are available at https://github.com/MrGiovanni/ModelsGenesis.

Overview of "Models Genesis"

The paper "Models Genesis" presents a novel approach to improving 3D medical image analysis through self-supervised learning. The authors introduce Generic Autodidactic Models, known as Models Genesis, which are designed to leverage unlabeled 3D imaging data without relying on traditional 2D transfer learning from natural images. This research addresses the limitations of reformulating 3D tasks into 2D ones, thereby preserving essential 3D anatomical information.

Key Contributions

1. Self-Supervised Learning Framework: The paper proposes a self-supervised learning framework that utilizes 3D medical images to train models without manual labeling. This framework is characterized by:
   • Non-linear Transformation: Enhances the ability to learn intensity distributions.
   • Local-Shuffling Transformation: Focuses on learning the texture and boundaries of anatomic structures.
   • Local and Global Context Learning: Employs outer-cutout and inner-cutout techniques to learn spatial layout and continuity of anatomical features.
2. Evaluation Across Applications: Models Genesis were tested on various tasks, including lung nodule detection and brain tumor segmentation, demonstrating superior performance compared to models trained from scratch and pre-trained supervised models.
3. Reduced Annotation Effort: The approach significantly reduces the need for annotated data, proving especially beneficial for underrepresented medical conditions.

Results and Implications

• Enhanced Performance: Models Genesis consistently outperform traditional 2D approaches and even some existing 3D pre-trained models like MedicalNet and I3D in specific tasks.
• 3D Context Utilization: By maintaining the spatial context of 3D medical images, the proposed models show a clear advantage over conventional 2D models.
• Generalizability and Transferability: The framework demonstrated the potential for both same-domain and cross-domain transferability, making it adaptable to various organs, diseases, and imaging modalities.

Future Directions

• Creation of a Medical ImageNet: While effective, the paper suggests that a labeled comprehensive dataset like ImageNet for medical images could further advance model performance.
• Cross-domain Learning: Future research could focus on enhancing cross-domain capabilities, allowing models to generalize more effectively across different medical imaging modalities and conditions.

Overall, the paper provides a thorough exploration of self-supervised learning within 3D medical imaging, highlighting its practical applications and potential for broader adoption in medical diagnosis and research. This work sets the stage for further investigation into harnessing unlabeled data to create robust and adaptable medical imaging models.