Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis (1908.06912v1)

Abstract: Transfer learning from natural image to medical image has established as one of the most practical paradigms in deep learning for medical image analysis. However, to fit this paradigm, 3D imaging tasks in the most prominent imaging modalities (e.g., CT and MRI) have to be reformulated and solved in 2D, losing rich 3D anatomical information and inevitably compromising the performance. To overcome this limitation, we have built a set of models, called Generic Autodidactic Models, nicknamed Models Genesis, because they are created ex nihilo (with no manual labeling), self-taught (learned by self-supervision), and generic (served as source models for generating application-specific target models). Our extensive experiments demonstrate that our Models Genesis significantly outperform learning from scratch in all five target 3D applications covering both segmentation and classification. More importantly, learning a model from scratch simply in 3D may not necessarily yield performance better than transfer learning from ImageNet in 2D, but our Models Genesis consistently top any 2D approaches including fine-tuning the models pre-trained from ImageNet as well as fine-tuning the 2D versions of our Models Genesis, confirming the importance of 3D anatomical information and significance of our Models Genesis for 3D medical imaging. This performance is attributed to our unified self-supervised learning framework, built on a simple yet powerful observation: the sophisticated yet recurrent anatomy in medical images can serve as strong supervision signals for deep models to learn common anatomical representation automatically via self-supervision. As open science, all pre-trained Models Genesis are available at https://github.com/MrGiovanni/ModelsGenesis.

Overview of "Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis"

The paper presents a self-supervised learning framework named "Models Genesis" for 3D medical image analysis. This framework aims to learn general-purpose visual representations through a series of transformations and subsequent restorations on medical images, thereby facilitating transfer learning across different medical imaging tasks.

Methodology

The core component of Models Genesis is the self-supervised learning framework which involves two key processes: image transformation and image restoration. The transformations devised include non-linear intensity transformation, local pixel shuffling, out-painting, and in-painting. Each transformation is designed to enable the model to learn various facets of medical images, such as the intensity, geometry, spatial layout, and texture of organs. These transformations are applied to patches of medical images, and the model learns representations by restoring the original patches from these transformed ones.

A distinguishing feature of the proposed framework is the unified self-supervised learning scheme. This scheme randomly employs any combination of the individual transformations to enrich the learning process, thereby creating a robust visual representation. As evident from the results, the unified framework outperforms individual transformation-based training, indicating the benefit of learning from augmented tasks.

Experimental Validation

Models Genesis were evaluated on several 3D medical imaging tasks, including segmentation and classification. The performance metrics indicated that Models Genesis outperforms traditional approaches, such as models trained from scratch or those pre-trained on ImageNet, specifically in 3D settings. For instance, Models Genesis achieved superior results on tasks like lung nodule characterization, outperforming both 2D-based solutions and the existing pre-trained networks like NiftyNet.

Implications and Future Work

The research establishes that pre-trained models in 3D medical imaging enhance task performance significantly. This eliminates the need for large annotated datasets which are a bottleneck in medical image analysis. Models Genesis demonstrates robust transferability across different domains, including cross-modality transfers such as from CT to MRI, suggesting that rich representation learning occurs within this framework.

Practically, the framework could expedite the development of annotated datasets by providing initial segmentation models requiring fewer expert annotations. It is suggested that while Models Genesis serves as an intermediate solution, there remains a demand for extensive annotated medical image datasets akin to ImageNet in computer vision.

In the future, fine-tuning of Genesis models on modality and organ-specific datasets is a promising area for continued research, as initial findings suggest that same-domain transfer learning remains preferable. Furthermore, exploring the full potential of cross-domain learning with Models Genesis could alleviate image acquisition constraints in clinical environments.

In conclusion, this work on Models Genesis provides substantial evidence for the effectiveness of self-supervised learning in medical image analysis by capitalizing on domain-specific transformations, thereby paving the way toward more efficient and effective 3D medical imaging applications.