The Impact of Scanner Domain Shift on Deep Learning Performance in Medical Imaging: an Experimental Study (2409.04368v2)

Abstract: Purpose: Medical images acquired using different scanners and protocols can differ substantially in their appearance. This phenomenon, scanner domain shift, can result in a drop in the performance of deep neural networks which are trained on data acquired by one scanner and tested on another. This significant practical issue is well-acknowledged, however, no systematic study of the issue is available across different modalities and diagnostic tasks. Materials and Methods: In this paper, we present a broad experimental study evaluating the impact of scanner domain shift on convolutional neural network performance for different automated diagnostic tasks. We evaluate this phenomenon in common radiological modalities, including X-ray, CT, and MRI. Results: We find that network performance on data from a different scanner is almost always worse than on same-scanner data, and we quantify the degree of performance drop across different datasets. Notably, we find that this drop is most severe for MRI, moderate for X-ray, and quite small for CT, on average, which we attribute to the standardized nature of CT acquisition systems which is not present in MRI or X-ray. We also study how injecting varying amounts of target domain data into the training set, as well as adding noise to the training data, helps with generalization. Conclusion: Our results provide extensive experimental evidence and quantification of the extent of performance drop caused by scanner domain shift in deep learning across different modalities, with the goal of guiding the future development of robust deep learning models for medical image analysis.

• The paper quantifies the impact of scanner domain shifts on deep learning models, highlighting a -0.097 AUC drop for MRI images.
• It uses seven datasets across different imaging modalities to compare diagnostic performance between same-domain and cross-domain evaluations.
• The study underscores the need for domain adaptation techniques to enhance the robustness of clinical deep learning applications.

An Experimental Study on Scanner Domain Shift in Medical Imaging

The paper "The Impact of Scanner Domain Shift on Deep Learning Performance in Medical Imaging: an Experimental Study" provides a comprehensive analysis of the influence of scanner domain shift on the performance of convolutional neural networks (CNNs) in medical image analysis. In medical imaging, domain shifts arise when radiological images are acquired using distinct scanners with differing image acquisition parameters, resulting in variations in the images despite their perceptual similarities. This work quantifies the impact of such domain shifts across different modalities including MRI, CT, and X-ray, elucidating their effect on the diagnostic performance of deep learning models.

Methodology and Modality-Specific Observations

The researchers conducted their paper using seven publicly available datasets, each covering different body regions and imaging modalities. They assessed various diagnostic tasks, such as tumor detection and staging, across multiple scanner domains. For each dataset, images were categorized into two domains based on the scanners from which they were obtained. Models were trained on images from one domain and tested on both the same and the alternative domain. Performance was primarily evaluated using the area under the receiver operating characteristic curve (AUC).

Crucially, the paper identified that domain shifts significantly affect model performance, with the severities differing markedly across modalities. Specifically, MRI was found to be the most affected by domain shifts, exhibiting a substantial performance degradation with an average AUC drop of -0.097 when tested on data from a different scanner domain. In contrast, CT imaging demonstrated minimal susceptibility to domain shifts, with an average AUC discrepancy of only -0.02, possibly attributable to the more standardized nature of CT image acquisition. The impact on X-ray data was intermediate.

Implications for Deep Learning in Clinical Settings

These findings have significant implications for the deployment of deep learning models in clinical practice. The variability caused by scanner domain shifts can lead to a decline in the robustness and reliability of automated diagnostic systems, particularly in modalities like MRI, which are heavily dependent on site-specific acquisition parameters. This underscores the necessity for domain adaptation techniques and more generalizable feature learning in constructing CNNs for medical imaging tasks.

Future Directions

The exploration of strategies to mitigate domain shift effects is paramount. Potential future work includes the incorporation of more sophisticated domain adaptation techniques, such as adversarial training or domain-specific augmentation, to bolster model generalization across varying imaging pipelines. Additionally, further research could include an in-depth examination of the underlying features learned by models from different domains to elucidate specific characteristics that lead to domain-specific biases.

The paper sets a precedent for further investigation into domain shifts within medical imaging. Ensuring the development of robust deep learning models necessitates acknowledging and addressing these shifts, especially as these models transition from research to clinical environments where scanner heterogeneity is a critical factor. By providing empirical evidence on the extent of performance drops due to scanner domain shift, this work paves the way for developing more universally applicable AI models in medical diagnostics.