Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks (2312.02366v4)

Abstract: The integration of deep learning systems into healthcare has been hindered by the resource-intensive process of data annotation and the inability of these systems to generalize to different data distributions. Foundation models, which are models pre-trained on large datasets, have emerged as a solution to reduce reliance on annotated data and enhance model generalizability and robustness. DINOv2 is an open-source foundation model pre-trained with self-supervised learning on 142 million curated natural images that exhibits promising capabilities across various vision tasks. Nevertheless, a critical question remains unanswered regarding DINOv2's adaptability to radiological imaging, and whether its features are sufficiently general to benefit radiology image analysis. Therefore, this study comprehensively evaluates the performance DINOv2 for radiology, conducting over 200 evaluations across diverse modalities (X-ray, CT, and MRI). To measure the effectiveness and generalizability of DINOv2's feature representations, we analyze the model across medical image analysis tasks including disease classification and organ segmentation on both 2D and 3D images, and under different settings like kNN, few-shot learning, linear-probing, end-to-end fine-tuning, and parameter-efficient fine-tuning. Comparative analyses with established supervised, self-supervised, and weakly-supervised models reveal DINOv2's superior performance and cross-task generalizability. The findings contribute insights to potential avenues for optimizing pre-training strategies for medical imaging and enhancing the broader understanding of DINOv2's role in bridging the gap between natural and radiological image analysis. Our code is available at https://github.com/MohammedSB/DINOv2ForRadiology

The integration of AI into medical imaging has been advancing steadily, and a notable progression in this field is the use of foundation models pre-trained on large datasets. These models aim to reduce the necessity for extensive annotated data while enhancing the adaptability of AI systems across various data distributions, which is a significant issue in the medical field due to privacy concerns and the resource-intensive nature of data annotation.

This experimental paper focuses on assessing the viability of DINOv2—a state-of-the-art foundation model originally trained with self-supervised learning on an extensive dataset of natural images—for medical image analysis. The model's potential for generalization was put to the test through over 100 experiments involving diverse radiological image types including X-ray, CT scans, and MRI imagery covering tasks such as disease classification and organ segmentation. These tasks were evaluated in different contexts: k-nearest neighbors, few-shot learning, linear probing, end-to-end fine-tuning, and parameter-efficient tuning, to gauge the effectiveness of DINOv2 embeddings.

The comparative analyses included well-established medical image analysis models like the U-Net and the TransUnet for segmentation tasks and other convolutional neural network (CNN) models and transformer models such as the Vision Transformer (ViT) for classification tasks trained with different learning paradigms. The performance metrics gave DINOv2 an edge in segmentation and competitive results in classification tasks, bringing to light its possibility to close the gap between analyzing natural images and those obtained from radiological procedures.

The paper's findings not only underscore DINOv2's robust performance across medical image analysis benchmarks but also suggest potential optimization of pre-training strategies specific to medical imaging. Furthermore, practical applications such as few-shot learning demonstrate the model's efficiency in scenarios with limited data, a common challenge in the medical domain. The use of parameter-efficient fine-tuning strategies is also shown to be competitive with traditional full model fine-tuning, yet it requires tuning significantly fewer parameters.

In addition to numerical results, qualitative analysis using Principal Component Analysis (PCA) visualizations provide insights into the adaptability of DINOv2 features from natural to medical images, showing promising signs of domain transfer. Despite the foundation model's training on non-medical images, its feature representations matched distinct medial imaging tasks effectively. The results of this comprehensive analysis pave the way for future research to augment foundation model pre-training with medical data for potentially even more robust and reliable AI diagnostic tools. This could herald a significant advancement in creating general-purpose, scalable models for medical image analysis, a critical step towards the more widespread adoption of AI in healthcare.