A Study of Acquisition Functions for Medical Imaging Deep Active Learning (2401.15721v2)

Abstract: The Deep Learning revolution has enabled groundbreaking achievements in recent years. From breast cancer detection to protein folding, deep learning algorithms have been at the core of very important advancements. However, these modern advancements are becoming more and more data-hungry, especially on labeled data whose availability is scarce: this is even more prevalent in the medical context. In this work, we show how active learning could be very effective in data scarcity situations, where obtaining labeled data (or annotation budget is very limited). We compare several selection criteria (BALD, MeanSTD, and MaxEntropy) on the ISIC 2016 dataset. We also explored the effect of acquired pool size on the model's performance. Our results suggest that uncertainty is useful to the Melanoma detection task, and confirms the hypotheses of the author of the paper of interest, that \textit{bald} performs on average better than other acquisition functions. Our extended analyses however revealed that all acquisition functions perform badly on the positive (cancerous) samples, suggesting exploitation of class unbalance, which could be crucial in real-world settings. We finish by suggesting future work directions that would be useful to improve this current work. The code of our implementation is open-sourced at \url{https://github.com/bonaventuredossou/ece526_course_project}

Introduction

In the domain of medical imaging, deep learning has made significant strides, yet the scarcity of labeled data often hinders the progress of machine learning algorithms. The paper by Bonaventure F.P. Dossou explores this very challenge applying Active Learning (AL) to medical imaging - specifically, melanoma detection from dermoscopic images using the ISIC 2016 dataset. The paper compares various acquisition functions - BALD, MeanSTD, MaxEntropy - in the context of deep active learning, assessing their effectiveness in model performance when labeled data is limited.

Acquisition Functions

Active Learning relies on the notion of iteratively selecting valuable samples from a pool of unlabeled data to be labeled and added to the training set. The premise is that by strategically querying specific samples that the model is most uncertain about, we optimize the learning process. Dossou's implementation tested three established acquisition functions: Maximum Entropy, which selects samples with the highest entropy across predictions; Mean Standard Deviation (MeanSTD), which looks for samples with the highest variance in the model's predictions; and BALD, focused on maximizing the mutual information between the model's predictions and its parameters. Each of these functions has the potential to selectively query new data points that could lead to significant improvements in a model, a principle particularly valuable in fields like medical imaging where data can be scarce and expensive to annotate.

Experiment and Methodology

The ISIC 2016 dataset, composed of 900 training images and 350 testing images, was employed in a binary classification task to determine the presence of melanoma. The approaches were benchmarked against a Bayesian Convolutional Neural Network (CNN), modified to account for uncertainty through Monte Carlo Dropout. The network's architecture was kept constant across experiments to isolate the effects of the different acquisition functions. Experiments evaluated model performance based on loss and accuracy after consecutive rounds of active learning, measuring the utility of uncertain data sampling.

Results and Discussion

The experimental outcomes were indicative of several crucial observations. Unsurprisingly, model uncertainty was found to be advantageous for melanoma detection. Among the acquisition functions, BALD outperformed MeanSTD and MaxEntropy, confirming hypotheses from previous literature that prioritize informational gain about model parameters. The acquisition functions' effectiveness varied when the least uncertain samples were acquired, with BALD again exhibiting superior performance. Interestingly, the paper also discovered that the test set performance didn't dramatically change with different query sizes, suggesting a potential bottleneck in model capacity or an inherent difficulty in machine learning with extremely small datasets.

This paper's conclusions serve as a crucial waypoint for future research. It clearly established the importance of acquisition functions in AL for medical imaging tasks, with BALD demonstrating particularly promising results. Future direction could explore more sophisticated models, larger datasets, or novel acquisition functions capable of handling data imbalance more effectively. Such research avenues could provide the key to unlocking even more potent AL strategies in the field of medical imaging, maximizing both efficiency and model performance.