Stochastic Batch Acquisition: A Simple Baseline for Deep Active Learning (2106.12059v3)

Abstract: We examine a simple stochastic strategy for adapting well-known single-point acquisition functions to allow batch active learning. Unlike acquiring the top-K points from the pool set, score- or rank-based sampling takes into account that acquisition scores change as new data are acquired. This simple strategy for adapting standard single-sample acquisition strategies can even perform just as well as compute-intensive state-of-the-art batch acquisition functions, like BatchBALD or BADGE, while using orders of magnitude less compute. In addition to providing a practical option for machine learning practitioners, the surprising success of the proposed method in a wide range of experimental settings raises a difficult question for the field: when are these expensive batch acquisition methods pulling their weight?

• The paper introduces a stochastic batch acquisition method that delivers competitive accuracy with reduced computational cost compared to traditional methods.
• It adapts single-point acquisition functions to batch learning using stochastic sampling strategies including Softmax, Power, and Soft-Rank distributions.
• Empirical results on datasets like Repeated-MNIST, EMNIST, and Synbols verify its efficacy by matching or surpassing more computationally intensive approaches.

Stochastic Batch Acquisition: A Simple Baseline for Deep Active Learning

The paper examines a streamlined approach for adapting single-point acquisition functions to the context of batch active learning. This topic is significant, as active learning strategies aim to reduce labeling efforts in scenarios where unlabelled data are abundant but labeling is expensive, a common occurrence in fields such as medical imaging or complex scientific experiments.

Method Overview

The proposed methodology revolves around a concept called stochastic batch acquisition. Here, instead of the traditionally used top-$K$ acquisition strategy—where the top-scoring points based on a given criterion (e.g., BALD) are selected—the paper suggests adopting a stochastic sampling approach. This method accounts for the dynamic nature of acquisition scores as new data are assimilated into the model, leveraging a distributional sampling mechanism influenced by score perturbations.

Three types of stochastic acquisition distributions are explored:

• Softmax Distribution: Here, acquisition scores are perturbed using Gumbel noise, effectively sampling from a softmax distribution.
• Power Distribution: For scenarios where scores are non-negative, this approach uses power transformations of the scores.
• Soft-Rank Distribution: It solely depends on the rank order of the scores, providing a robust acquisition strategy less sensitive to score magnitudes.

Key Insights and Numerical Results

The empirical evaluation of these stochastic strategies reveals competitive performance against more computationally intensive methods such as BatchBALD and BADGE, particularly noteworthy due to its reduced computational requirements ($\mathcal{O}(M \log K)$) which scale efficiently even in large pool scenarios.

Key experiments conducted on datasets like Repeated-MNIST, EMNIST, and Synbols demonstrate that the stochastic acquisition strategies not only outperform traditional top-$K$ methods but in many cases match the accuracy and effectiveness of more complex approaches. For example, PowerBALD, one of the stochastic strategies, matches the performance of BADGE on various datasets while being computationally cheaper, providing practitioners with a viable option for real-world applications.

Implications and Future Developments

This paper challenges the effectiveness of existing batch acquisition methods by introducing a simplistic yet effective strategy that raises questions about the weight carrying of computationally exhaustive processes. The stochastic approach opens up avenues for further optimization, with potential investigations into score dynamics forecasting and dynamic acquisition sizes appearing as natural extensions.

The implications of such research are twofold: practical and theoretical. Practically, reducing computational costs while maintaining high effectiveness in active learning can significantly impact industries reliant on data annotation. Theoretically, it provokes deeper inquiry into the interactions and dependencies between acquisition scores as learning progresses.

In conclusion, the simplicity and performance parity achieved through stochastic batch acquisition functions should motivate researchers to revisit conventional batch acquisition techniques and consider more computationally adept strategies moving forward.