Active learning with biased non-response to label requests (2312.08150v2)

Abstract: Active learning can improve the efficiency of training prediction models by identifying the most informative new labels to acquire. However, non-response to label requests can impact active learning's effectiveness in real-world contexts. We conceptualise this degradation by considering the type of non-response present in the data, demonstrating that biased non-response is particularly detrimental to model performance. We argue that biased non-response is likely in contexts where the labelling process, by nature, relies on user interactions. To mitigate the impact of biased non-response, we propose a cost-based correction to the sampling strategy--the Upper Confidence Bound of the Expected Utility (UCB-EU)--that can, plausibly, be applied to any active learning algorithm. Through experiments, we demonstrate that our method successfully reduces the harm from labelling non-response in many settings. However, we also characterise settings where the non-response bias in the annotations remains detrimental under UCB-EU for specific sampling methods and data generating processes. Finally, we evaluate our method on a real-world dataset from an e-commerce platform. We show that UCB-EU yields substantial performance improvements to conversion models that are trained on clicked impressions. Most generally, this research serves to both better conceptualise the interplay between types of non-response and model improvements via active learning, and to provide a practical, easy-to-implement correction that mitigates model degradation.

• The paper presents the UCB-EU method, integrating response probability into sampling strategies to counteract biased non-response.
• It reveals how non-response effects—both volume and imbalance—can substantially hinder model training if left unmanaged.
• Empirical results, including an e-commerce case study, demonstrate that addressing bias improves post-click model accuracy.

Introduction

Active Learning (AL) is an area of research in machine learning that focuses on improving the learning model by selectively acquiring new data points that are most useful to the model's learning process. Conventionally, AL is premised on the assumption that every request for the label of an unlabeled example will be met with a response—something that doesn't always align with real-world scenarios. Non-responses can be prevalent, especially when the data labeling process involves human interaction or is derived from user interactions with a platform, leading to potential biases that undermine AL's efficiency.

This paper addresses the challenges posed by biased non-response in the active learning context. The authors propose a method to adjust AL processes to account for potential non-response, presenting both theoretical analysis and empirical validations, including a case paper using a real-world e-commerce dataset.

Impact of Non-Response

It is argued that biased non-response can significantly impair a model's performance. Two kinds of impacts are theorized: a "volume effect," where the number of training samples is reduced, and an "imbalance effect," where the training examples' distribution becomes skewed. These effects can create a feedback loop that either stalls or deteriorates model improvement over time.

The paper introduces two forms of non-responses: Missing Completely at Random (MCAR), where non-response is distributed uniformly and does not relate to the characteristics of the data, and Missing at Random (MAR), where non-response is associated with observed features in the data. Dealing with MAR situations proves more challenging, particularly when high-value information coincides with high non-response rates, presenting significant hurdles for AL effectiveness.

Methodology for Addressing Non-Response

The authors propose an algorithmic correction termed the Upper Confidence Bound of the Expected Utility (UCB-EU), which adjusts the sampling strategy to consider the estimated probability of obtaining a response. This approach is designed to be compatible with various AL querying strategies and integrates an element of the estimated probability of label acquisition to avoid allocating resources to regions of the dataset where non-responses dominate.

Experimental Validation

The proposed method is subjected to experiments using synthetic data scenarios and a case paper involving the Taobao e-commerce platform. The simulations demonstrate that the UCB-EU correction can improve AL model performance, particularly in contexts where non-response is prevalent and biased. In the Taobao case paper, integrating UCB-EU leads to significant improvements in the post-click model, effectively handling the non-response biases generated by user interactions.

Conclusion

The presence of biased non-response in active learning can substantially impact the effectiveness of model improvement efforts. The UCB-EU correction presented in this paper provides a practical solution to this problem, facilitating more reliable and efficient model training in contexts where non-response biases are likely to occur. By merging the principles of expected utility with the realities of data acquisition challenges, this research bridges a significant gap between theoretical assumptions in AL and the messy complexities of real-world applications.