Active Learning by Acquiring Contrastive Examples (2109.03764v1)

Abstract: Common acquisition functions for active learning use either uncertainty or diversity sampling, aiming to select difficult and diverse data points from the pool of unlabeled data, respectively. In this work, leveraging the best of both worlds, we propose an acquisition function that opts for selecting \textit{contrastive examples}, i.e. data points that are similar in the model feature space and yet the model outputs maximally different predictive likelihoods. We compare our approach, CAL (Contrastive Active Learning), with a diverse set of acquisition functions in four natural language understanding tasks and seven datasets. Our experiments show that CAL performs consistently better or equal than the best performing baseline across all tasks, on both in-domain and out-of-domain data. We also conduct an extensive ablation study of our method and we further analyze all actively acquired datasets showing that CAL achieves a better trade-off between uncertainty and diversity compared to other strategies.

• The paper introduces Contrastive Active Learning (Cal), a novel approach using contrastive examples (similar in feature space, diverse in predictions) to enhance active learning efficiency by selecting data near the decision boundary.
• Experimental results across various NLP tasks show Cal achieves comparable or better in-domain accuracy and significantly improves out-of-domain robustness compared to traditional active learning strategies.
• Cal offers practical benefits by reducing annotation costs through more efficient data selection and theoretically unifies uncertainty and diversity sampling frameworks.

Active Learning by Acquiring Contrastive Examples

The paper "Active Learning by Acquiring Contrastive Examples" introduces a novel approach to active learning (AL) that enhances data efficiency by selecting contrastive examples. This approach combines the strengths of existing AL strategies by leveraging both uncertainty and diversity sampling techniques.

Methodology Overview

The proposed method, termed Contrastive Active Learning (Cal), identifies contrastive examples as data points that are similar in the model's feature space yet yield diverse predictive likelihoods. The authors define these examples based on two criteria: similarity in feature embeddings and maximum divergence in predictive distributions. The approach is hypothesized to efficiently identify data near the decision boundary of the model, thus enhancing learning from fewer labeled instances.

Cal is evaluated using a Bert model across four NLP tasks—sentiment analysis, topic classification, natural language inference, and paraphrase detection. The experiments conducted on seven datasets demonstrate that Cal performs comparably or better than traditional acquisition functions, particularly under scenarios involving out-of-domain data.

Experimental Insights

1. In-Domain Performance: Cal consistently achieves high in-domain accuracy across various NLP tasks, frequently surpassing or matching the performance of baseline acquisition strategies such as Entropy and Alps. Notably, Cal demonstrates notable robustness in challenging datasets like dbpedia and agnews.
2. Out-of-Domain Robustness: The paper shows that models trained on subsets selected by Cal generalize better on out-of-domain data compared to those trained with data selected by traditional methods. This indicates an enhanced capability to capture more generalizable features from the selected data.
3. Ablation Studies: Extensive ablation studies emphasize the importance of the selection criteria used in Cal. The studies confirm that selecting contrastive examples—especially those near the decision boundary—significantly contributes to effective model learning.
4. Efficiency Analysis: In terms of computational efficiency, Cal is more efficient compared to computationally heavy clustering-based approaches, maintaining linear complexity with respect to dataset size and acquisition steps.

Theoretical and Practical Implications

Theoretically, this work challenges the dichotomy between uncertainty and diversity in active learning, presenting a unified framework that effectively harnesses both. It introduces a practical approach that addresses common pitfalls of existing AL strategies, such as acquiring redundant data points that do not meaningfully enhance model performance.

Practically, Cal represents a valuable tool for reducing annotation costs. By effectively acquiring informative samples that lie near the decision boundary, it allows for a more efficient utilization of limited labeling budgets. This is particularly beneficial in domains with expensive data acquisition processes.

Future Directions

The paper opens several avenues for future research. There's potential for extending Cal to additional tasks beyond NLP, including computer vision and other domains where model interpretability and robustness are crucial. Future work could also investigate alternative representation spaces or evaluate Cal's integration with recently developed transformer models that incorporate richer contextual understanding.

Overall, by introducing contrastive examples as a selection criterion for active learning, this paper contributes a meaningful refinement to established techniques, highlighting the potential for improved data efficiency and expanded applicability across various machine learning tasks.