On Dataset Transferability in Active Learning for Transformers
Abstract: Active learning (AL) aims to reduce labeling costs by querying the examples most beneficial for model learning. While the effectiveness of AL for fine-tuning transformer-based pre-trained LLMs (PLMs) has been demonstrated, it is less clear to what extent the AL gains obtained with one model transfer to others. We consider the problem of transferability of actively acquired datasets in text classification and investigate whether AL gains persist when a dataset built using AL coupled with a specific PLM is used to train a different PLM. We link the AL dataset transferability to the similarity of instances queried by the different PLMs and show that AL methods with similar acquisition sequences produce highly transferable datasets regardless of the models used. Additionally, we show that the similarity of acquisition sequences is influenced more by the choice of the AL method than the choice of the model.
- Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671.
- Josh Attenberg and Foster Provost. 2011. Inactive learning? difficulties employing active learning in practice. ACM SIGKDD Explorations Newsletter, 12(2):36–41.
- ELECTRA: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555.
- BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
- Active learning for BERT: an empirical study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7949–7962.
- On statistical bias in active learning: How and when to fix it. arXiv preprint arXiv:2101.11665.
- To softmax, or not to softmax: that is the question when applying active learning for transformer models. arXiv preprint arXiv:2210.03005.
- Fine-tuning BERT for low-resource natural language understanding via active learning. arXiv preprint arXiv:2012.02462.
- Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83–97.
- Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
- RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
- Practical obstacles to deploying active learning. arXiv preprint arXiv:1807.04801.
- On the importance of effectively adapting pretrained language models for active learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 825–836.
- A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the ACL.
- GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543.
- Rissanen data analysis: Examining dataset characteristics via description length. In International Conference on Machine Learning, pages 8500–8513. PMLR.
- Sampling bias in deep active classification: An empirical study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4058–4068.
- Revisiting uncertainty-based query strategies for active learning with transformers. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2194–2203.
- Ozan Sener and Silvio Savarese. 2017. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489.
- Burr Settles. 2009. Active learning literature survey. Computer sciences technical report.
- Active learning for sequence tagging with deep pre-trained models and bayesian uncertainty estimates. arXiv preprint arXiv:2101.08133.
- Katrin Tomanek and Katherina Morik. 2011. Inspecting sample reusability for active learning. In Active Learning and Experimental Design workshop In conjunction with AISTATS 2010, pages 169–181. JMLR Workshop and Conference Proceedings.
- Towards computationally feasible deep active learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1198–1218.
- Attention is all you need. Advances in neural information processing systems, 30.
- Neural network acceptability judgments. arXiv preprint arXiv:1805.12471.
- Cold-start active learning through self-supervised language modeling. arXiv preprint arXiv:2010.09535.
- Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
- Fedor Zhdanov. 2019. Diverse mini-batch active learning. arXiv preprint arXiv:1901.05954.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
