Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysis (2306.07664v1)
Abstract: In recent years, LLMs (LMs) have made remarkable progress in advancing the field of NLP. However, the impact of data augmentation (DA) techniques on the fine-tuning (FT) performance of these LMs has been a topic of ongoing debate. In this study, we evaluate the effectiveness of three different FT methods in conjugation with back-translation across an array of 7 diverse NLP tasks, including classification and regression types, covering single-sentence and sentence-pair tasks. Contrary to prior assumptions that DA does not contribute to the enhancement of LMs' FT performance, our findings reveal that continued pre-training on augmented data can effectively improve the FT performance of the downstream tasks. In the most favourable case, continued pre-training improves the performance of FT by more than 10% in the few-shot learning setting. Our finding highlights the potential of DA as a powerful tool for bolstering LMs' performance.
- Natural language understanding approaches based on joint task of intent detection and slot filling for iot voice interaction. Neural Computing and Applications, 2020.
- Eigenvector-based graph neural network embeddings and trust rating prediction in bitcoin networks. ICAIF ’22, 2022.
- Stepgame: A new benchmark for robust multi-hop spatial reasoning in texts. In AAAI 2022.
- Attention-based ingredient parser. In ESANN, Bruges, Belgium, 2022.
- Joint geographical and temporal modeling based on matrix factorization for point-of-interest recommendation. In ECIR. Springer, 2020.
- Priming and actions: An analysis in conversational search systems. SIGIR, 2023.
- Evaluating the cranfield paradigm for conversational search systems. ICTIR, 2022.
- Self contrastive learning for session-based recommendation. arXiv preprint arXiv:2306.01266, 2023.
- Learning to execute actions or ask clarification questions. In Findings of NAACL 2022.
- Extending clip for category-to-image retrieval in e-commerce. In ECIR, 2022.
- When and what to ask through world states and text instructions: Iglu nlp challenge solution. NeurIPS IGLU Competition Workshop, 2023.
- How effective is task-agnostic data augmentation for pretrained transformers? In Findings of EMNLP 2020. ACL, 2020.
- Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
- FlipDA: Effective and robust data augmentation for few-shot learning. In ACL. ACL, May 2022.
- Making pre-trained language models better few-shot learners. In ACL, pages 3816–3830, Online, August 2021. ACL.
- Differentiable prompt makes pre-trained language models better few-shot learners. In ICLR, 2022.
- Exploiting cloze-questions for few-shot text classification and natural language inference. In ACL. ACL, April 2021.
- Rethinking semi-supervised learning with language models. In Findings of ACL 2023, Toronto, Canada, 2023. Association for Computational Linguistics.
- Don’t stop pretraining? make prompt-based fine-tuning powerful learner. In Arxiv, 2023.
- Don’t stop pretraining: Adapt language models to domains and tasks. In ACL, pages 8342–8360. ACL, July 2020.
- Recursive deep models for semantic compositionality over a sentiment treebank. In emnlp, 2013.
- Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In acl, 2005.
- Mining and summarizing customer reviews. In ACM SIGKDD international conference on Knowledge discovery and data mining, 2004.
- Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3), 2005.
- Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In acl, 2004.
- Building a question answering test collection. In SIGIR, 2000.
- SemEval task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. 2017.
- fairseq: A fast, extensible toolkit for sequence modeling. In NAACL-HLT, 2019.
- AdaPrompt: Adaptive model training for prompt-based NLP. In Findings of EMNLP 2022.
- Zhengxiang Shi (10 papers)
- Aldo Lipani (27 papers)