Self-training Improves Pre-training for Natural Language Understanding (2010.02194v1)

Abstract: Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semi-supervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with improvements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning.

Overview of "Self-training Improves Pre-training for Natural Language Understanding"

The paper under discussion presents a novel approach to enhance natural language understanding (NLU) using self-training methods. The authors introduce "SentAugment," a data augmentation procedure that effectively combines self-training with unsupervised pre-training techniques. Through a series of well-structured experiments, the paper focuses on demonstrating the complementary role of self-training alongside pre-trained models like RoBERTa-Large.

Methodology

The crux of the paper lies in its innovative SentAugment approach. The authors outline a multi-step procedure:

1. A RoBERTa-Large model serves as the teacher and is fine-tuned on a downstream task with cross-entropy loss.
2. Task-specific query embeddings originating from labeled data are used to extract substantial unannotated data from a vast corpus of web-crawled sentences.
3. This data is then synthetically annotated by the teacher model, resulting in a carefully selected set of top K samples for each class to form a synthetic dataset.
4. Finally, a student RoBERTa-Large model is fine-tuned on this dataset using KL-divergence.

This process distinctively differs from prior semi-supervised methods primarily in the data extraction phase, making it adaptable to open-domain scenarios. The paper underscores that self-training does not necessitate in-domain unlabeled data, enhancing its applicability across domains.

Experimental Results

Numerical results substantiate the efficacy of the proposed method. Key findings include:

• Across natural language understanding benchmarks, involving tasks such as sentiment analysis, product classification, and named entity recognition, the self-training approach yielded significant gains—up to 2.6% improvement over strong RoBERTa-Large baselines.
• Few-shot learning scenarios exhibited an average improvement of 3.5% in accuracy, demonstrating self-training's robustness when labeled data is scarce.
• For knowledge distillation, SentAugment improved distilled RoBERTa models by 2.9% in accuracy on average, minimizing the gap between teacher and student models.

These results not only validate SentAugment's effectiveness but also underline self-training's utility as a complementary approach to pre-training.

Contributions and Implications

Several contributions emerge from this work:

• SentAugment introduces a practical framework for leveraging extensive unannotated web data, broadening the horizon for applications without relying on in-domain data.
• It reveals that self-training and pre-training methods extract differing types of information, yet gain when cohesively applied, thereby providing further insights into semi-supervised learning dynamics.
• By advancing knowledge distillation and few-shot learning, this work sets the stage for more compact and efficient models in practical deployments.

Future Directions

The paper leaves several avenues open for future exploration. Ensuring scalable systems for broader applications remains a primary challenge. Additionally, exploring the potential impacts of increasing the diversity within the unannotated sentence bank could further enhance the results. Investigating how SentAugment can be applied to more sophisticated or emerging natural language tasks would also be beneficial for further research.

Overall, the paper makes substantial contributions to the field of NLU, demonstrating how self-training, when strategically combined with data augmentation tactics like SentAugment, can effectively elevate model performances even in resource-constrained settings. This work offers a compelling approach to leverage the synergy between unsupervised pre-training and self-training methodologies within AI research.