2000 character limit reached
Layer-wise Regularized Dropout for Neural Language Models
Published 26 Feb 2024 in cs.CL and cs.AI | (2402.16361v1)
Abstract: Among the various pre-trained neural LLMs that are popular today, dropout is already an indispensable regularization technique. To solve the inconsistency between training and inference caused by the randomness of dropout, some studies use consistency training to regularize dropout at the output layer. In this paper, we propose a novel Layer-wise Regularized Dropout (LR-Drop), which is specially designed for Transformer-based LLMs. Specifically, LR-Drop layer-wise regularizes each Transformer layer using the consistency training strategy. Each training sample passes through the two siamese sub-models sampled by dropout, and then LR-Drop forces the hidden states, multi-head attention matrices, and output distribution of the two siamese sub-models to be consistent. The proposed LR-Drop can be regarded as a "self-distillation" framework, in which each sub-model generated by dropout is the other's "teacher" model and "student" model. Through extensive experiments on 8 natural language understanding datasets, 6 neural machine translation datasets, and 1 abstractive summarization dataset (a total of 15 datasets), we show that LR-Drop achieves superior performances, including state-of-the-art results.
- Better fine-tuning by reducing representational collapse. In International Conference on Learning Representations.
- Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007). Association for Computational Linguistics.
- Jimmy Ba and Brendan Frey. 2013. Adaptive dropout for training deep neural networks. Advances in neural information processing systems, 26:3084–3092.
- Layer normalization. arXiv preprint arXiv:1607.06450.
- The second PASCAL recognising textual entailment challenge.
- What does bert look at? an analysis of bert’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286.
- Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations.
- The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment. Springer.
- William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing.
- Born again neural networks. In International Conference on Machine Learning, pages 1607–1616. PMLR.
- The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing.
- Explaining and harnessing adversarial examples. ICLR.
- Distilling the knowledge in a neural network.
- Sepp Hochreiter and Jürgen Schmidhuber. 1995. Simplifying neural nets by discovering flat minima. In Advances in neural information processing systems, pages 529–536.
- Do we need zero training loss after achieving zero training error? In International Conference on Machine Learning, pages 4604–4614. PMLR.
- First quora dataset release: Question pairs.
- Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174.
- Shakeout: A new regularized deep neural network training scheme. In Thirtieth AAAI Conference on Artificial Intelligence.
- Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
- On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 2017.
- Anders Krogh and John A Hertz. 1992. A simple weight decay can improve generalization. In Advances in neural information processing systems, pages 950–957.
- Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
- Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880.
- Regularization via structural label smoothing. In International Conference on Artificial Intelligence and Statistics, pages 1453–1463. PMLR.
- Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
- Dropout with expectation-linear regularization. arXiv preprint arXiv:1609.08017.
- Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351–3361.
- When does label smoothing help? arXiv preprint arXiv:1906.02629.
- Shiwen Ni and Hung-Yu Kao. 2023. Masked siamese prompt tuning for few-shot natural language understanding. IEEE Transactions on Artificial Intelligence.
- Dropattack: A masked weight adversarial training method to improve generalization of neural networks. arXiv preprint arXiv:2108.12805.
- Hat4rd: Hierarchical adversarial training for rumor detection in social media. Sensors, 22(17):6652.
- R-at: Regularized adversarial training for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6427–6440.
- Analyzing noise in autoencoders and deep networks. arXiv preprint arXiv:1406.1831.
- Prophetnet: Predicting future n-gram for sequence-to-sequencepre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2401–2410.
- SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
- Tim Salimans and Durk P Kingma. 2016. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29:901–909.
- Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP.
- Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958.
- Attention is all you need. Advances in neural information processing systems, 30.
- Regularization of neural networks using dropconnect. In International conference on machine learning, pages 1058–1066. PMLR.
- Neural network acceptability judgments. arXiv preprint 1805.12471.
- A broad-coverage challenge corpus for sentence understanding through inference. In NAACL.
- R-drop: Regularized dropout for neural networks. Advances in Neural Information Processing Systems, 34:10890–10905.
- Yuxin Wu and Kaiming He. 2018. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3–19.
- Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32.
- Delving deep into label smoothing. IEEE Transactions on Image Processing, 30:5984–5996.
- Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328–11339. PMLR.
- Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3713–3722.
- Zhilu Zhang and Mert Sabuncu. 2020. Self-distillation as instance-specific label smoothing. Advances in Neural Information Processing Systems, 33:2184–2195.
- Freelb: Enhanced adversarial training for natural language understanding. In ICLR.
- Fraternal dropout. In International Conference on Learning Representations.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.