Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Preparing Lessons for Progressive Training on Language Models (2401.09192v3)

Published 17 Jan 2024 in cs.LG and cs.AI

Abstract: The rapid progress of Transformers in artificial intelligence has come at the cost of increased resource consumption and greenhouse gas emissions due to growing model sizes. Prior work suggests using pretrained small models to improve training efficiency, but this approach may not be suitable for new model structures. On the other hand, training from scratch can be slow, and progressively stacking layers often fails to achieve significant acceleration. To address these challenges, we propose a novel method called Apollo, which prep\textbf{a}res lessons for ex\textbf{p}anding \textbf{o}perations by \textbf{l}earning high-\textbf{l}ayer functi\textbf{o}nality during training of low layers. Our approach involves low-value-prioritized sampling (LVPS) to train different depths and weight sharing to facilitate efficient expansion. We also introduce an interpolation method for stable model depth extension. Experiments demonstrate that Apollo achieves state-of-the-art acceleration ratios, even rivaling methods using pretrained models, making it a universal and efficient solution for training deep models while reducing time, financial, and environmental costs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Token Merging: Your ViT But Faster. CoRR, abs/2210.09461.
  2. Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation. CoRR, abs/2105.05537.
  3. Multi-level Residual Networks from Dynamical Systems View. In ICLR (Poster). OpenReview.net.
  4. bert2BERT: Towards Reusable Pretrained Language Models. In ACL (1), 2134–2148. Association for Computational Linguistics.
  5. Net2Net: Accelerating Learning via Knowledge Transfer. In ICLR.
  6. Universal Transformers. In ICLR (Poster). OpenReview.net.
  7. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT (1), 4171–4186. Association for Computational Linguistics.
  8. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR. OpenReview.net.
  9. Efficient Training of BERT by Progressively Stacking. In ICML, volume 97 of Proceedings of Machine Learning Research, 2337–2346. PMLR.
  10. On the Transformer Growth for Progressive BERT Training. In NAACL-HLT, 5174–5180. Association for Computational Linguistics.
  11. Exploring Low Rank Training of Deep Neural Networks. CoRR, abs/2209.13569.
  12. Adam: A Method for Stochastic Optimization. In ICLR (Poster).
  13. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In ICLR. OpenReview.net.
  14. Automated Progressive Learning for Efficient Training of Vision Transformers. In CVPR, 12476–12486. IEEE.
  15. Heuristic Rank Selection with Progressively Searching Tensor Ring Network. CoRR, abs/2009.10580.
  16. Compressing Recurrent Neural Networks with Tensor Ring for Action Recognition. In AAAI, 4683–4690. AAAI Press.
  17. Reusing Pretrained Models by Multi-linear Operators for Efficient Training. In NeurIPS.
  18. Knowledge Inheritance for Pre-trained Language Models. In NAACL-HLT, 3921–3937. Association for Computational Linguistics.
  19. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9.
  20. Know What You Don’t Know: Unanswerable Questions for SQuAD. In ACL (2), 784–789. Association for Computational Linguistics.
  21. SQuAD: 100, 000+ Questions for Machine Comprehension of Text. In EMNLP, 2383–2392. The Association for Computational Linguistics.
  22. A Primer in BERTology: What We Know About How BERT Works. Trans. Assoc. Comput. Linguistics, 8: 842–866.
  23. Green AI. CoRR, abs/1907.10597.
  24. Staged Training for Transformer Language Models. In ICML, volume 162 of Proceedings of Machine Learning Research, 19893–19908. PMLR.
  25. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. CoRR, abs/1909.08053.
  26. Attention is All you Need. In NIPS, 5998–6008.
  27. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In ICLR (Poster). OpenReview.net.
  28. Energy-Aware Neural Architecture Optimization with Fast Splitting Steepest Descent. CoRR, abs/1910.03103.
  29. Tensor Networks Meet Neural Networks: A Survey and Future Perspectives. CoRR, abs/2302.09019.
  30. Concatenated Tensor Networks for Deep Multi-Task Learning. In ICONIP (5), volume 1333 of Communications in Computer and Information Science, 517–525. Springer.
  31. Learning to Grow Pretrained Models for Efficient Transformer Training. In ICLR. OpenReview.net.
  32. Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks. In NeurIPS.
  33. Splitting Steepest Descent for Growing Neural Architectures. In NeurIPS, 10655–10665.
  34. Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting. CoRR, abs/2003.10392.
  35. Taking Notes on the Fly Helps Language Pre-Training. In ICLR. OpenReview.net.
  36. Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedup. CoRR, abs/2011.13635.
  37. Speeding up Deep Model Training by Sharing Weights and Then Unsharing. CoRR, abs/2110.03848.
  38. Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping. In NeurIPS.
  39. Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books. In ICCV, 19–27. IEEE Computer Society.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now