Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A General and Efficient Training for Transformer via Token Expansion (2404.00672v1)

Published 31 Mar 2024 in cs.LG, cs.AI, cs.CL, and cs.CV

Abstract: The remarkable performance of Vision Transformers (ViTs) typically requires an extremely large training cost. Existing methods have attempted to accelerate the training of ViTs, yet typically disregard method universality with accuracy dropping. Meanwhile, they break the training consistency of the original transformers, including the consistency of hyper-parameters, architecture, and strategy, which prevents them from being widely applied to different Transformer networks. In this paper, we propose a novel token growth scheme Token Expansion (termed ToE) to achieve consistent training acceleration for ViTs. We introduce an "initialization-expansion-merging" pipeline to maintain the integrity of the intermediate feature distribution of original transformers, preventing the loss of crucial learnable information in the training process. ToE can not only be seamlessly integrated into the training and fine-tuning process of transformers (e.g., DeiT and LV-ViT), but also effective for efficient training frameworks (e.g., EfficientTrain), without twisting the original training hyper-parameters, architecture, and introducing additional training strategies. Extensive experiments demonstrate that ToE achieves about 1.3x faster for the training of ViTs in a lossless manner, or even with performance gains over the full-token training baselines. Code is available at https://github.com/Osilly/TokenExpansion .

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Attention is all you need. NeurIPS, 30, 2017.
  2. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186, 2019.
  3. Language models are few-shot learners. NeurIPS, 33:1877–1901, 2020.
  4. Training data-efficient image transformers & distillation through attention. In ICLR, pages 10347–10357. PMLR, 2021.
  5. All tokens matter: Token labeling for training better vision transformers. NeurIPS, 34:18590–18602, 2021.
  6. End-to-end object detection with transformers. In ECCV, pages 213–229. Springer, 2020.
  7. Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS, 34:12077–12090, 2021.
  8. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
  9. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016.
  10. Chasing sparsity in vision transformers: An end-to-end exploration. NeurIPS, 34:19974–19988, 2021.
  11. Token merging: Your vit but faster. In ICLR, 2022.
  12. Network expansion for practical training acceleration. In CVPR, pages 20269–20279, 2023.
  13. Efficient training of bert by progressively stacking. In ICML, pages 2337–2346. PMLR, 2019.
  14. Global vision transformer pruning with hessian-aware saliency. In CVPR, pages 18547–18557, 2023.
  15. Width & depth pruning for vision transformers. In AAAI, volume 36, pages 3143–3151, 2022.
  16. Block pruning for faster transformers. In EMNLP, pages 10619–10629, 2021.
  17. Structured pruning learns compact and accurate models. In ACL, pages 1513–1528, 2022.
  18. Dynamicvit: Efficient vision transformers with dynamic token sparsification. NeurIPS, 34:13937–13949, 2021.
  19. Adavit: Adaptive vision transformers for efficient image recognition. In CVPR, pages 12309–12318, 2022.
  20. Adaptive token sampling for efficient vision transformers. In ECCV, pages 396–414. Springer, 2022.
  21. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In ECCV, pages 620–640. Springer, 2022.
  22. A-vit: Adaptive tokens for efficient vision transformer. In CVPR, pages 10809–10818, 2022.
  23. Q-detr: An efficient low-bit quantized detection transformer. In CVPR, pages 3842–3851, 2023.
  24. Q-vit: Accurate and fully quantized low-bit vision transformer. NeurIPS, 35:34451–34463, 2022.
  25. Bivit: Extremely compressed binary vision transformers. In ICCV, pages 5651–5663, 2023.
  26. Binaryvit: Pushing binary vision transformers towards convolutional models. In CVPR, pages 4664–4673, 2023.
  27. bert2bert: Towards reusable pretrained language models. In ACL, pages 2134–2148, 2022.
  28. Growing efficient deep networks by structured continuous sparsification. In ICLR, 2021.
  29. Autogrow: Automatic layer growing in deep convolutional networks. In KDD, pages 833–841, 2020.
  30. Efficienttrain: Exploring generalized curriculum learning for training visual backbones. In ICCV, pages 5852–5864, 2023.
  31. Automated progressive learning for efficient training of vision transformers. In CVPR, pages 12486–12496, 2022.
  32. Budgeted training for vision transformer. In ICLR, 2022.
  33. Deduplicating training data makes language models better. In ACL, pages 8424–8445, 2022.
  34. Efficientnetv2: Smaller models and faster training. In ICLR, pages 10096–10106. PMLR, 2021.
  35. Accelerating vision transformer training via a patch sampling schedule. arXiv preprint arXiv:2208.09520, 2022.
  36. On efficient training of large-scale deep learning models: A literature review. arXiv preprint arXiv:2304.03589, 2023.
  37. Efficient on-device training via gradient filtering. In CVPR, pages 3811–3820, 2023.
  38. Accelerating cnn training by pruning activation gradients. In ECCV, pages 322–338. Springer, 2020.
  39. Fractrain: Fractionally squeezing bit savings both temporally and spatially for efficient dnn training. NeurIPS, 33:12127–12139, 2020.
  40. E2-train: Training state-of-the-art cnns with over 80% energy savings. NeurIPS, 32, 2019.
  41. Budgeted training: Rethinking deep neural network training under resource constraints. In ICLR, 2019.
  42. Autoassist: A framework to accelerate training of deep neural networks. NeurIPS, 32, 2019.
  43. What’s the backward-forward flop ratio for neural networks? https://epochai.org/blog/backward-forward-FLOP-ratio, 2021. Accessed: 2023-9-28.
  44. Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9:2579–2605, 2008.
  45. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248–255. Ieee, 2009.
  46. Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
  47. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 32, 2019.
  48. Do vision transformers see like convolutional neural networks? NeurIPS, 34:12116–12128, 2021.
  49. You only look at one sequence: Rethinking transformer in vision through object detection. NeurIPS, 34:26183–26197, 2021.
  50. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer, 2014.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Wenxuan Huang (19 papers)
  2. Yunhang Shen (55 papers)
  3. Jiao Xie (11 papers)
  4. Baochang Zhang (113 papers)
  5. Gaoqi He (9 papers)
  6. Ke Li (723 papers)
  7. Xing Sun (94 papers)
  8. Shaohui Lin (45 papers)
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now