SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection (2401.15293v1)
Abstract: Vision transformers are known to be more computationally and data-intensive than CNN models. These transformer models such as ViT, require all the input image tokens to learn the relationship among them. However, many of these tokens are not informative and may contain irrelevant information such as unrelated background or unimportant scenery. These tokens are overlooked by the multi-head self-attention (MHSA), resulting in many redundant and unnecessary computations in MHSA and the feed-forward network (FFN). In this work, we propose a method to optimize the amount of unnecessary interactions between unimportant tokens by separating and sending them through a different low-cost computational path. Our method does not add any parameters to the ViT model and aims to find the best trade-off between training throughput and achieving a 0% loss in the Top-1 accuracy of the final model. Our experimental results on training ViT-small from scratch show that SkipViT is capable of effectively dropping 55% of the tokens while gaining more than 13% training throughput and maintaining classification accuracy at the level of the baseline model on Huawei Ascend910A.
- Speeding up resnet architecture with layers targeted low rank decomposition. arXiv preprint arXiv:2309.12412.
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv preprint arXiv:2305.13245.
- Gray-box Adversarial Attack of Deep Reinforcement Learning-based Trading Agents. arXiv:2309.14615.
- Deep State Inference: Toward Behavioral Model Inference of Black-Box Software Systems. IEEE Transactions on Software Engineering, 48(12): 4857–4872.
- ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255.
- Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314.
- Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
- An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
- SwiftLearn: A Data-Efficient Training Method of Deep Learning Models using Importance Sampling. arXiv preprint arXiv:2311.15134.
- Methods, systems, and media for computer vision using 2d convolution of 4d video data tensors. US Patent App. 17/502,588.
- Compressing Pre-trained Language Models using Progressive Low Rank Decomposition.
- Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
- GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values. arXiv preprint arXiv:2311.03426.
- A short study on compressing decoder-based language models. arXiv preprint arXiv:2110.08460.
- Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations. arXiv:2202.07800.
- Fixing Weight Decay Regularization in Adam. CoRR, abs/1711.05101.
- Dynamic spatial sparsification for efficient vision transformers and convolutional neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence.
- Shazeer, N. 2019. Fast Transformer Decoding: One Write-Head is All You Need. arXiv:1911.02150.
- Training data-efficient image transformers & distillation through attention. In International conference on machine learning, 10347–10357. PMLR.
- Attention is all you need. Advances in neural information processing systems, 30.
- Attention Is All You Need. arXiv:1706.03762.
- Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers. arXiv:2211.11586.
- Foozhan Ataiefard (6 papers)
- Walid Ahmed (13 papers)
- Habib Hajimolahoseini (10 papers)
- Saina Asani (3 papers)
- Farnoosh Javadi (5 papers)
- Mohammad Hassanpour (6 papers)
- Omar Mohamed Awad (8 papers)
- Austin Wen (6 papers)
- Kangling Liu (3 papers)
- Yang Liu (2253 papers)