Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer (2210.06707v1)

Abstract: The large pre-trained vision transformers (ViTs) have demonstrated remarkable performance on various visual tasks, but suffer from expensive computational and memory cost problems when deployed on resource-constrained devices. Among the powerful compression approaches, quantization extremely reduces the computation and memory consumption by low-bit parameters and bit-wise operations. However, low-bit ViTs remain largely unexplored and usually suffer from a significant performance drop compared with the real-valued counterparts. In this work, through extensive empirical analysis, we first identify the bottleneck for severe performance drop comes from the information distortion of the low-bit quantized self-attention map. We then develop an information rectification module (IRM) and a distribution guided distillation (DGD) scheme for fully quantized vision transformers (Q-ViT) to effectively eliminate such distortion, leading to a fully quantized ViTs. We evaluate our methods on popular DeiT and Swin backbones. Extensive experimental results show that our method achieves a much better performance than the prior arts. For example, our Q-ViT can theoretically accelerates the ViT-S by 6.14x and achieves about 80.9% Top-1 accuracy, even surpassing the full-precision counterpart by 1.0% on ImageNet dataset. Our codes and models are attached on https://github.com/YanjingLi0202/Q-ViT

Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer

The paper "Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer" focuses on addressing the computational and memory overhead inherent in Vision Transformers (ViTs) by exploring techniques for full quantization of these models. The authors identify quantization as a promising approach to reduce computation and memory usage, although existing low-bit quantization approaches for ViTs suffer from substantial performance degradation compared with their full-precision equivalents. They propose a novel solution involving components such as the Information Rectification Module (IRM) and Distribution Guided Distillation (DGD) scheme to tackle this issue.

Overview

Vision Transformers, inspired by their success in NLP tasks, have been shown to perform exceptionally well on various computer vision tasks. However, the computational cost and memory footprint of these models are significant, particularly when used on resource-constrained devices. Given these constraints, efficient compression techniques like quantization are crucial. Quantization involves reducing the bit-width of network parameters, making it suitable for AI chips but often results in a performance drop when executed on ViTs. This research expands on creating methods to maintain accuracy when quantizing ViTs aggressively.

Methodology

The authors' approach involves two core innovations:

1. Information Rectification Module (IRM): The IRM addresses the quantization-induced information distortion within self-attention maps of ViTs. By maximizing the information entropy of quantized data, IRM helps to restore the representational power of the attention mechanisms, thus rectifying the distribution-shift in attention maps that typically occur due to quantization.
2. Distribution Guided Distillation (DGD) Scheme: DGD improves the optimization process during backpropagation by employing attention-based distillation mechanisms. This method ensures that the quantized models learn effectively from full-precision teacher models, focusing on mitigating distribution mismatches that can hinder training efficacy.

Results

The paper demonstrates that their methods achieve competitive results compared to full-precision models while using significantly lower bit-widths. The Q-ViT approach retains similar accuracy levels on ImageNet when compared against standard quantized ViT baselines and traditional quantization techniques such as LSQ. For instance, their 4-bit Q-ViT variant not only reduces computation significantly but occasionally surpasses the performance of full-precision counterparts, notably achieving a 6.14× theoretical speedup over the ViT-S with an accuracy improvement of 1.0% on the ImageNet dataset.

Implications

This research indicates a viable path forward for the deployment of ViTs in more constrained environments, thereby expanding their applicability to edge devices where computational resources are limited. From a theoretical perspective, it underscores the potential benefits of integrating information-theoretic principles with model compression techniques. Additionally, enhancements in quantization-aware training protocols can lead to further advancements in the efficiency and accessibility of deep learning models.

Future Directions

Future work could explore applying similar quantization techniques to other emerging architectures beyond ViTs. The application framework outlined by the authors could be adapted and optimized for other tasks within computer vision and beyond, fostering improvements in both the compressibility of models and the generalizability of quantization techniques. AI hardware design might also benefit from these advances, leveraging ultra-low-bit quantizations for efficient and robust deep learning deployments. Additionally, investigating the combination of quantization with other compression approaches, such as pruning or low-rank decomposition, could yield further reductions in model size and computational demands.

In conclusion, the paper presents a substantial contribution to the field of model compression in Vision Transformers by addressing some of the key challenges in deploying these models efficiently while maintaining high performance through novel quantization strategies.