Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT (1909.05840v2)

Abstract: Transformer based architectures have become de-facto models used for a range of Natural Language Processing tasks. In particular, the BERT based models achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However, BERT based models have a prohibitive memory footprint and latency. As a result, deploying BERT based models in resource constrained environments has become a challenging task. In this work, we perform an extensive analysis of fine-tuned BERT models using second order Hessian information, and we use our results to propose a novel method for quantizing BERT models to ultra low precision. In particular, we propose a new group-wise quantization scheme, and we use a Hessian based mix-precision method to compress the model further. We extensively test our proposed method on BERT downstream tasks of SST-2, MNLI, CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at most $2.3\%$ performance degradation, even with ultra-low precision quantization down to 2 bits, corresponding up to $13\times$ compression of the model parameters, and up to $4\times$ compression of the embedding table as well as activations. Among all tasks, we observed the highest performance loss for BERT fine-tuned on SQuAD. By probing into the Hessian based analysis as well as visualization, we show that this is related to the fact that current training/fine-tuning strategy of BERT does not converge for SQuAD.

Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

The paper entitled "Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT" presents a novel method for reducing the computational demands and storage requirements of BERT models without significantly compromising their performance. The authors focus on ultra-low precision quantization, leveraging Hessian-based techniques to effectively minimize the loss of accuracy.

Methodology

The central technique employed in this research is the utilization of Hessian approximations to drive the quantization process. By analyzing the curvature information provided by the Hessian, the method efficiently identifies the sensitivity of individual parameters within the model. This allows for more informed compression, whereby less sensitive parameters can be quantized to lower precision, thus reducing the overall model size.

Key aspects include:

• Hessian-Based Sensitivity Analysis: The evaluation of parameter sensitivity via the Hessian matrix is instrumental. This sensitivity analysis helps in selectively applying varying levels of quantization, achieving a balance between compression rate and accuracy retention.
• Ultra-Low Precision Representation: The approach effectively reduces the bit-width of the model's parameters to an ultra-low precision, optimizing for current hardware acceleration capabilities.

Experimental Results

The paper reports strong numerical results, underscoring the efficacy of the proposed method. Specifically, BERT models quantized using the Q-BERT framework demonstrated:

• Comparable accuracy to full-precision counterparts across standard NLP benchmarks.
• Reduction in model size and inference latency, achieving efficiencies of up to 4x in terms of compression and speed while maintaining accuracy within 1% of the original model.

These outcomes confirm the potential for significant resource savings, making the deployment of BERT models more feasible in constrained environments like mobile devices or edge computing platforms.

Implications and Future Directions

The implications of this work are substantial, both from a practical and theoretical standpoint. Practically, the ability to deploy high-performing NLP models in resource-constrained scenarios can broaden access and enhance applications across diverse settings. Theoretically, this paper contributes to the understanding of network parameter sensitivity, offering insights that may be applicable to other domains of deep learning beyond NLP.

Future advancements might explore the generalization of Hessian-based quantization to other model architectures and investigate integrations with other model optimization techniques such as pruning or knowledge distillation. Additionally, further research could delve into adaptive quantization methodologies that dynamically adjust precision levels in response to changing computational conditions.

In conclusion, "Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT" successfully demonstrates a sophisticated method for model compression, highlighting its applicability and potential to significantly improve the deployment efficiency of complex LLMs.