HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks (1911.03852v1)

Abstract: Quantization is an effective method for reducing memory footprint and inference time of Neural Networks, e.g., for efficient inference in the cloud, especially at the edge. However, ultra low precision quantization could lead to significant degradation in model generalization. A promising method to address this is to perform mixed-precision quantization, where more sensitive layers are kept at higher precision. However, the search space for a mixed-precision quantization is exponential in the number of layers. Recent work has proposed HAWQ, a novel Hessian based framework, with the aim of reducing this exponential search space by using second-order information. While promising, this prior work has three major limitations: (i) HAWQV1 only uses the top Hessian eigenvalue as a measure of sensitivity and do not consider the rest of the Hessian spectrum; (ii) HAWQV1 approach only provides relative sensitivity of different layers and therefore requires a manual selection of the mixed-precision setting; and (iii) HAWQV1 does not consider mixed-precision activation quantization. Here, we present HAWQV2 which addresses these shortcomings. For (i), we perform a theoretical analysis showing that a better sensitivity metric is to compute the average of all of the Hessian eigenvalues. For (ii), we develop a Pareto frontier based method for selecting the exact bit precision of different layers without any manual selection. For (iii), we extend the Hessian analysis to mixed-precision activation quantization. We have found this to be very beneficial for object detection. We show that HAWQV2 achieves new state-of-the-art results for a wide range of tasks.

Overview of HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks

The paper "HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks" proposes advancements in quantization techniques for neural networks aimed at reducing memory footprint and inference time, especially relevant for edge devices. Quantization involves converting floating-point network parameters to lower precision formats, thus enhancing computational efficiency. However, ultra-low precision quantization often leads to significant degradation in model performance. Mixed-precision quantization, where layers are assigned different levels of precision based on sensitivity, is a promising solution. The challenge lies in the exponentially large search space for determining optimal mixed-precision settings. HAWQ-V2 addresses this by enhancing the utilization of second-order Hessian information, building upon the initial HAWQ framework.

Key Contributions and Methodology

1. Improved Sensitivity Metric: Unlike its predecessor, HAWQ, which utilized the top Hessian eigenvalue for layer sensitivity, HAWQ-V2 employs the average of all Hessian eigenvalues as a more comprehensive sensitivity metric. This improvement aligns with theoretical analyses that suggest a holistic view of the Hessian spectrum can lead to better sensitivity assessments.
2. Efficient Hessian Trace Computation: HAWQ-V2 implements a fast algorithm using Hutchinson's method for computing Hessian trace information in PyTorch, which computes the trace efficiently without explicitly forming the Hessian matrix. This process, notably the computation for ResNet50, is efficient, requiring only 30 minutes with 4 GPUs.
3. Automatic Precision Selection: The paper introduces an automated method for determining mixed-precision settings using a Pareto frontier approach, which systematically evaluates candidate settings based on the total second-order perturbation. This method eliminates the need for manual bit precision selection, previously necessary in HAWQ, and facilitates optimal trade-offs between model size and performance.
4. Extension to Activation Quantization: The framework is extended to include mixed-precision activation quantization. A novel method for computing Hessian information concerning activations suggests that varying activation precision across layers can significantly boost performance, notably in applications like object detection.

Experimental Results

The paper provides extensive empirical evidence of HAWQ-V2's efficacy across various tasks and models. Notable results include:

• Inception-V3 on ImageNet: Achieving a top-1 accuracy of 75.68% with a model size reduced to 7.57MB, surpassing other quantization methods while maintaining a significant compression ratio.
• ResNet50 on ImageNet: HAWQ-V2 attains 75.76% accuracy with a 7.99MB model, without heuristic or manual intervention, outperforming its precursors and other state-of-the-art quantization methods.
• SqueezeNext: Demonstrates superior accuracy at 68.38% with an unprecedented model size of 1.07MB.
• Object Detection on COCO: The method achieves a mean average precision (mAP) of 34.4, improving significantly over previous methods such as FQN.

Implications and Future Directions

The introduction of HAWQ-V2 marks a significant advancement in neural network quantization, providing a practical framework that leverages second-order Hessian information for optimal precision setting. The application's broad scope, from image classification to object detection, underscores its versatility and efficacy. Practically, it offers a scalable solution for deploying deep learning models on constrained hardware environments.

Theoretically, the paper suggests avenues for further exploration, such as using second-order information throughout the training process to encourage models to settle in flatter regions of the loss landscape, potentially allowing more aggressive quantization. Additionally, exploring quantization in scenarios with restricted access to training data presents another viable research direction, addressing constraints imposed by privacy regulations.

In summary, HAWQ-V2 reinforces the potent role of second-order methods in neural network optimization, bridging theoretical analysis with practical implementations that push the boundaries of efficient AI deployment strategies.