Mixed Precision DNNs: All you need is a good parametrization (1905.11452v3)

Abstract: Efficient deep neural network (DNN) inference on mobile or embedded devices typically involves quantization of the network parameters and activations. In particular, mixed precision networks achieve better performance than networks with homogeneous bitwidth for the same size constraint. Since choosing the optimal bitwidths is not straight forward, training methods, which can learn them, are desirable. Differentiable quantization with straight-through gradients allows to learn the quantizer's parameters using gradient methods. We show that a suited parametrization of the quantizer is the key to achieve a stable training and a good final performance. Specifically, we propose to parametrize the quantizer with the step size and dynamic range. The bitwidth can then be inferred from them. Other parametrizations, which explicitly use the bitwidth, consistently perform worse. We confirm our findings with experiments on CIFAR-10 and ImageNet and we obtain mixed precision DNNs with learned quantization parameters, achieving state-of-the-art performance.

• The paper demonstrates that learning quantization stepsize and dynamic range yields stable gradients and enhanced performance compared to fixed bitwidth methods.
• The paper shows that optimizing with memory constraints enables efficient deployment on resource-limited devices without compromising accuracy.
• The paper validates its approach on CIFAR-10 and ImageNet, achieving lower error rates than conventional mixed precision techniques.

Mixed Precision DNNs: A Comprehensive Overview of Optimal Parametrization

The concept of mixed precision in deep neural networks (DNNs) involves the application of variable precision across different network parameters, with the ultimate aim of enhancing performance without significantly increasing the computational complexity. This paper methodically explores a potentially effective strategy for parameterizing mixed precision DNNs, focusing on the efficient quantization of network parameters for use in devices with constrained computational resources.

Objective and Methodology

A central problem addressed by the paper is the difficulty associated with optimally choosing bitwidths for network parameters, which dictates the efficiency and speed of these models on limited-capacity devices. Traditional methods of quantization utilize fixed bitwidths across all network layers, which may not yield optimal performance. Mixed precision networks resolve some of these issues by allowing adaptive bitwidths across layers, thus achieving a higher performance-to-complexity ratio.

The researchers investigate differentiable quantization (DQ) techniques that utilize straight-through gradient estimators (STE) to tune quantization parameters, such as stepsize and dynamic range, using gradient descent methods. The authors propose a novel parametrization method in which directly learning bitwidth is avoided. Instead, stepsize and dynamic range are directly learned, with bitwidth inferred consequently. This approach purportedly provides higher stability during training and averts the high variance observed in stochastic training methods.

Key Findings

1. Parametrization Efficiency: The paper identifies three potential parametrizations for both uniform and power-of-two quantization schemas. It concludes that parameterizing with stepsize and dynamic range yields gradients with stable norms, hence optimizing performance in comparison with parametrization schemes directly involving bitwidth.
2. Resource Constraint FulfiLLMent: Addressing practical deployment issues, the authors define optimization strategies that integrate memory constraints, facilitating model deployment on devices with finite memory resources. Experiments demonstrate that learning quantization parameters with these constraints results in high-performing networks without exceeding preset memory limits.
3. Empirical Validation and Performance: The proposed methodologies are validated on standard datasets, including CIFAR-10 and ImageNet. Here, mixed precision models trained with the proposed parameterized DQ show performance enhancements aligning with state-of-the-art results while maintaining stringent memory constraints. For example, training a ResNet-20 on CIFAR-10 with the novel parametrization yields lower error rates than competing methods such as TQT and conventional fixed bitwidth quantization.

Implications and Future Directions

The findings advocate for a paradigm shift from direct bitwidth learning to a more nuanced approach using stepsize and dynamic range, which predicts the appropriate bitwidth. This has substantial implications for the design of efficient neural networks in real-world applications, particularly those reliant on mobile and embedded devices where computational resources are limited. Moreover, the adoption of this parametrization can streamline the computational process, depress oscillation issues in training, and retain model performance under quantization.

Future developments in neural network quantization could draw inspiration from these findings to explore different combinations of quantization schemes and parametrization techniques. Investigating broader application types and extending methodologies to novel architecture designs could lead to ubiquitous applicability across various machine learning domains and hardware environments.

Overall, the insights garnered from this research endeavor offer a compelling value proposition to the computer science community striving to reconcile the performance-efficiency conundrum persisting in contemporary deep learning architectures.