Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM (1707.09870v2)

Abstract: Although deep learning models are highly effective for various learning tasks, their high computational costs prohibit the deployment to scenarios where either memory or computational resources are limited. In this paper, we focus on compressing and accelerating deep models with network weights represented by very small numbers of bits, referred to as extremely low bit neural network. We model this problem as a discretely constrained optimization problem. Borrowing the idea from Alternating Direction Method of Multipliers (ADMM), we decouple the continuous parameters from the discrete constraints of network, and cast the original hard problem into several subproblems. We propose to solve these subproblems using extragradient and iterative quantization algorithms that lead to considerably faster convergency compared to conventional optimization methods. Extensive experiments on image recognition and object detection verify that the proposed algorithm is more effective than state-of-the-art approaches when coming to extremely low bit neural network.

• The paper introduces an ADMM-based framework that reformulates low bit quantization as a discretely constrained optimization problem.
• It employs extragradient and iterative quantization methods to decouple continuous weights from discrete constraints, boosting convergence and efficiency.
• Experimental results show that using only 3-bit weights, the method achieves accuracy comparable to full precision networks on datasets like ImageNet and Pascal VOC.

An Overview of Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM

The paper presents a focused analysis and approach for compressing and accelerating deep neural networks by implementing extremely low bit quantization of network weights. The central thrust is to address the computational inefficiencies and storage burdens that often accompany comprehensive neural networks, such as the 16-layer VGG model with its significant 528 MB parameter size. In constrained environments with limited memory or computational resources, there is a compelling need to optimize these models.

Formulating the Problem

The authors model the quantization of neural networks to extremely low bits as a discretely constrained optimization problem. This is achieved by introducing auxiliary variables to decouple continuous parameters from discrete constraints using the Alternating Direction Method of Multipliers (ADMM). This methodology allows the problem of low bit quantized neural networks, which are typically NP-hard due to their non-convex nature, to be divided into solvable subproblems. This reformulation results in improved convergence and computational efficiency.

Methodology

The approach utilizes both extragradient and iterative quantization algorithms to solve the proximization and projection phases of ADMM, respectively. The extragradient method accelerates convergence by utilizing an additional gradient fix for better approximation of the region around network weights before transitioning to the corrected step. Iterative quantization, on the other hand, optimizes auxiliary variables through a stepwise fixed-point update, minimizing the approximation error by projecting against low bit constraints.

Experimental Validation

Experimental evaluations conducted on prominent datasets such as ImageNet and Pascal VOC, across widely-used architectures like AlexNet, VGG-16, ResNet-18, ResNet-50, and GoogleNet, exhibit superior performance over existing low bit quantization methods such as BWN and TWN. Notably, the proposed method achieves comparable accuracy with state-of-the-art full precision networks using only 3 bits, indicating a significant reduction of model memory footprint without a proportionate sacrifice of accuracy.

The approach also demonstrates applicability to object detection tasks. When applied within the SSD framework, quantization consistently yields notable results across different base networks. Noteworthy is the observation of more substantial degradations in architectures heavily relying on 1x1 convolutions unless quantized with higher bits.

Implications

The work's implications resonate both practically and theoretically. Practically, it minimizes the footprint of deep models, making them applicable in resource-constrained environments like IoT devices or mobile platforms. Theoretically, it lays groundwork for further exploration into ADMM-enhanced optimization heuristics, particularly their utility in tackling quantization problems while balancing convergence stability and computational efficiency.

Future Directions

Future research in extremely low bit neural networks could expand upon several domains: automated bit-width determination for different layers to optimize the regularization effects, the automation of the auxiliary variable adjustment process, and the fine-grained adaptiveness of algorithms to specific tasks and load profiles. The potential integration of these quantization techniques with broader aspects of neural network architecture design also remains a fertile area of exploration.

In conclusion, this work offers a rigorous analytical and computational framework conducive to advancing the deployment of deep neural networks in practical settings where efficiency and scalability are paramount. The contributions presented highlight the adaptive and robust nature of using ADMM in compression tasks, offering a sustainable path toward more efficient deep learning practices.