AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets (2208.08084v2)

Abstract: This paper studies the Binary Neural Networks (BNNs) in which weights and activations are both binarized into 1-bit values, thus greatly reducing the memory usage and computational complexity. Since the modern deep neural networks are of sophisticated design with complex architecture for the accuracy reason, the diversity on distributions of weights and activations is very high. Therefore, the conventional sign function cannot be well used for effectively binarizing full-precision values in BNNs. To this end, we present a simple yet effective approach called AdaBin to adaptively obtain the optimal binary sets ${b_1, b_2}$ ($b_1, b_2\in \mathbb{R}$) of weights and activations for each layer instead of a fixed set (\textit{i.e.}, ${-1, +1}$). In this way, the proposed method can better fit different distributions and increase the representation ability of binarized features. In practice, we use the center position and distance of 1-bit values to define a new binary quantization function. For the weights, we propose an equalization method to align the symmetrical center of binary distribution to real-valued distribution, and minimize the Kullback-Leibler divergence of them. Meanwhile, we introduce a gradient-based optimization method to get these two parameters for activations, which are jointly trained in an end-to-end manner. Experimental results on benchmark models and datasets demonstrate that the proposed AdaBin is able to achieve state-of-the-art performance. For instance, we obtain a 66.4% Top-1 accuracy on the ImageNet using ResNet-18 architecture, and a 69.4 mAP on PASCAL VOC using SSD300. The PyTorch code is available at \url{https://github.com/huawei-noah/Efficient-Computing/tree/master/BinaryNetworks/AdaBin} and the MindSpore code is available at \url{https://gitee.com/mindspore/models/tree/master/research/cv/AdaBin}.

Enhancing Binary Neural Networks with Adaptive Binary Sets: A Review of AdaBin

The paper "AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets" introduces a novel approach to optimize Binary Neural Networks (BNNs) by employing Adaptive Binary Sets. This method addresses the limitations associated with traditional BNNs, which binarize weights and activations to a fixed set of values, typically $[-1, +1]$. By adapting the binary values within each layer, AdaBin enhances representational capacity, bridging the performance gap between binary and full-precision neural networks.

Overview and Method

The paper acknowledges the efficiency of BNNs in terms of memory usage and computational efficiency due to the binarization of weights and activations into 1-bit values. However, the simplicity of using a fixed binary set restricts the potential to accurately represent diverse distributions of weights and activations across layers. The paper proposes AdaBin to dynamically adjust the binary parameters, thereby improving performance without significantly increasing computational overhead.

AdaBin introduces two key innovations in the binarization process:

1. Weight Equalization: AdaBin leverages statistical analysis to align the center of binary weight distribution to the real-valued distribution, minimizing Kullback-Leibler divergence. This approach results in a better match between binarized and full-precision weights, leading to improved accuracy.
2. Gradient-Based Activation Optimization: For activations, AdaBin employs a gradient-based method to optimize binary parameters during training. Learning these parameters end-to-end facilitates achieving optimal binary sets that adapt to varying input distributions.

The method maintains computational efficiency, offering approximately $60.85\times$ acceleration and $31\times$ memory savings, comparable to conventional BNNs that rely on XNOR and BitCount operations.

Experimental Results

AdaBin demonstrates superior performance across various benchmarks. On the ImageNet dataset using the ResNet-18 architecture, AdaBin achieves a Top-1 accuracy of 63.1%, outperforming existing methods like ReCU by 2.1%. Furthermore, in experiments involving binary-specific architectures such as BDenseNet and MeliusNet, AdaBin consistently improves accuracy with negligible additional computational costs.

On the PASCAL VOC dataset for object detection, AdaBin significantly enhances mAP scores, demonstrating improved generalization capabilities in complex tasks compared to both general and specialized binary network methods.

Implications and Future Directions

This research highlights the potential to improve BNNs by treating binary sets as dynamic parameters rather than fixed constants. As a result, AdaBin offers a viable pathway to enhance model performance while maintaining efficiency. This adaptability makes AdaBin advantageous for deployment on resource-constrained devices and real-time applications.

Future explorations might involve extending the AdaBin framework to other network architectures and tasks beyond vision, and further optimizing the binarization process to include considerations of energy efficiency. Integrating AdaBin with neural architecture search techniques may also yield insightful advances, potentially leading to new benchmarks in low-power AI deployment.

In summary, the paper presents a concrete advancement in the domain of neural network quantization, underscoring how adaptive methodologies can substantially enhance performance while adhering to the constraints of binary computation.