Controlling Information Capacity of Binary Neural Network (2008.01438v1)

Abstract: Despite the growing popularity of deep learning technologies, high memory requirements and power consumption are essentially limiting their application in mobile and IoT areas. While binary convolutional networks can alleviate these problems, the limited bitwidth of weights is often leading to significant degradation of prediction accuracy. In this paper, we present a method for training binary networks that maintains a stable predefined level of their information capacity throughout the training process by applying Shannon entropy based penalty to convolutional filters. The results of experiments conducted on SVHN, CIFAR and ImageNet datasets demonstrate that the proposed approach can statistically significantly improve the accuracy of binary networks.

• The paper presents an entropy-based regularization technique that penalizes information loss in binary convolutional filters to enhance prediction accuracy.
• The method computes Shannon entropy for binary weights and integrates an information loss penalty into the training process of binary neural networks.
• Empirical results on datasets like ImageNet, CIFAR-10, and SVHN demonstrate that sustaining higher information entropy significantly boosts BNN performance.

Controlling Information Capacity in Binary Neural Networks Through Entropy-Based Regularization

Introduction to Binary Neural Networks

Binary Neural Networks (BNNs) offer a promising solution for deploying deep learning models on resource-constrained devices, such as mobile phones and IoT devices, by reducing the memory footprint and computational requirements compared to traditional 32-bit models. These networks achieve this by representing weights and activations with binary values (+1 or -1), thus significantly decreasing the model size and accelerating inference. Despite these advantages, a major challenge with BNNs is the significant drop in prediction accuracy attributed to the reduced information capacity resulting from binarization.

Novel Approach to Maintaining Information Capacity

This paper introduces an innovative method to mitigate the accuracy loss in BNNs by controlling and stabilizing their information capacity throughout the training process. The proposed technique applies a Shannon entropy-based penalty to the convolutional filters, aiming to optimize the representation of information and, consequently, improve prediction accuracy. The primary contributions of this research include:

• The introduction of the concept of information capacity regularization in Convolutional Neural Networks (CNNs).
• Development of a new regularization technique utilizing Shannon entropy to create an information loss penalty specifically for binary CNNs.
• Empirical evidence demonstrating that maintaining a higher level of information entropy in convolutional filters enhances the prediction accuracy of BNNs.

Methodology for Entropy-Based Regularization

The methodology revolves around calculating the Shannon entropy for binary weights within convolutional filters and formulating an information loss penalty. This penalty is integrated into the overall loss function, guiding the training process to preserve a predefined level of information entropy in the filters. This approach necessitates modifications to conventional binary weight training procedures, incorporating the transition from real-valued to binary weights while maintaining differentiability of the new loss function.

Empirical Verification and Results

Experiments conducted on benchmark datasets such as SVHN, CIFAR-10, CIFAR-100, and ImageNet demonstrate the efficacy of the proposed method. When comparing BNNs trained with the information loss penalty to standard binary and full-precision networks, the modified BNNs exhibit statistically significant improvements in accuracy. For example, on the ImageNet dataset, incorporating the information loss penalty with an entropy target of 0.97 led to notable accuracy improvements across several network architectures, including ResNet-18 and DenseNet-121. These findings underscore the potential of the proposed entropy-based regularization technique to narrow the performance gap between BNNs and their full-precision counterparts.

Conclusion and Future Directions

The paper presents a pioneering approach to addressing the performance limitations of binary neural networks through entropy-based regularization. By controlling the information capacity within the network, this research provides a pathway to enhance the prediction accuracy of BNNs, thus extending their applicability in resource-constrained environments. Future work may explore the optimization of entropy targets for different architectures and tasks, as well as the integration of this regularization technique with other strategies aimed at improving the efficiency and accuracy of binary networks.

Acknowledgements

The authors express gratitude to Dr. Alexander Nikolaevich Filippov from the Russian Research Center of Huawei Technologies for his insightful contributions to this research.