Rotated Binary Neural Network (2009.13055v3)

Abstract: Binary Neural Network (BNN) shows its predominance in reducing the complexity of deep neural networks. However, it suffers severe performance degradation. One of the major impediments is the large quantization error between the full-precision weight vector and its binary vector. Previous works focus on compensating for the norm gap while leaving the angular bias hardly touched. In this paper, for the first time, we explore the influence of angular bias on the quantization error and then introduce a Rotated Binary Neural Network (RBNN), which considers the angle alignment between the full-precision weight vector and its binarized version. At the beginning of each training epoch, we propose to rotate the full-precision weight vector to its binary vector to reduce the angular bias. To avoid the high complexity of learning a large rotation matrix, we further introduce a bi-rotation formulation that learns two smaller rotation matrices. In the training stage, we devise an adjustable rotated weight vector for binarization to escape the potential local optimum. Our rotation leads to around 50% weight flips which maximize the information gain. Finally, we propose a training-aware approximation of the sign function for the gradient backward. Experiments on CIFAR-10 and ImageNet demonstrate the superiorities of RBNN over many state-of-the-arts. Our source code, experimental settings, training logs and binary models are available at https://github.com/lmbxmu/RBNN.

• The paper introduces a novel rotation-based method that addresses angular bias in binary neural networks, significantly reducing quantization errors.
• It employs a bi-rotation framework and dynamic weight adjustment to lower computational complexity while refining weight alignment and gradient propagation.
• Empirical results on CIFAR-10 and ImageNet demonstrate superior accuracy compared to traditional binary neural network approaches.

Overview of Rotated Binary Neural Network

The paper, "Rotated Binary Neural Network," addresses the persistent challenge in the field of Binary Neural Networks (BNNs): the significant performance degradation due to high quantization errors. Traditional BNNs simplify deep neural networks by reducing the weight precision to binary values, facilitating execution on resource-constrained devices. Despite these advantages, BNNs often suffer from large quantization errors, primarily due to discrepancies between the full-precision weights and the binarized weights. Previous methods have concentrated on mitigating the norm disparity, with little attention paid to angular biases contributing to these errors.

Key Contributions

The paper introduces a novel approach called the Rotated Binary Neural Network (RBNN) to tackle angular bias, a previously underexplored avenue in BNN optimization. The proposed method integrates a rotation matrix into the network's training regime, aligning the angle between full-precision and binary weight vectors. This alignment helps minimize the angular bias, subsequently reducing quantization errors. A bi-rotation framework is utilized, leveraging two smaller rotation matrices over a single large one, significantly reducing computational complexity.

Additionally, RBNN suggests a dynamic weight adjustment strategy to avoid sub-optimal local minima during training. This involves an adjustable rotated weight vector approach, which refines the weight modifications using a learning parameter, further minimizing angular bias. RBNN demonstrates substantially increased weight flips (approximately 50%) compared to traditional methods, maximizing information gain during training.

Moreover, RBNN introduces a training-aware gradient approximation for the sign function, accommodating the propagation of gradients during backward pass and enhancing the training of BNNs.

Results and Implications

The empirical results on CIFAR-10 and ImageNet validate the efficacy of RBNN, showcasing superior accuracy over several state-of-the-art approaches. For example, on CIFAR-10 using the ResNet-18 architecture, RBNN achieves a notable improvement in accuracy compared to established methods such as IR-Net. Similarly, when tested on ImageNet with architectures like ResNet-18, RBNN not only surpasses but also sets new benchmarks in binary network classification performance.

From a practical perspective, RBNN offers an effective solution for deploying high-performance neural networks on devices with limited computing resources, such as mobile devices and IoT devices. The reduction in quantization error without increasing computational overhead makes RBNN a compelling choice for real-time applications requiring efficient model inference.

In terms of theoretical implications, this work opens new research avenues by emphasizing the importance of addressing angular bias in BNNs. It suggests potential crossovers between geometric transformations and network optimization, encouraging further exploration into bi-rotation or similar techniques in other neural network paradigms.

Future Directions

Future developments might explore the application of RBNN principles in different domains or network types, potentially augmenting the design of Spiking Neural Networks or other quantized architectures. Additionally, investigations could extend to more complex weight transformation schemes or adaptive rotation matrices guided by more sophisticated learning rules. An area of interest could also be developing more comprehensive frameworks that incorporate auxiliary learning objectives tailored to various tasks or data distributions.

In conclusion, the Rotated Binary Neural Network provides a robust framework for enhancing BNN accuracy, offering meaningful insights for both the broader machine learning community and the deployment of deep learning models in constrained environments.