XNOR-Net++: Improved Binary Neural Networks (1909.13863v1)

Abstract: This paper proposes an improved training algorithm for binary neural networks in which both weights and activations are binary numbers. A key but fairly overlooked feature of the current state-of-the-art method of XNOR-Net is the use of analytically calculated real-valued scaling factors for re-weighting the output of binary convolutions. We argue that analytic calculation of these factors is sub-optimal. Instead, in this work, we make the following contributions: (a) we propose to fuse the activation and weight scaling factors into a single one that is learned discriminatively via backpropagation. (b) More importantly, we explore several ways of constructing the shape of the scale factors while keeping the computational budget fixed. (c) We empirically measure the accuracy of our approximations and show that they are significantly more accurate than the analytically calculated one. (d) We show that our approach significantly outperforms XNOR-Net within the same computational budget when tested on the challenging task of ImageNet classification, offering up to 6\% accuracy gain.

• The paper introduces a novel method that learns combined scaling factors via backpropagation to enhance the trade-off between efficiency and accuracy.
• It details multiple formulations, from channel-wise to spatially decomposed factors, optimizing binary approximations while reducing parameter count.
• Empirical evaluations on ImageNet show up to a 6% increase in top-1 accuracy, demonstrating the practical benefits for resource-constrained applications.

An Examination of XNOR-Net++: Enhanced Training Algorithms for Binary Neural Networks

The paper "XNOR-Net++: Improved Binary Neural Networks" by Adrian Bulat and Georgios Tzimiropoulos presents an advancement in the domain of binary neural networks (BNNs), specifically addressing the training methodology to improve the trade-off between model efficiency and accuracy. The authors propose a new training algorithm that discerns the computation of scaling factors in BNNs, offering substantial performance gains over existing methods like XNOR-Net.

Key Contributions and Methodology

The authors introduce an innovative approach to computing the scaling factors used in BNNs. The central claim is that previous analytic computations for these factors, as seen in XNOR-Net, are sub-optimal. Instead, the authors suggest these should be learned discriminatively through backpropagation, merging both activation and weight scaling factors into a singular learned factor. This fusing approach is shown to optimize the scaling factors concerning the task-specific loss functions rather than focusing on the errors from binary approximations.

The paper outlines several methodologies for constructing the shape of these scaling factors while maintaining the computational budget. Four primary cases are examined, each involving different formulations of the scaling factor, denoted as $\Gamma$:

• Case 1 considers $\Gamma$ as a singular channel-wise scaling factor.
• Case 2 expands $\Gamma$ to consider spatial dimensions, capturing more intricate data representations.
• Case 3 and Case 4 further refine this by decomposing spatial dimensions into separate factors, reducing parameter count and enhancing model expressivity.

Empirical Evaluation and Performance

The research empirically measures the efficacy of the proposed scaling factor methods on tasks such as ImageNet classification. A significant improvement is reported, with the proposed methodology achieving an up to 6% increase in top-1 accuracy over XNOR-Net under the same computational constraints. A detailed empirical analysis of approximation accuracy further substantiates these results. The paper also discusses the theoretical gains concerning computation efficiency, demonstrating that the proposed approach virtually retains the speed props of previous BNNs while delivering enhanced accuracy.

Theoretical and Practical Implications

The development of more accurate binary neural networks is pivotal for deploying deep learning models on resource-constrained devices, such as mobile phones and embedded systems. The proposed improvement by Bulat and Tzimiropoulos represents a significant step forward in this context. By demonstrating that scaling factors in BNNs can be learned effectively, the authors open new avenues for tailoring BNN training procedures that are both computationally efficient and accurate.

Additionally, this work underscores a general lesson in machine learning: the value of tailoring simpler, often overlooked elements like scaling and quantization in network architectures to achieve considerable improvements without more complex model structures.

Future Directions

Future developments in BNN training may focus on even more fine-grained strategies for learning model parameters in binary configurations. Expanding on notions of learned scaling factors, future work could explore adaptive learning rates for BNN parameters, further refinements in network architecture optimized for binary constraints, or leveraging these methodologies in transferring knowledge across domains.

Overall, XNOR-Net++ represents a meaningful progression in making high-performance neural networks accessible across diverse and resource-limited application environments, promising both theoretical insights and practical benefits for the AI community.