XNOR-Net++: Advanced Binary Neural Networks

• The paper introduces a learnable scale factor fusion that replaces static analytic scales, optimizing quantization error and boosting classification accuracy.
• It presents architectural innovations including real-activation shortcuts to reduce information loss during binarization, thereby enhancing representational capacity.
• Empirical results on ImageNet show that XNOR-Net++ significantly narrows the accuracy gap between binary and real-valued networks without increasing computational cost.

XNOR-Net++ is an advanced class of binary neural network (BNN) architectures and training methodologies designed to bridge the accuracy gap between 1-bit convolutional networks and their real-valued counterparts while retaining the memory and computational benefits of extreme quantization. The term primarily refers to two closely related but technically distinct contributions: the "Improved Binary Neural Networks" approach that introduces learnable scale factors for binary convolutions (Bulat et al., 2019), and the Bi-Real Net strategy that augments standard 1-bit CNNs with identity shortcuts and advanced training techniques (Liu et al., 2018). Both approaches are motivated by the limitations of XNOR-Net, which achieves efficient inference but suffers notable accuracy degradation due to aggressive binarization.

1. Mathematical Foundations and Motivation

XNOR-Net approximates real-valued convolutions with bitwise operations on binarized weights and activations, modulated by analytically computed real-value scaling factors. In XNOR-Net, a convolutional layer computes: $$Y \approx \bigl(\mathrm{sign}(W) \otimes \mathrm{sign}(X)\bigr) \odot (K\alpha)$$ where $W$ and $X$ are the real weights and activations, $K$ and $\alpha$ are input- and weight-derived scale factors, $\otimes$ denotes XNOR convolution, and $\odot$ is pointwise multiplication.

The analytic approach computes $K$ and $\alpha$ to minimize the quantization error $\|X * W -$binary-conv$W$0, yet these factors are independent of the downstream task loss and cannot be refined during end-to-end training. Furthermore, recomputation of $W$1 for each input incurs additional runtime cost, and no gradient signal flows directly to adapt the scale parameters.

XNOR-Net++ (Bulat et al., 2019) recognizes that analytic scaling is sub-optimal for classification performance. Instead, it proposes a learnable scale tensor $W$2 that replaces $W$3 and $W$4, enabling joint optimization with all other network parameters. Similarly, Bi-Real Net (Liu et al., 2018) identifies inherent representational bottlenecks when every block’s post-activation is immediately binarized, motivating architectural and training enhancements beyond scale tuning.

2. Architectural and Algorithmic Innovations

Learnable Scale Factor Fusion (XNOR-Net++):

The core change is to replace the product of analytic scales $W$5 by a single tensor $W$6, learned discriminatively via backpropagation. This tensor can take several forms:

• One scale per channel: $W$7.
• One scale per channel & spatial location: $W$8.
• Low-rank outer product: $W$9.
• Full rank-1 factorization (channel × height × width): $X$0, with broadcast multiplication.

All configurations maintain the test-time computational budget, as the scaling factors can be fused.

Real-Activation Shortcut (Bi-Real Net):

To address information loss due to immediate binarization, Bi-Real Net augments each block with an identity shortcut carrying real-valued activations (post-BatchNorm, pre-Sign) forward. The computation per residual block becomes: $X$1 This identity addition increases both the pre-binarization representational capacity and the number of distinguishable post-binarization partitions, offering a substantial reduction in information loss over standard 1-bit CNNs.

3. Advanced Training Methodologies

XNOR-Net++ and Bi-Real Net both rely on differentiable approximations to the non-differentiable sign function, typically using a straight-through estimator (STE) that passes gradients through when $X$2. XNOR-Net++ applies this in both activation and weight branches of the backward pass and enables end-to-end learning of the fused scale tensor.

Bi-Real Net introduces three further training innovations:

• Tight Approximation of $X$3: The discontinuity is fit with a piecewise quadratic, whose derivative is a tent-shaped function, yielding a closer gradient match and more stable convergence.
• Magnitude-Aware Weight Gradient: Instead of updating real weights solely based on their sign, the update rule includes the magnitude of the real-valued weight, allowing more responsive training dynamics and effective sign flipping.
• Clip-Based Pretraining: Replacing ReLU with an activation clip $X$4 in pretraining aligns the activation statistics with those attainable by sign binarization, enabling more effective initialization and transfer to the binary regime.

4. Empirical Results and Ablation Analyses

XNOR-Net++ achieves significant improvements on standard large-scale visual benchmarks. On ImageNet with ResNet-18:

Method	Top-1 (%)	Top-5 (%)
XNOR-Net	51.2	73.2
XNOR-Net++, Case 4	57.1	79.9
Real-valued	69.3	89.2

Ablation studies confirm that richer scale parameterizations (rank-1 channel × height × width) deliver the best tradeoff between parameter count and accuracy. Quantization-error metrics ($X$5 distance to real convolution output) also favor the learned fusion approach.

Bi-Real Net demonstrates similar enhancements:

• Bi-Real-18 achieves 56.4% Top-1 (ImageNet), versus XNOR-Net-18's 51.2%.
• Ablations of training tricks show additive gains: "tent" gradients (+12% rel.), magnitude-aware weights (+23% rel.), and clip-init (+13% rel.), closely recovering full-precision ResNet ceilings.
• The added real-value addition per block has negligible compute and memory cost (model size 33.6 Mbit, ≈11.1x smaller than full-precision baseline) (Liu et al., 2018).

5. Implementation, Computational Cost, and Practical Considerations

Both XNOR-Net++ and Bi-Real Net maintain the efficient bitwise convolutional kernel (XNOR + popcount) crucial for BNN deployments. The learned scale fusion in XNOR-Net++ does not increase inference-time computational complexity, as the scaling tensor can be pre-computed and fused before deployment. Bi-Real Net's extra identity add is elementwise and performed on short intermediate real activations.

Standard optimization uses Adam with conservative weight decay; training proceeds for 80 epochs with batch sizes consistent with the backbone (256 for ResNet-18). The first and last convolutional layers remain real-valued to preserve input-output fidelity.

6. Relationship to Broader Quantization Research

XNOR-Net++ addresses fixed 1-bit quantization, but related research generalizes this approach to mixed-precision and multi-branch binary encoding. For example, decomposition into multiple binary branches at arbitrary precision (1–8 bits) with learnable or decomposable scaling (e.g., via encoding schemes and MBitEncoder approaches) further enhances the accuracy vs. compression tradeoff and supports hardware-friendly deployment (Sun et al., 2021). This suggests that XNOR-Net++ forms part of a larger trend towards tunable, hardware-amenable quantized networks that balance accuracy, model size, and energy use.

7. Impact and Best Practices

XNOR-Net++ and Bi-Real Net effectively close a substantial portion of the accuracy gap between binary and real-valued networks without substantially increasing memory footprint or compute cost. Both approaches are compatible with common architectures such as AlexNet and ResNet-18 and are empirically validated on large-scale tasks. The key recommendation is to employ a learned, high-rank scale tensor (Case 4) for maximal accuracy within fixed resource budgets (Bulat et al., 2019). In the context of 1-bit CNNs, augmenting with real-activation shortcuts and refined training techniques further minimizes binarization loss (Liu et al., 2018). These insights clarify the optimal regimes for binary neural inference in practical settings and guide further research into scalable quantization.