Gated Residuals in Binary Networks

• The paper introduces the BBG framework that combines balanced weight binarization with a gated residual path to recover critical floating-point information lost in 1-bit activations.
• It employs a mathematically principled constrained entropy maximization strategy to ensure balanced binary filter distributions and effective gradient flow.
• The approach integrates seamlessly into architectures like ResNet, VGG, and SSD, achieving improved accuracy and efficiency on benchmarks like CIFAR and ImageNet.

Gated residuals in binary networks address critical challenges in extreme quantization, where both weights and activations are restricted to 1-bit for optimal efficiency. The Balanced Binary Neural Networks with Gated Residual (BBG) framework introduces mathematically principled modifications to the binarization process: (1) balanced weight binarization, which enforces maximal entropy of binary filters, and (2) a gated residual path that carries forward a modulated floating-point signal to restore information otherwise lost during activation binarization. These techniques can be integrated as generic modules within prevailing deep learning architectures, such as VGG, ResNet, and SSD, delivering consistent gains in accuracy and efficiency relative to prior binary neural networks (Shen et al., 2019).

1. Maximizing Information Entropy with Balanced Weight Binarization

In conventional binary networks, a full-precision weight tensor $W$ is binarized as $W^b = \alpha \, \mathrm{sign}(W)$, with $\alpha$ a per-layer scaling coefficient set by the mean absolute value of $W$. This process typically yields imbalanced distributions of $+1$ and $-1$ in $W^b$, undermining the expressive capacity of the filters. To address this, BBG formulates binarization as a constrained entropy maximization:

$$\max_{W^b_i\in\{+\alpha, -\alpha\} } H\left( \frac{1}{n} \#\{i:W^b_i=+\alpha\} \right) \quad \text{s.t.} \quad \sum_{i=1}^n W^b_i = 0$$

where $H(p) = -p\log p - (1-p)\log(1-p)$ denotes Bernoulli entropy, maximized at $p=0.5$ under the constraint that half the weights are $W^b = \alpha \, \mathrm{sign}(W)$0 and half are $W^b = \alpha \, \mathrm{sign}(W)$1. Exact combinatorial optimization is intractable, so BBG introduces a differentiable centering operation: for floating-point proxy $W^b = \alpha \, \mathrm{sign}(W)$2, the centered weights are $W^b = \alpha \, \mathrm{sign}(W)$3, guaranteeing $W^b = \alpha \, \mathrm{sign}(W)$4 in floating-point. The binarized weights become $W^b = \alpha \, \mathrm{sign}(W)$5 with $W^b = \alpha \, \mathrm{sign}(W)$6. This mechanism ensures maximal filter diversity and information retention during binarization, and enables direct gradient flow to $W^b = \alpha \, \mathrm{sign}(W)$7 during backpropagation (Shen et al., 2019).

2. Gated Residual Path for Floating-Point Activation Restoration

Activation binarization in standard binary networks severely limits representational fidelity by discarding channel-wise variations through a hard $W^b = \alpha \, \mathrm{sign}(W)$8 operation. BBG introduces a parallel gated residual branch for each layer: for input $W^b = \alpha \, \mathrm{sign}(W)$9, the binarized activation $\alpha$0 proceeds through a binary convolution and batch normalization, $\alpha$1. The gated residual is computed as $\alpha$2, with $\alpha$3 the per-channel learnable gates (initialized at 1.0). The final output is $\alpha$4. This additive path restores floating-point activation detail and provides a direct gradient path, $\alpha$5, reducing the mismatch induced by the straight-through estimator. The additional compute, $\alpha$6 multiplies per layer, is negligible relative to conventional convolution costs (Shen et al., 2019).

3. Modular Integration within Deep Learning Architectures

Both balanced binarization and gated residual modules can be slotted into standard convolutional blocks. In a ResNet configuration, each "Conv→BN→ReLU" sequence is replaced by:

• Center weight tensor: $\alpha$7
• Binarize: $\alpha$8
• Forward activations: binarize $\alpha$9
• Binary convolution, batch normalization: $W$0
• Gated residual: $W$1
• Fusion: $W$2

In VGG and SSD backbones, each convolutional layer is substituted similarly, with binarization typically omitted for first, last, or downsampling layers. This modular design allows seamless adoption without extra channels, additional layers, or architectural modification (Shen et al., 2019).

4. Training and Hyperparameter Regimes

BBG networks are trained from scratch with standard objectives: cross-entropy loss for classification and typical SSD losses (localization and confidence) for detection. Stochastic Gradient Descent with momentum 0.9 and weight decay $W$3 is used. Conventional learning-rate schedules are retained: e.g., starting at $W$4 (CIFAR) or $W$5 (ImageNet), with learning rate reductions at 30%, 60%, and 90% of total epochs. Important hyperparameters include:

Hyperparameter	Value	Purpose
Gate-init	$W$6	All per-channel gates initialized to $W$7
Bit width	1 bit (weights, acts)	Full binarization in interior layers
Exclusions	No binarization on first, last, or downsampling layers	Preserves critical precision at bottlenecks

No further regularization beyond that employed in the original backbone (dropout, batch norm) is required (Shen et al., 2019).

5. Empirical Evaluation and Comparative Performance

Table: Performance indicators for BBG against baselines.

Dataset/Task	Network	Baseline (1-bit)	BBG-Only	BBG (joint)	Full-precision
CIFAR-10 Classify	ResNet-20	84.13%	84.71% (bal), 84.89% (gate)	85.34%	92.1%
CIFAR-100 Classify	ResNet-20	53.37%	-	55.62%	-
ImageNet Classify	ResNet-18	≈51% (XNOR-Net)	-	60.8% top-1	69.3%
Pascal VOC Detect	SSD-ResNet34	55.1% (XNOR), 59.5% (TBN)	-	62.8% mAP	75.5%
Pascal VOC Detect	VGG-16	<68.5% (DenseNet-bin)	-	68.5% mAP	-

Increasing network width or resolution further narrows the accuracy gap, and BBG reaches near full-precision performance at lower memory and compute budgets. On resource-constrained devices (Raspberry Pi 3B, ARM), BBG achieves $W$8 speedup over 32-bit ResNet-18, with inference at 251 ms per ImageNet image; the minor computational overhead from balancing and gating is outweighed by bitwise acceleration (Shen et al., 2019).

6. Broader Significance and Theoretical Perspective

BBG demonstrates that theorem-inspired constraints on binarization—maximizing entropy for weights and restoring information with per-channel moderately sized floating-point residuals—jointly yield accuracy, speed, and memory advantages over prior binarization schemes, such as XNOR-Net and TBN. The framework is architecture-agnostic, applicable to both classification and detection tasks, unifying efficient quantization with restoration of essential representational flexibility. The empirical ablations underscore the complementary benefits of balancing and gating, and the plug-in nature of the modules supports broad adoption in highly resource-constrained deep learning deployment scenarios (Shen et al., 2019).