Balanced Weight Binarization

• Balanced weight binarization methods center and scale weights to enforce a near 50:50 ±1 distribution, effectively minimizing quantization error.
• Dual-bell transformation techniques reshape weight distributions into symmetric bimodal forms, improving fidelity in both vision and language models.
• Iterative training and per-layer calibration maintain high accuracy while enabling significant computational savings in on-device and large-scale deployments.

Balanced weight binarization is a family of methods for quantizing neural network weights to 1-bit values (±1) while explicitly optimizing the distribution of binary assignments to maximize model expressiveness and minimize quantization error. The central principle is to enforce or induce a close-to-balanced occurrence of +1 and –1 assignments per layer or channel, preserving information entropy and reducing representational collapse. This approach addresses the systematic information loss observed in naive sign-based binarization, particularly when the original weight distributions are highly concentrated or biased.

1. Motivation and Theoretical Principles

Standard sign-based binarization maps real-valued weights to binary values solely via their sign. In layers where the floating-point weights are distributed quasi-Gaussian (unimodal, “single-bell”) and centered near zero—a common scenario in LLMs and CNNs—this operation leads to two fundamental issues: (a) significant quantization error, as most weights near zero may be flipped incorrectly; and (b) imbalanced +1/–1 assignment, where one value dominates, decreasing entropy and expressive capacity. High representational entropy, as measured by the bitwise Shannon entropy $H(p) = -p\log p - (1-p)\log(1-p)$ with $p$ the proportion of +1s, is only maximized at $p = ½$ (Ye et al., 18 Jun 2025, Shen et al., 2019).

Balanced weight binarization methods thus aim to minimize sign-quantization error and enforce near-zero mean in the assignment of binary weights, either statistically (in expectation) or structurally (by transformation).

2. Balanced Binarization via Centering and Scaling

A canonical approach, introduced in "Balanced Binary Neural Networks with Gated Residual" (BBG), centers the real-valued proxy weights before binarization to ensure a zero-mean prior to the sign operation:

$w = v - \frac{1}{d}1(1^\top v)$

where $v \in \mathbb{R}^d$ are the trainable real-valued parameters, $1$ is the all-ones vector, and $d$ is the weight vector length. After centering, a per-layer or per-filter scaling factor $\alpha = \frac{1}{d}\sum_{i=1}^d|w_i|$ is applied such that the binary weights are

$w^b = \alpha \cdot \operatorname{sign}(w)$

This drives the binarized weights toward balanced ±1 assignment and preserves the magnitude energy to approximate the floating-point kernel's statistics (Shen et al., 2019).

This routine can be implemented generically in convolutional and dense layers, with centering and scaling performed at each forward pass, and gradients propagated using straight-through estimators (STE). Centering does not guarantee perfect balance per channel but ensures that, in expectation, the binary weights have zero mean, which is sufficient to maximize entropy as the network is trained.

3. Inducing Dual-Bell Distributions for Near-Ideal Binarization

DBellQuant introduces an alternative which directly transforms the underlying weight distribution from a single-bell (unimodal, mean-zero) to a dual-bell (bimodal, symmetric around zero) form prior to binarization (Ye et al., 18 Jun 2025). Given $W \in \mathbb{R}^{n \times m}$ (a linear layer's weight) and channel-scaling vector $p$0, the per-channel weights are transformed:

$p$1

and activations inversely scaled:

$p$2

leaving the layer’s overall function unchanged.

The scaling vector $p$3 is initialized heuristically, e.g.,

$p$4

and then trained to shape $p$5 into two distinct, symmetric clusters via dual-centric loss terms:

• $p$6 (penalizes deviation from two symmetric centers)
• $p$7 (normalizes deviation relative to cluster amplitude)

This procedure ensures that the sign operation after centering (i.e., per-channel mean subtraction and then sign) will allocate weights as near as possible to 50:50 ±1, and that the real-valued weights are close to their binarized values, thereby minimizing quantization error.

4. Balanced Binarization in Iterative Training Schedules

Beyond per-layer statistical balancing, the full training schedule can influence balanced binarization effectiveness. The iterative layer-binarization approach introduces binary quantization incrementally, binarizing one layer at a time guided by a sensitivity metric (validation error when a layer is individually binarized), recalculating batch normalization statistics, and allowing preceding float-precision layers to adapt to quantization noise (Lan, 2021).

Batch normalization before and after each binarized layer is essential to stabilize the layer statistics around zero, improving ±1 balance. This staged approach consistently yields lower final test error than naive, all-at-once binarization, especially in deeper networks.

5. Empirical Performance and Applications

Balanced weight binarization methods yield demonstrably improved accuracy and efficiency across vision and LLMs.

For convolutional neural networks (ResNet, VGG) and detection tasks (SSD), BBG achieves higher accuracy than prior 1-bit schemes (e.g., XNOR-Net, DoReFa-Net, Bi-Real Net) at equivalent or reduced FLOPs. For example, on ImageNet with ResNet-18, BBG reaches 57.6% top-1 accuracy versus 51.2% (XNOR-Net) and 56.4% (Bi-Real Net); and on Pascal VOC detection, BBG obtains 68.5% mAP (SSD-VGG16, 1-bit) versus the 74.3% full-precision baseline (Shen et al., 2019).

DBellQuant extends these gains to LLMs. It achieves a perplexity of 14.39 on LLaMA2-13B (6-bit activation), compared to BiLLM’s 21.35 (without activation quantization), and preserves >90% of full-precision QA accuracy on multiple zero-shot benchmarks in the 1-bit/6-bit regime. The mean binarization error is minimized, and the ±1 assignment is consistently balanced across layers and channels (Ye et al., 18 Jun 2025).

In on-device inference scenarios (e.g., ResNet-18 on Raspberry Pi 3B), balanced binarization yields a ≈5.8× speedup over float inference with minimal accuracy sacrifice (Shen et al., 2019).

6. Methodological Comparisons and Limitations

Naive binarization methods that apply sign(x) without explicit centering or balancing mechanisms tend to produce imbalanced binary weights, leading to filter collapse and reduced representational power. Previous enhancements, such as learnable thresholds or per-layer scaling, do not address information entropy directly and remain susceptible to bias in ±1 assignment (Shen et al., 2019).

Balanced binarization methods, via either centering/scaling or explicit dual-bell transformation, enforce maximal entropy in binary weight vectors and reduce quantization error, a critical factor when compressing models for edge and large-scale deployment. However, the transformation and training steps incur additional (albeit minimal) calibration or computational overhead, especially in LLMs.

A plausible implication is that further research may explore fully data-driven transformations for even more robust entropy-preserving binarization in more heterogeneous architectures.

7. Practical Deployment and Future Directions

All balanced weight binarization methods discussed can be integrated into standard training or post-training workflows with negligible adaptation. The simple two-step centering/scaling paradigm is universally applicable in convolutional and linear layers, while the dual-bell transformation in DBellQuant is compatible with high-dimensional transformer weights and post-training quantization.

Down-sample, first, and last layers typically remain full-precision due to sensitivity, whereas all remaining layers use balanced binarization for maximal compression. In practice, these methods significantly close the performance gap to float-precision base models while achieving large reductions in memory and computation.

Advancements are ongoing in making the balancing process even more adaptive and automated, with further improvements in hardware/software co-design anticipated, especially for ultra-LLMs and high-frequency vision inference on edge devices.

---

Key references:

• "DBellQuant: Breaking the Bell with Double-Bell Transformation for LLMs Post Training Binarization" (Ye et al., 18 Jun 2025)
• "Balanced Binary Neural Networks with Gated Residual" (Shen et al., 2019)
• "Iterative Training: Finding Binary Weight Deep Neural Networks with Layer Binarization" (Lan, 2021)