Binary Weight Multi-Bit Activation (BWMA)

• BWMA is a quantization approach where binary weights simplify arithmetic while multi-bit activations preserve representational power.
• It is applied across diverse architectures such as CNNs, SNNs, Transformers, and LLM PTQ to optimize memory, speed, and energy use.
• Representative instantiations like BitSplit-Net and ReverB-SNN demonstrate that BWMA can achieve competitive accuracy with notable hardware and runtime benefits.

Searching arXiv for the cited BWMA-related papers to ground the article in current literature. {"query": "\"Binary Weight Multi-Bit Activation\" OR BWMA OR ReverB-SNN OR BWLA OR BWTA", "max_results": 10} {"query": "\"ReverB-SNN\" (Guo et al., 9 Jun 2025)", "max_results": 5} Binary Weight Multi-Bit Activation (BWMA) is a quantization regime in which weights are binary, typically in $\{-1,+1\}$, while activations remain multi-bit rather than fully binary. Across CNNs, Transformers, LLM post-training quantization, spiking neural networks, and compute-in-memory deployments, BWMA is used to retain the memory and arithmetic advantages of binary weights while reducing the accuracy degradation associated with aggressively quantized activations. Representative realizations include BitSplit-Net, which propagates $K$ binary activation bit-planes independently; ReverB-SNN, which uses binary weights and real-valued spike activations; BWTA, which uses binary weights and ternary or boolean activations; BWLA, which targets W1A6 and W1A4 LLM inference; Efficiera Residual Networks, which use binary weights and 2-bit activations throughout a ResNet-compatible architecture; and CIM-oriented BWMA methods that pair 1-bit weight storage with 4-bit activations (Kim et al., 2019, Guo et al., 9 Jun 2025, Ding et al., 5 Apr 2026, Zhao et al., 1 May 2026, Takahashi et al., 2024, Zhou et al., 29 Aug 2025).

1. Motivation and conceptual basis

A recurring motivation for BWMA is the empirical observation that reducing activation precision often harms accuracy more severely than reducing weight precision. In quantized ANNs, low-bit activations are reported to degrade performance more severely than low-bit weights, and ReverB-SNN explicitly adopts this premise to argue that SNNs suffer acutely because traditional binary spike activation maps restrict the information carried forward in time and across layers (Guo et al., 9 Jun 2025). In the ANN setting, the same logic appears in probabilistic discrete training: binary weights drastically cut memory and enable XNOR+bitcount arithmetic, while multi-bit activations often recover accuracy and still permit efficient bit-plane accumulations (Berger et al., 2023).

Within this regime, the binary side and the multi-bit side play different roles. Binary weights primarily compress model storage and simplify arithmetic. Multi-bit activations preserve representational power, dynamic range, or zero-preserving behavior that would otherwise be lost under full binarization. This division is explicit in several domains. BitSplit-Net treats multi-bit activations as collections of binary bit-planes that propagate independently through a network with shared weights; BWTA preserves a zero activation level to reduce zero-point distortion in Transformers; BWLA argues that end-to-end acceleration requires not only 1-bit weights but also low-bit activations; and CIM-oriented BWMA identifies 4-bit activations as the balance point between hardware cost and model performance (Kim et al., 2019, Ding et al., 5 Apr 2026, Zhao et al., 1 May 2026, Zhou et al., 29 Aug 2025).

BWMA is therefore not a single algorithm. It is a family of design choices in which binary weights are combined with a non-binary activation representation whose exact form depends on the model class and hardware target. In some cases the activation alphabet is uniform $k$-bit; in some it is ternary with an explicit zero; in some it is 2-bit ReLU-like integer activation; and in ReverB-SNN it is real-valued spike amplitude gated by threshold (Guo et al., 9 Jun 2025, Ding et al., 5 Apr 2026, Takahashi et al., 2024).

2. Quantization formulations and training mechanisms

The canonical binary-weight form uses a sign map,

$w_b = \mathrm{sign}(w)\in\{-1,+1\}.$

This appears directly in ReverB-SNN, BitSplit-Net, BWTA, and several hardware mappings, often with an associated scale factor such as $\alpha$, $s_W$, or per-channel mean-absolute scaling (Guo et al., 9 Jun 2025, Kim et al., 2019, Ding et al., 5 Apr 2026). In ERNs, the binary form is written as

$W_b = \alpha\,\mathrm{sign}(W),$

with XNOR-Net scaling used by default and a fixed shared constant $s_{\mathrm{const}}$ used at residual-add sites to keep both branches in the same integer domain before addition (Takahashi et al., 2024).

On the activation side, one common formulation is uniform $k$-bit quantization. In LAR-Net, multi-bit activations are defined by a discrete level set $S^A=\{s_0,\dots,s_{K-1}\}$ and thresholds $K$0, with categorical probabilities

$K$1

where $K$2 and $K$3 are the Gaussian pre-activation mean and standard deviation under local reparameterization (Berger et al., 2023). In CIM-oriented BWMA, the forward quantizer is the ideal uniform rule

$K$4

while the backward pass uses a smooth surrogate built from a cubic integral $K$5 whose derivative is the quadratic proxy $K$6 (Zhou et al., 29 Aug 2025). In ERNs, the activation is a 2-bit ReLU-like integer quantizer,

$K$7

so activations lie in $K$8 throughout the network (Takahashi et al., 2024).

Several BWMA formulations depart from uniform symmetric $K$9-bit quantization. BWTA uses ternary activations for signed signals,

$k$0

and boolean quantization for strictly non-negative activations such as attention probabilities (Ding et al., 5 Apr 2026). ReverB-SNN replaces the standard binary spike by a real-valued spike activation,

$k$1

so the neuron emits its membrane potential when it crosses threshold; this carries magnitude information forward while keeping binary weights in the synaptic path (Guo et al., 9 Jun 2025).

Training methods are correspondingly heterogeneous. ReverB-SNN uses STBP through time and applies STE only to weight binarization. LAR-Net avoids STE on discrete weights by sampling Gaussian pre-activations with the local reparameterization trick and samples discrete activations via Gumbel-Softmax. BWTA trains with Smooth Multi-Stage Quantization, a Levelwise Degradation Strategy, and a Magnitude-Alignment Projection Factor. BWLA is post-training quantization rather than QAT, and alternates the Orthogonal-Kronecker Transformation and Proximal SVD Projection. The LLM PTQ method with configuration W(1+1)A(1×4) uses Hessian-aware grouping, EM-based codebook estimation, GPTQ-style residual compensation, and activation bit-slicing with scale smoothing (Guo et al., 9 Jun 2025, Berger et al., 2023, Ding et al., 5 Apr 2026, Zhao et al., 1 May 2026, Song et al., 7 Apr 2025).

3. Representative instantiations across model classes

The same BWMA principle is instantiated differently across domains.

Domain	Representative formulation	Characteristic mechanism
CNNs	Binary weights + $k$2-bit activations	Bit-plane decomposition or uniform low-bit quantization
SNNs	Binary weights + real-valued spike activations	Threshold-gated real spikes and addition/subtraction synapses
Transformers	Binary weights + ternary/boolean or low-bit activations	Zero-preserving activation alphabets and bitwise MatMul
LLM PTQ	W1A6, W1A4, or W(1+1)A(1×4)	Offline calibration, rotations, grouping, and binary kernels
Edge CNNs	Full-network W1A2	Integer residual addition via shared constant scaling
CIM CNNs	1-bit cells + 4-bit activations	Per-layer binary centroids aligned with ADC/DAC constraints

BitSplit-Net is a direct BWMA realization when configured with binary weights and $k$3-bit activations. Its defining step is activation bit-splitting: $k$4 followed by decomposition into binary planes $k$5 with fixed significance weights $k$6. Each bit-plane propagates independently through binary-activation layers and the network merges only at the end (Kim et al., 2019).

ReverB-SNN applies the BWMA idea to spiking computation by reversing the bit roles of conventional SNNs: binary weights and real-valued spike activations. It also introduces a learnable per-channel amplitude $k$7 during training,

$k$8

and then folds $k$9 into the previous layer’s activation at inference so that the weight path returns to $w_b = \mathrm{sign}(w)\in\{-1,+1\}.$0 weights (Guo et al., 9 Jun 2025).

BWTA is the $w_b = \mathrm{sign}(w)\in\{-1,+1\}.$1 Transformer case. It uses binary weights with mean-centering and learnable scales, ternary activations in $w_b = \mathrm{sign}(w)\in\{-1,+1\}.$2 for signed signals, and boolean activations for non-negative quantities. The paper’s core claim is that reserving a zero activation level avoids zero-point distortion in attention and linear operators (Ding et al., 5 Apr 2026).

BWLA is a post-training realization of BWMA for LLMs. It targets all linear projections in attention and MLP layers, uses symmetric per-channel binarization for weights and per-token asymmetric quantization for activations, and stabilizes W1A6 or W1A4 by reshaping weights into symmetric bimodal forms with OKT and absorbing residual structure with PSP (Zhao et al., 1 May 2026).

ERNs represent a full ultra-low-bit CNN instantiation. All convolutional weights, including the initial and output layers, are binary, and activations are 2-bit everywhere. A learned generalized thermometer encoding maps 8-bit pixels to a multi-channel 2-bit representation so the first convolution also operates as W1A2 (Takahashi et al., 2024).

CIM-oriented BWMA uses a different binary-weight parameterization. Rather than fixed $w_b = \mathrm{sign}(w)\in\{-1,+1\}.$3, it computes per-layer centroids by matching the first two moments of the original weight distribution,

$w_b = \mathrm{sign}(w)\in\{-1,+1\}.$4

and sets the two binary values to $w_b = \mathrm{sign}(w)\in\{-1,+1\}.$5 and $w_b = \mathrm{sign}(w)\in\{-1,+1\}.$6, with median-based assignment (Zhou et al., 29 Aug 2025).

4. Arithmetic, hardware mapping, and deployment implications

The principal hardware attraction of BWMA is that binary weights simplify the weight path even when activations remain multi-bit. In conventional ANN and Transformer settings this usually means bitwise kernels or bit-plane accumulation. LAR-Net writes a binary-weight, $w_b = \mathrm{sign}(w)\in\{-1,+1\}.$7-bit activation dot product as a sum over activation bit-planes using bitwise AND, NOT, and population count: $w_b = \mathrm{sign}(w)\in\{-1,+1\}.$8 where $w_b = \mathrm{sign}(w)\in\{-1,+1\}.$9 encodes the binary weight mask and $\alpha$0 is the $\alpha$1-th activation bit-plane (Berger et al., 2023). BitSplit-Net uses the same logic in a network-level form: each activation bit-plane is processed by a binary path, so convolution is implemented with AND/XNOR and POPCNT rather than conventional MACs (Kim et al., 2019).

Transformer-specific kernels make the same principle explicit. BWTA develops binary/ternary MatMul kernels on NVIDIA GPUs using instruction-level parallel bit-packing and mma.and or mma.xor plus popcount. For linear layers with binary weights and ternary activations, activations are stored as positive and negative bitmaps $\alpha$2 and $\alpha$3, and the accumulated int32 counts are scaled by $\alpha$4. The same paper gives specialized kernels for attention-value multiplication $\alpha$5 and score computation $\alpha$6 (Ding et al., 5 Apr 2026).

BWMA can also be realized without bitwise multiply emulation. ReverB-SNN exploits the fact that multiplying a real-valued activation by a binary weight $\alpha$7 reduces to addition or subtraction. After re-parameterization, the synaptic update is

$\alpha$8

and the runtime path uses only $\alpha$9. If $s_W$0, the activation is added; if $s_W$1, it is subtracted. This preserves event-driven computation because downstream work is triggered only when the thresholded real-valued spike is non-zero (Guo et al., 9 Jun 2025).

In accelerator design, Loom provides a precision-scaling analysis directly relevant to BWMA. For convolutional layers,

$s_W$2

and for fully connected layers,

$s_W$3

When $s_W$4, a BWMA configuration yields $s_W$5 and $s_W$6. Loom also trims activation precision at runtime per group of 256 activations using OR trees and a leading-one detector, and can exploit sub-layer weight precision variability as well (Sharify et al., 2017).

Edge and CIM deployments impose additional constraints. ERNs make residual additions integer-only by sharing a fixed scaling convention across main and skip paths and fusing scale into post-add quantizer thresholds, so float arithmetic is avoided on the accelerator until the final convolution layer (Takahashi et al., 2024). CIM-oriented BWMA maps binary weights to 1-bit conductance states and uses multi-bit activations driven by $s_W$7-bit DACs and digitized by $s_W$8-bit ADCs. The hardware study reports that ADCs account for about $s_W$9 of total latency, accumulation circuits about $W_b = \alpha\,\mathrm{sign}(W),$0, and other periphery about $W_b = \alpha\,\mathrm{sign}(W),$1, with 4-bit activation quantization identified as the optimal cost/performance point under DNN+NeuroSim (Zhou et al., 29 Aug 2025).

5. Reported empirical behavior

Across image classification, neuromorphic vision, Transformer inference, and LLM PTQ, BWMA consistently appears as an accuracy-efficiency compromise rather than a purely compression-oriented endpoint.

In SNNs, ReverB-SNN reports that on CIFAR-10 with ResNet20 and $W_b = \alpha\,\mathrm{sign}(W),$2, a vanilla SNN reaches $W_b = \alpha\,\mathrm{sign}(W),$3 top-1 and ReverB-SNN reaches $W_b = \alpha\,\mathrm{sign}(W),$4, while the learnable variant with $W_b = \alpha\,\mathrm{sign}(W),$5 reaches $W_b = \alpha\,\mathrm{sign}(W),$6. On CIFAR-100 under the same setting, the figures are $W_b = \alpha\,\mathrm{sign}(W),$7, $W_b = \alpha\,\mathrm{sign}(W),$8, and $W_b = \alpha\,\mathrm{sign}(W),$9. On ImageNet with ResNet34 and $s_{\mathrm{const}}$0, ReverB-SNN reports $s_{\mathrm{const}}$1 and the learnable variant $s_{\mathrm{const}}$2, compared with prior SNN training results in the $s_{\mathrm{const}}$3 to $s_{\mathrm{const}}$4 range. On CIFAR-10 energy accounting for ResNet20 and $s_{\mathrm{const}}$5, the reported energy increase over a vanilla SNN is only about $s_{\mathrm{const}}$6, from $s_{\mathrm{const}}$7 to $s_{\mathrm{const}}$8, while top-1 accuracy improves by $s_{\mathrm{const}}$9 absolute (Guo et al., 9 Jun 2025).

For CNNs outside the spiking setting, BitSplit-Net shows that moving from binary activation to 2- to 4-bit activation closes much of the accuracy gap while preserving binary-style kernels. On ResNet-18 ImageNet, reported Top-1/Top-5 for BitSplit are $k$0 at $k$1, $k$2 at $k$3, $k$4 at $k$5, and $k$6 at $k$7. A WRPN-style widened BitSplit $k$8 model at $k$9 reaches $S^A=\{s_0,\dots,s_{K-1}\}$0 against an FP32 baseline of $S^A=\{s_0,\dots,s_{K-1}\}$1 (Kim et al., 2019).

ERNs push a uniform W1A2 CNN design to practical ImageNet accuracy. The reported Top-1/Top-5 scores are $S^A=\{s_0,\dots,s_{K-1}\}$2 for ERNs-18-0.75×, $S^A=\{s_0,\dots,s_{K-1}\}$3 for ERNs-18, $S^A=\{s_0,\dots,s_{K-1}\}$4 for ERNs-34, $S^A=\{s_0,\dots,s_{K-1}\}$5 for ERNs-50, and $S^A=\{s_0,\dots,s_{K-1}\}$6 for ERNs-101. On a Kria KV260 FPGA device, the same work reports $S^A=\{s_0,\dots,s_{K-1}\}$7 FPS for ERNs-18-0.75×, $S^A=\{s_0,\dots,s_{K-1}\}$8 FPS for ERNs-50, and $S^A=\{s_0,\dots,s_{K-1}\}$9 FPS for ERNs-101, with DSP usage equal to zero (Takahashi et al., 2024).

For Transformers, BWTA reports that binary weights with ternary or boolean activations approach full-precision performance for BERT, with an average drop of about $K$00 on GLUE and less than $K$01 on five tasks. The same paper reports $K$02 to $K$03 times kernel-level speedup over FP16 on NVIDIA GPUs and about $K$04 to $K$05 tokens/s end-to-end prefill speedup when replacing $K$06 to $K$07 of layers with BWTA (Ding et al., 5 Apr 2026).

For LLM PTQ, BWLA reports that on Qwen3-32B under A6, WikiText-2 perplexity is $K$08, whereas prior methods are reported around $K$09 and often collapse on zero-shot tasks. On NVIDIA RTX A6000 with LLaMA2-13B, BWLA W1A8 decoding throughput is $K$10 tokens/s versus FP16 $K$11 tokens/s, corresponding to a $K$12 speedup, and parameter memory falls from $K$13 GB to $K$14 GB (Zhao et al., 1 May 2026). A different LLM PTQ route, W(1+1)A(1×4), reports that naive W1A4 GPTQ collapses to perplexity $K$15 on LLaMA1-7B, while the full method reaches $K$16; it also reports about $K$17 single-layer matmul speedup over CUTLASS INT4/INT8 kernels on A6000 (Song et al., 7 Apr 2025).

In CIM-oriented CNN quantization, BWMA reports ResNet-18 accuracy of $K$18 on CIFAR-10 and $K$19 on ImageNet under $K$20, compared with $K$21 for EGQ, $K$22 for Q-PIM, and $K$23 for Huang et al. The same study reports gains of $K$24 to $K$25 on CIFAR-10 and $K$26 to $K$27 on ImageNet over existing methods, and again identifies 4-bit activations as the best hardware trade-off (Zhou et al., 29 Aug 2025).

6. Limitations, trade-offs, and recurring misconceptions

A first misconception is that BWMA is equivalent to “fully binarized” inference. Several papers explicitly retain non-binary behavior where accuracy or stability requires it. ReverB-SNN uses real-valued spike amplitudes; BWTA uses ternary or boolean activations and keeps LayerNorm and Softmax in FP16/BF16; BWLA focuses on the large linear maps and does not claim embedding or LM-head binarization; the W(1+1)A(1×4) LLM method states that the model is not strictly 1-bit because weights use 2-bit equivalents and activations use 4-bit equivalents plus group parameters (Guo et al., 9 Jun 2025, Ding et al., 5 Apr 2026, Zhao et al., 1 May 2026, Song et al., 7 Apr 2025).

A second misconception is that binary weights always imply the same arithmetic. In ANNs and Transformers, binary weights are commonly paired with bit-plane accumulation or bitwise logic. In SNNs, binary weights can imply addition/subtraction directly because the activation is a sparse thresholded event stream. In CIM, binary storage constrains cell precision but the end-to-end cost is still dominated by converters and periphery rather than only by the binary codebook (Guo et al., 9 Jun 2025, Berger et al., 2023, Zhou et al., 29 Aug 2025).

Training and calibration remain significant trade-offs. ReverB-SNN restores multiplication-free inference only after re-parameterization; during training the $K$28 path uses real-valued multiplications. LAR-Net depends on CLT and local reparameterization assumptions, and notes that small channel counts or early layers can weaken the Gaussian approximation. BWTA requires stable SMSQ schedules and explicitly preserves zero to avoid attention distortion. BWLA depends on calibration with 128 WikiText-2 segments and finds A4 more challenging than A6 or A8. The W(1+1)A(1×4) method relies on Hessian-aware grouping, outlier retention in INT8, and careful scale smoothing. ERNs require correct BN and threshold fusion for integer-only residual additions (Guo et al., 9 Jun 2025, Berger et al., 2023, Ding et al., 5 Apr 2026, Zhao et al., 1 May 2026, Song et al., 7 Apr 2025, Takahashi et al., 2024).

Activation precision is also not monotone in utility. Several works report that moving from binary activation to 2-, 3-, or 4-bit activation yields substantial gains, but that further increases can produce diminishing returns relative to hardware cost. BitSplit-Net reports the largest improvement from $K$29 to $K$30, with smaller gains beyond that. CIM-oriented BWMA finds that 3-bit activations are too coarse and 5- or 6-bit activations add area and energy with limited accuracy benefit, making 4-bit the preferred point. BWLA stabilizes W1A6 and even W1A4, but still notes noticeable degradation at A4. This suggests that the optimal activation alphabet is task- and hardware-dependent rather than fixed by the BWMA label itself (Kim et al., 2019, Zhou et al., 29 Aug 2025, Zhao et al., 1 May 2026).

In that sense, BWMA is best understood as a structured compromise: binary weights define the compression and arithmetic substrate, while the activation representation is chosen to preserve the information that full binarization would discard. The literature shows that this compromise can be instantiated through probabilistic training, thresholded real-valued spikes, stagewise ternarization, post-training rotations and low-rank refinement, integer-only residual pipelines, or closed-form moment matching. The unifying principle is not a single optimizer or codebook, but the deliberate asymmetry between the weight path and the activation path (Berger et al., 2023, Guo et al., 9 Jun 2025, Ding et al., 5 Apr 2026, Zhao et al., 1 May 2026, Takahashi et al., 2024, Zhou et al., 29 Aug 2025).