Quantized Neural Networks Overview

• Quantized Neural Networks are defined as systems that replace 32-bit operations with lower-bit representations (e.g. 8-bit, ternary, binary), reducing computation cost and energy consumption.
• They employ techniques like quantization-aware training, straight-through estimators, and alternative optimization methods to mitigate accuracy loss while enabling efficient inference.
• Practical QNN implementations balance trade-offs between bit-width, network depth, and hardware constraints to optimize energy efficiency, memory usage, and deployment on specialized processors.

Searching arXiv for recent and foundational work on Quantized Neural Networks to ground the article with relevant papers and ids. arxiv_search(query="Quantized Neural Networks survey training inference energy verification quantization-aware integer-only", max_results=10, sort_by="relevance") Searching arXiv now. Quantized Neural Networks (QNNs) replace full-precision, typically 32-bit floating-point, weights, activations, or gradients with lower-bit representations such as 8-bit integers, ternary values, or binary values. Their central motivation is to reduce memory footprint, memory bandwidth, computation cost, and energy consumption, especially for mobile, embedded, edge, and microcontroller deployment, while retaining predictive performance that is often close to full-precision baselines (Guo, 2018, Hubara et al., 2016, Abushahla et al., 20 Aug 2025). Across the literature, QNNs are not a single method but a family of numerical representations, training procedures, hardware mappings, and verification techniques whose behavior depends strongly on bit-width, quantization granularity, network topology, and deployment platform (Moons et al., 2017).

1. Numerical model and quantization taxonomy

A standard formulation treats quantization as a mapping from a real-valued tensor to a discrete low-precision set. For uniform quantization, one common expression is

$$Q(x)=\mathrm{clip}\Bigl(\Delta\cdot\mathrm{round}(x/\Delta),\,x_{\min},\,x_{\max}\Bigr),\qquad \Delta=\frac{x_{\max}-x_{\min}}{2^k-1},$$

where $k$ is the bit-width and $\Delta$ is the quantization step size (Guo, 2018). In fixed-point QNNs trained at arbitrary precision $Q$, a real-valued parameter $w$ can be quantized by

$$q=\mathrm{clip}_{[-1,\;1-2^{-(Q-1)}]}\!\left(\frac{\mathrm{round}(2^{Q-1}w)}{2^{Q-1}}\right),$$

which reduces to $q=\mathrm{sign}(w)$ at $Q=1$, yielding BinaryNets (Moons et al., 2017).

Aspect	Variants	Technical note
Quantizer spacing	Uniform; non-uniform	Uniform uses equally spaced levels; non-uniform uses levels matched to a predefined or learned distribution
Representation	Fixed-point; integer; binary; ternary	Binary uses $\{-1,+1\}$; ternary uses $\{-1,0,+1\}$
Granularity	Layer-wise; channel-wise; group-wise	Scales or codebooks may be shared per layer, per channel, or per group

The literature distinguishes several representational choices. Uniform quantization partitions a real-valued range into equally spaced levels, whereas non-uniform quantization uses levels such as logarithmic, k-means centroid, or power-of-two values (Guo, 2018). Fixed-point and affine integer schemes are common in deployment-oriented toolchains; a widely used affine form on microcontrollers is

$k$0

with scale factor $k$1 and zero-point $k$2 (Abushahla et al., 20 Aug 2025). Symmetric quantization uses $k$3, while asymmetric quantization allows skewed ranges, which is often useful for activation distributions after ReLU (Abushahla et al., 20 Aug 2025).

QNNs are not restricted to weights alone. Foundational training work quantized weights and activations at run-time and also quantized parameter gradients to 6 bits, enabling gradients computation using only bit-wise operation (Hubara et al., 2016). This broader scope matters because memory traffic from activations and gradients can dominate the cost profile of training and inference.

A core numerical difficulty is that quantizers are piecewise constant. The standard remedy is the Straight-Through Estimator (STE), which replaces the zero derivative of rounding or sign with a clipped surrogate. A common form is

$k$4

and, for binary QNN training, $k$5 (Guo, 2018, Moons et al., 2017). This surrogate underlies much of QAT, but later work explicitly questions whether STE is the only viable route.

2. Training formulations and optimization strategies

The literature separates post-training quantization (PTQ) from quantization-aware training (QAT). PTQ takes a fully trained full-precision model, quantizes weights and activations offline, and may fine-tune afterward; QAT inserts fake quantization in the forward pass while retaining real weights for gradient updates, typically with STE in the backward pass (Guo, 2018, Abushahla et al., 20 Aug 2025). Early large-scale results showed that QNNs trained with low-precision weights and activations can remain competitive with 32-bit models; for example, a quantized AlexNet with 1-bit weights and 2-bit activations achieved 51.0% top-1 and 73.7% top-5 on ImageNet, while a 4-bit GoogLeNet achieved 66.5% top-1 and 83.4% top-5 (Hubara et al., 2016).

Several strands of work modify the optimization problem itself rather than relying exclusively on hard quantizers plus STE. ProxQuant formulates training as

$k$6

where $k$7 vanishes exactly on the discrete quantized set, and then applies non-lazy prox-gradient descent. The method is presented as a principled alternative to straight-through training, with theoretical convergence to stationary points for the smooth regularized objective and a stability analysis showing that BinaryConnect can oscillate unless a restrictive sign condition holds (Bai et al., 2018). “Quantization Networks” instead replace hard quantizers by a differentiable nonlinear quantization function

$k$8

with a temperature $k$9 that is ramped up so that the Sigmoid approaches the hard step used at inference (Yang et al., 2019). This framework treats weight and activation quantization uniformly and is reported to outperform prior approximation-based and optimization-based methods on ImageNet classification and Pascal VOC detection (Yang et al., 2019).

Other work targets the gradient-mismatch problem more directly. The Asymptotic-Quantized Estimator (AQE) uses

$\Delta$0

which interpolates between the identity and the ideal low-bit quantizer during training and becomes exact low-bit quantization as $\Delta$1 (Chen et al., 2020). On CIFAR-10, MINW-Net with AQE is reported to achieve prediction accuracy 1.5% higher than a BNN with STE for the same 1-bit weights and activations; on AlexNet and ResNet-18 on ImageNet, AQE closes part of the low-bit accuracy gap relative to prior QNN baselines (Chen et al., 2020). For power-of-two quantization toward FPGAs, reconstructed-gradient training introduces

$\Delta$2

so that $\Delta$3, explicitly avoiding vanishing gradients under staircase quantization (Chen et al., 2020).

Accuracy recovery can also be framed as a distributional or architectural problem. Balanced Quantization recursively partitions parameters by percentiles into balanced bins before uniform quantization, increasing effective bitwidth utilization without adding inference cost; on ImageNet, the reported top-5 error of a 4-bit quantized GoogLeNet is 12.7%, compared with 16.6% for the cited prior 4-bit baseline (Zhou et al., 2017). “Widening and Squeezing” argues that quantized features are weaker than full-precision features and therefore widens each layer by a factor $\Delta$4, then prunes redundant channels via network-slimming and optional knowledge distillation. In the reported ImageNet experiments, a 1-bit ResNet-18 with width multiplier $\Delta$5 reaches 71.08/89.74, surpassing the cited full-precision 70.79/89.50 baseline before pruning; after pruning, knowledge distillation restores much of the lost accuracy (Liu et al., 2020).

A separate line of work addresses batch normalization as an obstacle to true integer-only inference. Progressive Tandem Learning trains a BN-free, fully integer student from a pretrained BN-enabled quantized teacher by local layer initialization and layer-wise tandem distillation. On ImageNet with AlexNet, the BN-free student at 4-bit weights and activations records top-1 error 49.45% versus 48.50% for the BN-enabled teacher, and at 8-bit it records 47.72% versus 47.00%; on CIFAR-10 with VGG-Small at 1-bit/1-bit, the BN-free student reaches 90.8% accuracy versus 90.2% for the DoReFa-Net baseline (Sun et al., 18 Dec 2025).

3. Accuracy, precision, energy, and model size trade-offs

The defining systems-level property of QNNs is that lower bit-width reduces per-operation arithmetic complexity and memory per weight, but often requires deeper or wider networks to recover the same accuracy. This iso-accuracy trade-off is made explicit in “Minimum Energy Quantized Neural Networks,” where the minimum-energy design is posed as a joint optimization over bit-width and topology subject to accuracy and on-chip memory constraints (Moons et al., 2017). In that framework, total inference energy is

$\Delta$6

and the on-chip MAC energy scales as

$\Delta$7

The paper decomposes hardware energy into compute, weight-fetch, and activation-fetch terms and shows that the optimal bit-width depends on the interaction between arithmetic energy, model size, feature-map size, and memory hierarchy (Moons et al., 2017).

The reported empirical picture is strongly non-monotone. Across CIFAR-10, MNIST, and SVHN, energy consumption varies orders of magnitude at iso-accuracy depending on bit-width. For a “typical” 4 Mb on-chip memory, BinaryNets minimize energy at high allowable error on CIFAR-10 ($\Delta$8), int4 is optimal around $\Delta$9–$Q$0 error and gives $Q$1 lower energy than int8 and $Q$2 lower than $Q$3, while BinaryNets regain the advantage at strict error targets ($Q$4) by scaling the network deeper. Across all cases, int4 implementations outperform int8 by $Q$5–$Q$6 at iso-accuracy. With tighter on-chip memory (1 Mb), BinaryNets are always optimal because off-chip DRAM accesses overwhelm arithmetic savings; with effectively infinite on-chip memory, int2 or int4 are preferred (Moons et al., 2017). The same study also reports a pronounced “accuracy cliff”: improving CIFAR-10 error from 13% to 10% with int4 increases energy by approximately $Q$7, whereas going from 17% to 13% costs only approximately $Q$8 (Moons et al., 2017).

The survey literature gives the corresponding compression and arithmetic view. An 8-bit weight costs 1 byte and a 1-bit weight only 0.125 bytes, giving up to $Q$9 smaller storage than 32-bit weights; integer or bitwise operations such as XNOR+popcount are much cheaper than floating-point MAC, and ALUs consume $w$0–$w$1 less energy than FPUs (Guo, 2018). On the empirical side, sample ImageNet results for ResNet-18 show graceful degradation from 30.4 top-1 error at 32/32 to 30.6 at 8/8, 30.8 at 4/4, 33.1 at 2/2, and 51.2 at 1/1 (Guo, 2018). The microcontroller-focused review generalizes this pattern: INT8 offers a 4× smaller model size than FP32, approximately 20–30× energy saving in MACs and adds, and 3–7× speedup on integer-optimized hardware, while 4-bit typically requires QAT and per-channel or mixed precision to keep accuracy loss within 1–3% (Abushahla et al., 20 Aug 2025).

Kim et al. extend the trade-off to federated learning by quantizing both local training and uplink transmission in fixed precision. Their formulation includes an analytical energy model for local training, a wireless transmission model, and a convergence bound with a quantization-noise term proportional to $w$2; the optimal precision $w$3 is chosen by minimizing total expected energy under a target convergence criterion (Kim et al., 2021). On an MNIST-based CNN and a 50-device FL setup, the reported energy reduction is up to 53% relative to standard 32-bit FL, with an optimal $w$4 bits. The same study emphasizes the central nonlinearity of the design problem: very small $w$5 slows convergence and increases the number of rounds, whereas very large $w$6 raises per-iteration energy (Kim et al., 2021).

4. Integer-only deployment and hardware realizations

A major practical theme in QNN research is that low-bit arithmetic is valuable only if the inference graph can be mapped cleanly to integer hardware. “Streamlined Deployment for Quantized Neural Networks” presents a flow that converts all QNN inference operations to integer ones by expressing quantization as successive thresholding, collapsing floating-point linear operations into a single affine transform, and then absorbing that transform into the thresholds (Umuroglu et al., 2017). After streamlining, each layer reduces to an integer matrix multiply followed by thresholding with integer thresholds. The same work uses bit-serial GEMM, decomposing low-bit tensors into bit-planes and accumulating weighted binary products with AND/XOR/XNOR and popcount. On a single ARM Cortex-A57 @1.9 GHz, the reported end-to-end result on quantized AlexNet is 7.7 FPS for the interleaved bit-serial implementation versus 2.2 FPS for the baseline, a 3.5× speed-up; the peak 1×1 BitSerialGEMM reaches approximately 150 GOPS, about 6.8× faster than the cited 8-bit baseline on in-cache sizes (Umuroglu et al., 2017).

Hardware co-design appears in more specialized forms as well. For FPGAs, n-BQ-NN constrains weights to power-of-two levels so that multiplications can be replaced by shifts, and the corresponding shift vector processing element array removes most DSP-based multiplications from convolution (Chen et al., 2020). On Xilinx ZCU102 at 200 MHz, the reported throughput for AlexNet rises from 332 GOP/s with the baseline VPE implementation to 957.4 GOP/s with SVPE, a 2.9× speed-up; power falls from 28.7 W to 19.6 W, and the summary states that average energy consumption is reduced to 68.7% of the VPE array with 16-bit (Chen et al., 2020). The reported model accuracy remains close to full precision, including 56.0% top-1 and 79.5% top-5 for AlexNet with 3-bit weights and 16-bit activations, within 0.6% of the cited 32-bit baseline (Chen et al., 2020).

True integer-only execution also requires eliminating floating-point normalization. In Progressive Tandem Learning, BN is removed rather than folded into high-precision weights, and each student layer uses an integer scale factor that is absorbed into the quantization schedule so that the forward pass becomes integer convolution or matrix multiply followed by right-shift or multiplication by a precomputed integer factor. The result is a computation graph with no BN operations and no floating-point inference path, specifically targeting accelerators with strictly integer ALUs such as microcontrollers, integer-only DSPs, and memristor crossbars (Sun et al., 18 Dec 2025).

QNN research also includes non-von-Neumann and in-memory realizations. MTJ-based stochastic training uses a ternary 4T2R synapse, near-memory processing, and pulse-controlled stochastic switching to implement quantized updates in situ. For a 127×127 array, the reported energy efficiency is 18.3 TOPs/W for feedforward, 1.43 TOPs/W for backpropagate, and 3.0 TOPs/W for weight update, while ternary-network accuracy remains within 1% degradation relative to the GXNOR algorithm on MNIST, SVHN, and CIFAR-10 (Toledo et al., 2019). This line of work shows that QNNs are as much about memory movement and update locality as about arithmetic precision itself.

The platform landscape in the recent review spans TensorFlow Lite Micro with CMSIS-NN kernels on ARM Cortex-M, ExecuTorch, ai8x-training and ai8x-synthesis for MAX78000/02 NPUs, GAPflow for GAP8/GAP9, and hybrid or micro-NPU platforms such as Ethos-U55 + Cortex-M55 and STM32N6+Neural-ART (Abushahla et al., 20 Aug 2025). This suggests that the dominant deployment question is no longer whether low-bit inference is possible, but which quantization granularity and operator set are supported efficiently by a given software-hardware stack.

5. Verification, adversarial robustness, and fault tolerance

Because QNNs use discrete arithmetic and clipping, their assurance problems differ materially from those of floating-point networks. QVIP formulates local robustness verification and maximum robustness radius computation for feed-forward QNNs as integer linear programming. The key construction introduces integer variables for pre-activations and post-quantized activations together with Boolean interval variables encoding the piecewise-constant rounding-and-clamping semantics; the resulting encoding is both sound and complete (Zhang et al., 2022). The prototype emits ILP in Gurobi’s format and uses interval analysis to prune inactive quantization intervals. On QNNs over MNIST and Fashion-MNIST with quant-bits $w$7, the reported outcome is often 100× speedup on the common subset of problems relative to prior SMT/bitvector approaches, with robustness radii up to 30 and average maximum robustness radius 9 for a 6-bit model versus 5 for a 4-bit model on one benchmark (Zhang et al., 2022).

Certified robustness under quantization requires verifying the quantized representation itself rather than assuming that guarantees transfer from a floating-point model. QA-IBP adapts interval bound propagation to discrete fixed-point semantics by propagating lower and upper bounds through integer linear layers, quantized rounding, and clamping, and then couples that training procedure with a complete branch-and-bound verifier that runs entirely on GPU (Lechner et al., 2022). The reported certified robust accuracy on 8-bit models is 98.8% on MNIST and 80.0% on Fashion-MNIST at $w$8, and 95.6% and 59.8% respectively at $w$9; the paper emphasizes that prior work had shown floating-point networks verified as robust can become vulnerable after quantization (Lechner et al., 2022).

Security work further shows that quantization alone is not a general defense. TriQDef begins from the observation that quantization can distort the gradient landscape and weaken conventional pixel-level attacks, yet offers limited robustness against localized adversarial patches that remain transferable across bit-widths (Guesmi et al., 16 Aug 2025). The proposed tri-level framework combines a Feature Disalignment Penalty, a Gradient Perceptual Dissonance Penalty, and a joint multi-bit QAT protocol. The abstract reports that TriQDef reduces Attack Success Rates by over 40% on unseen patch and quantization combinations while preserving high clean accuracy, and the detailed CIFAR-10 results under LAVAN show unseen-patch ASR of 35.6/33.3/29.1/27.3% at 32/5/4/2-bit, versus 77.2/73.7/67.8/65.3% for the cited PBAT baseline (Guesmi et al., 16 Aug 2025). A common misconception is therefore that low precision is automatically protective; the patch-transfer results directly contradict that view.

Reliability under hardware faults is another distinct concern. Selective neuron splitting for 8-bit QNNs first computes a Neuron Vulnerability Factor to identify critical neurons and then replaces each critical neuron by two sub-neurons with halved dynamic range, together with a Lightweight Correction Unit that performs bitwise correction without modifying the PE datapath (Ahmadilivani et al., 2023). At an NVF threshold of 20%, the reported overhead and protection on MLP-7, LeNet-5, and AlexNet are similar to selective TMR while using one extra sub-neuron rather than two. The abstract summarizes the method as having a twice smaller overhead than selective TMR while achieving a similar level of fault resiliency (Ahmadilivani et al., 2023).

6. Scope, misconceptions, and current directions

The accumulated literature corrects several recurrent simplifications. First, QNNs are not synonymous with binary networks. The field explicitly covers 8-bit integer, 4-bit, 2-bit, ternary, and binary regimes, as well as arbitrary fixed-point precision $$q=\mathrm{clip}_{[-1,\;1-2^{-(Q-1)}]}\!\left(\frac{\mathrm{round}(2^{Q-1}w)}{2^{Q-1}}\right),$$0; training from scratch has been reported to converge stably across $$q=\mathrm{clip}_{[-1,\;1-2^{-(Q-1)}]}\!\left(\frac{\mathrm{round}(2^{Q-1}w)}{2^{Q-1}}\right),$$1 (Moons et al., 2017, Guo, 2018). Second, quantization is not merely a final compression step. PTQ is one path, but QAT, proximal regularization, differentiable quantization functions, asymptotic estimators, balanced histogram equalization, widening and squeezing, and BN-free progressive tandem learning all modify the optimization landscape or the architecture itself (Bai et al., 2018, Yang et al., 2019, Chen et al., 2020, Zhou et al., 2017, Liu et al., 2020, Sun et al., 18 Dec 2025). Third, low precision does not guarantee either adversarial robustness or certifiability; dedicated training, verification, and defense procedures are required (Lechner et al., 2022, Guesmi et al., 16 Aug 2025).

The application scope is correspondingly broad. The cited works include CNNs on MNIST, CIFAR-10, SVHN, and ImageNet; SSD300 detection on Pascal VOC; RNNs and LSTMs on Penn Treebank; wireless federated learning; FPGA, mobile CPU, microcontroller, and memristive or MTJ-based deployment; and certifiable or formally verified models (Hubara et al., 2016, Yang et al., 2019, Kim et al., 2021, Abushahla et al., 20 Aug 2025). This breadth suggests that “QNN” is best understood as a cross-layer design space rather than a single algorithm.

Current open problems are also well defined in the surveyed literature. The 2018 survey emphasizes gradient mismatch and unstable STE training, optimal bit-width allocation per layer or channel, task-specific quantization for NLP, speech, and segmentation, theoretical guarantees in non-convex settings, and hardware-software co-design for maximal end-to-end efficiency (Guo, 2018). The 2025 microcontroller review adds mainstream lack of native sub-8-bit support, sparse operator coverage, rigid static tensor arenas, on-device training and continual learning, quantization for transformers and diffusion models, alternative number systems such as Posit, BF16, TF32, FP8, and trans-precision frameworks, and full-stack co-optimization across quantization, compiler autotiling, and hardware microarchitecture (Abushahla et al., 20 Aug 2025).

A plausible implication is that future QNN research will continue to move away from isolated “low-bit network” recipes and toward integrated precision allocation, training, compilation, verification, and deployment flows. The evidence already assembled across energy modeling, integer-only execution, certified robustness, and hardware specialization indicates that bit-width is not an isolated hyperparameter but a systems variable that couples accuracy, topology, arithmetic, memory hierarchy, and assurance requirements (Moons et al., 2017, Abushahla et al., 20 Aug 2025).