8-Bit Quantization in Deep Learning

• 8‑bit quantization is a process that maps floating-point neural network parameters to 8‑bit representations to reduce memory and accelerate computations.
• It employs methods like symmetric linear, per-channel, and non-uniform quantization, along with calibration techniques to minimize error.
• Empirical results show near lossless accuracy, significant speedups, and reduced memory usage across diverse tasks such as translation, vision, and speech.

8‑bit quantization refers to the process of mapping floating-point neural network parameters, activations, gradients, or even optimizer states to 8‑bit integer or floating-point representations, with the goal of reducing memory, accelerating computation, and enabling deployment on resource- or latency-constrained hardware. The 8‑bit regime is the canonical boundary between high-precision (FP32/16) and aggressive model compression, and is now supported end-to-end in inference and, more recently, training, across a wide range of deep learning models and tasks.

1. Quantization Schemes and Mathematical Formalism

Canonical 8-bit quantization schemes fall into several categories:

• Symmetric Linear Quantization: Real-valued data $x \in [m, M]$ is mapped to $q \in [q_{\min}, q_{\max}] = [-128, 127]$ using a scale factor $s = \frac{M-m}{q_{\max}-q_{\min}}$ and (when needed) a zero-point $z = \mathrm{round}(q_{\min} - m/s)$. Quantization and dequantization proceed via:

$$q = \mathrm{clip}(\mathrm{round}(x/s) + z, q_{\min}, q_{\max}), \quad x \approx s\,(q - z)$$

This formulation underlies popular frameworks (TensorFlow QuantizeV2, GEMMLOWP, PyTorch quantization) and appears in production for Transformers and RNNs (Bhandare et al., 2019, Li et al., 2021, Quinn et al., 2018).

• Per-tensor/Per-channel Quantization: Most practical systems compute distinct quantization parameters for each tensor, or per-output channel for weight matrices with large dynamic range variation.
• Calibration: Quantization ranges $(m, M)$ (or related scale/zero-point) are chosen by minimizing some divergence measure (often KL-divergence) between the original tensor histogram and the 8-bit quantized version, or by minimizing mean squared error, or maximizing cosine similarity between post-layer outputs (as in EasyQuant (Wu et al., 2020)).
• Non-uniform/Block-wise Quantization: Nonlinear or non-uniform quantization (e.g., tanh-based weight scaling (Zeng et al., 2022), Lloyd-Max quantizers (Zhen et al., 2022), block-wise dynamic quantization for optimizer states (Dettmers et al., 2021)) is critical for highly heavy-tailed or outlier-dominated distributions.
• Alternatives—Floating Point 8-bit (FP8): Recent trends explore 8-bit floating formats (e.g. E4M3, E3M4, E5M2) balancing dynamic range and local precision; per-layer exponent/mantissa configuration and bias selection further allow close matching of heavy-tailed distributions, and have shown superior performance in vision and diffusion models (Zhang et al., 2023, Chen et al., 2024).
• Gradient and Optimizer Quantization: Complete 8-bit training requires quantizing forward activations, weights, gradients, and optimizer states, demanding careful error propagation management (Banner et al., 2018, Yang et al., 2019, Dettmers et al., 2021).

2. Engineering Implementation and System Integration

The transformation of floating-point inference pipelines to 8-bit integer is deeply system-specific. Typical steps include:

• Operator Replacement: Replace FP32 MatMul/Conv with INT8 kernels (e.g., Intel MKL-DNN’s GEMM_S8U8S32 for x86 AVX512-VNNI), replacing float GEMMs with integer GEMMs, and fusing scale/offset arithmetic (Bhandare et al., 2019, Quinn et al., 2018).
• Graph Surgery: Automatically insert quantize/dequantize ops around eligible layers, optimize out redundant conversions, and replace default kernels to minimize unnecessary int-float transitions (Bhandare et al., 2019).
• Nonlinear Component Handling: Some non-linearities (softmax, layernorm, division, sqrt) remain in FP32 due to catastrophic information loss when quantized; all inputs are dequantized to full precision at these points (Bhandare et al., 2019, Lin et al., 2020).
• Functionality for Training: During quantized training (e.g., with WAGEUBN (Yang et al., 2019)), all major data paths—weights, activations, gradients, errors, updates, BatchNorm parameters, optimizer state—are mapped to INT8, relying on bespoke quantizers (e.g. direct-quantization, constant-quantization, shift-quantization).
• Hardware Specialization: Efficient integer computation is predicated on hardware support (VNNI, NEON, SMLAL, custom ASICs for 8×8→32 MAC)—as well as memory hierarchy engineering to exploit reduced bitwidth throughout the system (Bhandare et al., 2019, Jin et al., 2022, Dettmers et al., 2021).

3. Empirical Performance and Task-Specific Effects

8-bit quantization, when carefully applied, achieves near lossless model fidelity across modalities:

Model/Task	Baseline (FP32)	8‑bit accuracy/perf	ΔAccuracy	Speedup	Memory Δ
Transformer NMT	27.68 BLEU	27.30–27.33 BLEU	–0.35 BLEU	up to 3.7×	4× less
ResNet-50 (ImageNet)	74.66%	~69–72%	–2–5%	up to 3×	4× less
LSTM ASR	WER 6.6%	WER 6.7%	+0.1%	2×	4× less
RoBERTa-L (GLUE)	88.6	88.7	–	≈1×	2–3× less
LLM CL (INT8, SOTA)	74.44% → 60.25/35%	Outperforms FP16 on forward accuracy	see below	2×*

Key remarks:

• Degradation in image classification is typically <1% Top-1 when using clipping and weight reshaping optimizations (Yang et al., 5 Oct 2025).
• End-to-end speedups are strongly hardware-dependent; up to 3.7× for integer GEMM on AVX512-VNNI (Bhandare et al., 2019), 2–4× in inference and distributed training (Dettmers, 2015, Banner et al., 2018).
• Extreme low-bit quantization (4 or 5 bits) often degrades accuracy sharply except with non-uniform quantization or advanced regularization (Zhen et al., 2022, Zeng et al., 2022).
• For continual learning, INT8 can surpass FP16 in retention/plasticity trade-off, attributed to implicit regularization by quantization noise (Zhang et al., 22 Dec 2025).

4. Optimization, Calibration, and Best Practices

Optimal 8‑bit quantization accuracy relies on several protocol choices and empirical recommendations:

• Per-tensor vs Per-channel: Per-channel quantization recovers substantial accuracy when cross-channel dynamic range is high.
• Calibration objective: Minimizing layer output MSE or maximizing cosine similarity between pre-/post-quantization outputs in a calibration set empirically yields minimal accuracy loss (Wu et al., 2020, Yang et al., 5 Oct 2025).
• Outlier Handling: Heavy-tailed or multi-modal tensors (e.g. Q/K/V weights in attention) benefit from explicit clipping, possibly with power-law reshaping (e.g., $\sqrt{|w|}$) before quantization (Yang et al., 5 Oct 2025).
• Nonlinear Fallbacks: SoftMax, LayerNorm, BN, and similar numerically sensitive ops are by default left in FP32 or at increased bitwidth (16 bits for errors in WAGEUBN) (Bhandare et al., 2019, Yang et al., 2019).
• BatchNorm Specialization: Range BatchNorm, which replaces variance-based normalization with range-based, is robust to quantization noise and relies only on max/min operations (Banner et al., 2018).
• Fine-tuning / QAT: Even for post-training quantization, light QAT or calibration-based retraining with a handful of epochs can recover most of the FP32 accuracy loss (Jin et al., 2022, Yang et al., 2019).
• Batching, Pipelining, and Input Engineering: For NLP applications with variable input lengths, bin-packing or token sorting maximizes hardware utilization in deployment (Bhandare et al., 2019).

5. Special Variants and Extensions

The 8-bit regime admits numerous extensions and elaborations:

• FP8 and Mixed Precision: FP8 formats (E5M2, E4M3, etc.) outperform INT8 for distributions with heavy tails; mixed INT8/FP8 quantization per-layer further closes the accuracy gap on more challenging tasks such as object detection and language understanding (Zhang et al., 2023, Chen et al., 2024).
• Sub-8-bit and Mixed-Precision Search: GRU and RNN architectures benefit from modular schemes assigning independent bitwidth per-operator (2–8 bits), optimized via genetic algorithms for Pareto-optimal memory/accuracy trade-off (Miccini et al., 2024).
• Block-wise Quantized Optimizers: Adam and Momentum optimizer states can be block quantized to 8 bits (per-tensor block) via dynamic tree quantization, requiring no adaptation of training hyperparameters (Dettmers et al., 2021). This yields up to 75% memory reduction for optimizer states with no performance loss.
• Hot-Swap Quantization: Training one model for simultaneous 1–8 bit hot-swappable modes is possible by learning per-bitwidth reconstruction and quantization hyperparameters associated to a shared set of weights (wavelet decomposition, per-bit BN/clip settings) (Sun et al., 2021).
• Two-Stage and Nonlinear QAT: Two-stage QAT methods for sub-8-bit deployment—nonlinear (e.g., tanh-based) quantization of weights followed by linear quantization of all other parameters—enable full-precision parity for keyword-spotting and small audio models at 4–5 bits (Zeng et al., 2022).

6. Open Challenges, Limitations, and Lessons Learned

Despite robust empirical performance, successful 8-bit quantization depends on several factors and raises open issues:

• Sensitive Distributions: Narrow, sparse, or multi-modal weight/activation tensors can cause dramatic quantization error; outlier-aware calibration and fallback to FP32 in rare cases are necessary (Bhandare et al., 2019, Yang et al., 5 Oct 2025).
• Nonlinearities and Residuals: Scaling errors accumulate through deep residual networks; careful quantization of residual adds and normalization layers is required to avoid accuracy loss (Lin et al., 2020).
• Training Instability: Direct quantization of optimizer state can destabilize embedding layers in LMs; block-wise quantization and special stable initialization/normalization mitigate this (Dettmers et al., 2021).
• Batch Size and Statistics: Small batch sizes degrade accuracy due to poor activation statistics (especially for BN), motivating the use of per-batch adaptive normalization or larger accumulators (Yang et al., 2019).
• Hardware Constraints: Full benefit of INT8/F8 quantization depends on native hardware support for vector-matrix multiplication, accumulator width, and ability to exploit reduced memory bandwidth (Bhandare et al., 2019, Jin et al., 2022, Zhang et al., 2023).

Empirically, the combination of robust calibration (regional search, outlier clipping, or automated FL assignment), judicious use of fallback to higher precision for sensitive ops, and post-training fine-tuning define state-of-the-art 8-bit quantization pipelines across vision, language, and generative models.

---

References:

• "Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model" (Bhandare et al., 2019)
• "When Less is More: 8-bit Quantization Improves Continual Learning in LLMs" (Zhang et al., 22 Dec 2025)
• "Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers" (Yang et al., 2019)
• "Quantization Range Estimation for Convolutional Neural Networks" (Yang et al., 5 Oct 2025)
• "A Little Bit More: Bitplane-Wise Bit-Depth Recovery" (Punnappurath et al., 2020)
• "EasyQuant: Post-training Quantization via Scale Optimization" (Wu et al., 2020)
• "Towards a tailored mixed-precision sub-8-bit quantization scheme for Gated Recurrent Units using Genetic Algorithms" (Miccini et al., 2024)
• "Low-Bitwidth Floating Point Quantization for Efficient High-Quality Diffusion Models" (Chen et al., 2024)
• "Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognition" (Zhen et al., 2022)
• "8-Bit Approximations for Parallelism in Deep Learning" (Dettmers, 2015)
• "On the quantization of recurrent neural networks" (Li et al., 2021)
• "F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization" (Jin et al., 2022)
• "One Model for All Quantization: A Quantized Network Supporting Hot-Swap Bit-Width Adjustment" (Sun et al., 2021)
• "Pieces of Eight: 8-bit Neural Machine Translation" (Quinn et al., 2018)
• "Exploring the Potential of Flexible 8-bit Format: Design and Algorithm" (Zhang et al., 2023)
• "Towards Fully 8-bit Integer Inference for the Transformer Model" (Lin et al., 2020)
• "Scalable Methods for 8-bit Training of Neural Networks" (Banner et al., 2018)
• "Sub 8-Bit Quantization of Streaming Keyword Spotting Models for Embedded Chipsets" (Zeng et al., 2022)
• "8-bit Optimizers via Block-wise Quantization" (Dettmers et al., 2021)