---
title: INT8 Scalar Quantization in Deep Learning
url: https://www.emergentmind.com/topics/int8-scalar-quantization
type: topic
---

# INT8 Scalar Quantization in Deep Learning

INT8 scalar quantization is a foundational technique for compressing and accelerating DNNs and LLMs by mapping high-precision floating-point tensors to 8-bit signed integers through a linear (scalar) mapping. The resulting 4× reduction in size (FP32→INT8) enables both memory savings and execution on fixed-point hardware with minimal performance loss when properly calibrated. Key applications include inference and training across vision, language, and retrieval models, with increasing deployment on mobile, edge, and data-center accelerators.

## 1. Mathematical Formulation and Core Algorithms

INT8 scalar quantization maps a floating-point value $w \in \mathbb{R}$ to $q \in \mathbb{Z}_{8}$ with a scale factor (often called $\gamma$ or $S$):
\[
q = \mathrm{clip}\left(\mathrm{round}(w \cdot \gamma), q_{\min}, q_{\max}\right), \qquad \hat w = \frac{q}{\gamma}
\]
where $q_{\min} = -128$, $q_{\max} = 127$, and $b=8$. The scaling factor $\gamma$ (or $S$) can be derived via the data range or second-moment statistics:
- In post-training quantization (PTQ): $\gamma = \frac{2^b - 1}{w_{\max} - w_{\min}}$.
- In quantization-aware training (QAT): $\gamma = \sqrt{ \mathbb{E}[w^2] / \mathbb{E}[Q(w;\gamma)^2] }$, enforcing energy preservation between quantized and original tensors [2411.06084].

Zero-point is typically omitted (symmetric quantization), particularly in high-performance INT8 implementations [2006.16669, 2409.16997].

Per-tensor, per-channel, or per-block scaling may be employed:
- Per-tensor: One scale/zero-point per tensor (dense retrieval, basic LLMs) [2511.13057].
- Per-channel: One scale per output channel (CNNs, increases accuracy for variable distributions) [2102.04782].
- Per-block: Partition tensor into $B \times B$ blocks, each with its own scale (transformer/GPU-friendly, balances error and computation) [2403.12422].

## 2. Calibration, Scale Optimization, and Practical Implementation

The selection of scale/zero-point critically impacts quantization error. Calibration is typically performed on small calibration sets:
- For PTQ, initial scale is determined by dynamic range:
  \[
  s = \frac{w_{\max} - w_{\min}}{2^b-1}, \quad \gamma = 1/s
  \]
- In frameworks like EasyQuant, alternating greedy optimization maximizes cosine similarity between quantized output and true FP output per layer, often using 1D grid search over scale for weights and activations [2006.16669].

Table: Example INT8 Quantization Performance (EasyQuant, ImageNet-1k, Top-1)

| Model         | FP32  | INT8 (TRT) | INT8 (EQ) |
|---------------|-------|------------|-----------|
| ResNet-50     | 75.20%| 75.04%     | 75.13%    |
| MobileNet-V1  | 69.33%| 68.74%     | 68.84%    |

Advanced schemes deploy per-channel or per-block scale factors to further minimize error, especially where per-tensor statistics are too coarse (notably for gradient quantization in training pipelines [2102.04782], block-level in transformers [2403.12422], or token-level in self-attention [2409.16997]).

Affine quantization (with zero-point) is also used for embeddings:
- $q_i = \mathrm{clamp}( \mathrm{round}( x_i / s + z ), q_{\min}, q_{\max} )$
- $z = \mathrm{round}( -x_{\min}/s ) + q_{\min}$
Enables mapping arbitrary float ranges to INT8 for dense retrieval, yielding near-lossless retrieval performance in practice at 4× compression [2511.13057].

## 3. Training and Backpropagation with INT8 Quantization

End-to-end INT8 training is realized by quantizing not just weights and activations, but also gradients, errors, batch-norm statistics, and updates [1909.02384]:

- Forward and backward passes operate in the INT8 domain using integer quantization, e.g., $Q_{A}(a) = Q(a,k_{A})$, $Q_{E_1}(e) = SQ(e, k_{E_1})$.
- Gradients are quantized either channelwise (vectorized quantization) or globally, with channelwise approaches yielding lower error, especially in non-Gaussian distributions—adapted via magnitude-aware clipping and per-channel threshold detection [2102.04782].
- Quantized momentum and optimizer updates are implemented with bit-width constraints ensuring no "escape hatch" to FP in the training loop.

WAGEUBN implements direct, constant, and shift quantizers to address rounding artifacts and precision mismatches in different data paths, while keeping the full loop in INT8 [1909.02384].

Empirical results on ResNet/Imagenet show within 1–3% top-1 accuracy drop for pure INT8 end-to-end training, with 4× memory reduction and up to 10–30× energy efficiency improvement on FPGA [1909.02384].

## 4. Hardware Realization and Execution Efficiency

Modern inference and training accelerators favor scalar INT8 quantization due to:
- Fixed 8-bit integer arithmetic units present in all major hardware backends (ARM NEON, NVIDIA Tensor Cores, mobile DSPs, etc.).
- Quantized representations being 4× smaller than FP32, increasing arithmetic intensity and memory throughput.

For CPUs lacking native INT8 SIMD, software techniques such as Scalar Arithmetic Multiple Data (SAMD) partition machine words into parallel INT8 lanes for efficient bit-precise arithmetic, achieving 4–10× speedups over native 8-bit or FP32 on both ARM and x86 [1809.10572].

On GPUs, INT8 tensor-cores accelerate matmuls, and quantized dataflow designs eliminate frequent quantize/dequantize overheads (e.g., Jetfire INT8 transformer pretraining is 1.4× faster and 1.5× leaner in memory than FP16 [2403.12422]). INT-FlashAttention implements per-token or per-block INT8 quantization to realize $>70\%$ inference speedup versus FP16, with $>45\%$ quantization error reduction compared to FP8 [2409.16997].

Empirical deployment on edge devices (e.g., RK3399, Qualcomm Hexagon) demonstrates 2.4× throughput and 40% power reduction at minimal (<1%) accuracy loss [2411.06084, 2006.16669, 2405.14024].

## 5. Extensions, Innovations, and Advanced Error Mitigation

Recent work addresses limits and extents of scalar INT8 quantization:
- Mixed-precision optimization with classical Lagrangian allocation across layers: $b^*_l = \frac{1}{2} \log_2 ( \frac{\alpha_l \sigma^2_l}{\lambda} )$, but in uniform INT8 all $b_l = 8$ [2411.06084].
- Redundant output representations (2D Hilbert curve mapping) reduce quantization error 5× for bounded regression outputs at $<7\%$ runtime cost [2405.14024].
- Clipping and scale distribution strategies, including outlier-aware per-block partitioning, per-channel scaling, and magnitude-aware loss terms, mitigate harsh artifacts and performance drops in highly nonuniform tensor distributions [2102.04782].
- For dense retrieval, INT8 scalar quantization with global per-tensor scaling achieves $<0.2\%$ nDCG@10 drop over float32, outperforming same-compression-ratio autoencoders by $>6\%$ [2511.13057].

Table: Model Compression and Effect on Retrieval (BEIR SciFact)

| Type        | Bytes/vec | Compression | $\Delta$ nDCG@10 |
|-------------|-----------|-------------|------------------|
| FP32        | 1536      | 1×          | 0.0000           |
| FP16        | 768       | 2×          | –0.00018         |
| INT8        | 384       | 4×          | –0.00178         |
| Binary      | 48        | 32×         | –0.46621         |

## 6. Deployment, Toolchains, and Limitations

Deployment pipelines exploit vendor-supplied automation (TensorRT, ONNX Runtime, QAT) for quantization graph rewriting and calibration [2411.06084, 2006.16669]. INT8 is universally supported, unlike INT4 or custom bit-widths.

Main practical limitations are:
- Sensitivity to outliers: A single large element can distort global scale, motivating per-channel/block strategies.
- Quantization-aware training (QAT) can further close accuracy gaps at additional training and memory cost.
- Scaling parameters ($\gamma$, $S$) must be stored and supported in inference code but are negligible in overhead.
- Extremely low-bit quantization (e.g., binary) causes catastrophic performance drops in high-dimensional retrieval and regression [2511.13057, 2405.14024].
- Pure post-training quantization may incur higher errors in highly sensitive layers or ill-conditioned models, for which training-aware schemes are preferable [2411.06084].

## 7. Impact and Empirical Results

INT8 scalar quantization, when combined with robust scale calibration and, where suitable, quantization-aware training, yields:

- $68\%$ model size reduction and $40\%$ compute/energy savings in LLMs for up to $1$B parameters with $<6\%$ performance drop [2411.06084].
- Speedups of 1.4–2.4× (and up to 10× in hand-optimized/packed software) for both training and inference on edge, CPU, and GPU, typically at $<1\%$ accuracy loss for vision and retrieval [2006.16669, 1809.10572].
- End-to-end INT8 training matches or slightly lags FP16/FP32, with full-precision convergence achieved by tuning mid-layer errors, using mixed-precision for sensitive paths, or leveraging redundant output representations [1909.02384, 2102.04782, 2405.14024].

INT8 scalar quantization thus represents a practical, rigorously validated, and broadly supported approach for efficient deployment of state-of-the-art neural networks on diverse hardware platforms [2411.06084, 2006.16669, 2511.13057, 2409.16997, 2403.12422, 2102.04782, 1909.02384, 1809.10572, 2405.14024].

Source: https://www.emergentmind.com/topics/int8-scalar-quantization