Dynamic INT8 Quantization

Updated 28 November 2025

Dynamic INT8 quantization is a runtime method that transforms floating-point neural network parameters and activations into 8-bit integers, offering improved efficiency.
It reduces memory footprint and computation by computing quantization scales dynamically per token, tensor, or block while minimizing accuracy degradation.
Empirical results show up to 73% speedup and near FP32 accuracy across transformers and CNNs, highlighting its practical benefits.

Dynamic INT8 quantization is a set of methodologies for converting floating-point neural network weights, activations, and intermediate results into signed 8-bit integer representations where quantization scales are calculated at runtime or at a fine level of granularity (e.g., per-token, per-row, per-block), as opposed to being fixed statically during calibration. Dynamic approaches enable substantial reductions in memory footprint and arithmetic complexity, especially crucial in the context of large-scale transformer models, while minimizing the quantization-induced degradation in model accuracy.

1. Quantization Formulations and Schemes

Dynamic INT8 quantization maps each floating-point tensor element $x \in \mathbb{R}$ to an integer $x_q \in \mathbb{Z}$ using a linear (typically symmetric) transformation:

$x_q = \text{round} \left( \text{clamp} \left( \frac{x}{\alpha} + z, z_\text{min}, z_\text{max} \right) \right)$

where $\alpha > 0$ is the scale, $z$ is the zero-point (commonly $0$ in symmetric quantization), and $[z_\text{min}, z_\text{max}]$ is the valid INT8 range (e.g., $[-127, 127]$ ). Dequantization reconstructs floating-point values as $x \approx \alpha x_q$ .

Dynamic quantization diverges from static post-training quantization by determining scale values $\alpha$ post-deployment, typically by analyzing the current minibatch or tensor block, rather than from a precomputed calibration set. Dynamic approaches include:

Per-token (row-wise) quantization: Each token (row of, e.g., $Q$ , $K$ matrices) gets its own scale, capturing token-specific dynamic ranges (Chen et al., 25 Sep 2024, Yao et al., 2023).
Per-tensor quantization: A single scale for an entire activation or weight tensor, yet recomputed dynamically for each invocation (El-Kurdi et al., 2022).
Block-level quantization: Tensors are divided into blocks, each block is quantized with its own runtime-determined scale, enabling finer adaptation especially important in presence of outliers or heavy-tailed distributions (Zhang et al., 11 Mar 2025).
Per-channel quantization: Each channel (e.g., filter in CNNs) gets its own scale, with scales adapted online to the distribution of the gradients during training (Zhao et al., 2021).

Specialized dynamic quantization methods, such as TM-IQR-based clipping (El-Kurdi et al., 2022), employ interquartile range rules to suppress activation outliers before quantization, further reducing quantization error.

2. Dynamic Range Calibration and Outlier Handling

Unlike static quantization, which calibrates scale factors $\alpha$ based on dataset statistics, dynamic methods compute them at runtime. The calibration process is tightly coupled to the distribution of activation or gradient values in the current minibatch, token, or block. Key mechanisms include:

Row/Token-Level Scaling: For each token $i$ , the scale is set as $S_{i} = \max_d |A_{i,d}|/127$ for tensor $A$ ; this preserves per-token variation (Chen et al., 25 Sep 2024, Yao et al., 2023).
Tukey's IQR Clipping: Applied per activation tensor, extreme outliers are identified using Tukey's rule applied to row maxima; activations are clipped at $t = Q3 + 1.5(Q3-Q1)$ before quantization (El-Kurdi et al., 2022).
Block Fallback with Outlier Detection: For training with GLU-style activations that exhibit rare, large outliers, each block's maximum absolute value is monitored. If it exceeds a dynamic threshold, quantization falls back to higher-precision or employs a two-stage residual quantization, maintaining training stability and avoiding divergence (Zhang et al., 11 Mar 2025).

In all cases, dynamic calibration ensures that quantization scales tightly track the actual data distribution, eliminating the excessive clipping and information loss prevalent in static schemes.

3. Integration With Neural Network Operators and Pipelines

Dynamic INT8 quantization has been integrated into various model architectures and operators, primarily in inference for transformers and, more recently, in training for CNNs and transformers:

Fused Foward Kernels: INT-FlashAttention implements a fully-INT8 tile-wise pipeline for FlashAttention by carrying all $Q$ , $K$ , $V$ matrices and their per-token scales in INT8, and applying dynamic requantization to the softmax output $P$ within each block (Chen et al., 25 Sep 2024).
Transformer Feedforward Layers: Zero-shot dynamic quantization with TM-IQR targets only the largest source of activation outliers (activations before FF2 GEMM in transformers), resulting in significant accuracy recovery with low runtime overhead (El-Kurdi et al., 2022).
Unified Inference Pipelines: ZeroQuant-HERO fuses per-token/feature dynamic quantization passes into memory-bound operations (LayerNorm, Softmax) and uses static or feature-wise quant within compute-intensive GEMMs, maintaining end-to-end INT8 dataflows with selective module-wise fallback to FP16/BF16 for accuracy-sensitive components (Yao et al., 2023).
Training Pipelines: Channel-wise, dynamically-adaptive quantization has been applied to activations and gradients during CNN training with MCS, ensuring minimal quantization noise for influencing gradients (Zhao et al., 2021). Dynamic block-level fallback has been developed for transformer training, falling back to higher precision selectively when outlier blocks are detected (Zhang et al., 11 Mar 2025).

4. Empirical Results and Performance Characteristics

Systematic benchmarking across hardware platforms demonstrates that dynamic INT8 quantization yields substantial performance benefits:

Speed: INT-FlashAttention achieves $31\%-73\%$ faster inference than FP16 FlashAttention for sequence lengths 1k–16k, and $72\%$ speedup over FP16 at maximal context (Chen et al., 25 Sep 2024). Dynamic block-level fallback training on large transformers yields up to $1.57 \times$ speedup with $\sim 38\%$ activation memory savings (Zhang et al., 11 Mar 2025).
Accuracy: TM-IQR dynamic quantization recovers almost all accuracy loss vs. FP32 in BERT/RoBERTa on GLUE and QA (typically within $0.2$--$0.5$ points of FP32), outperforming naïve INT8 by $1$--$5$ points on harder benchmarks (El-Kurdi et al., 2022). INT-FlashAttention records up to $82\%$ lower mean relative error vs. block-FP8 methods for both normal and uniform input distributions (Chen et al., 25 Sep 2024).
Robustness: Distribution-adaptive INT8 training for CNNs preserves “lossless” accuracy (often within $\pm 0.1\%$ of FP32) over large-scale datasets and models (Zhao et al., 2021). Dynamic quantization in ZeroQuant-HERO mode M1 (≥70% of modules in INT8) incurs only $<0.2$ point average accuracy drop on BERT/GLUE tasks (Yao et al., 2023).

A synthesis of results is presented below:

Method	Accuracy Δ vs FP32	Typical Throughput Gain	Notes
INT-FlashAttention	≤ 0.1–0.5 points	31–73% (Ampere)	72% speedup vs FP16; 82% lower error vs FP8
TM-IQR (El-Kurdi et al., 2022)	≤ 0.5 points	<2% slowdown vs naive INT8	Per-layer dynamic for FF2 only
MCS CNN quant (Zhao et al., 2021)	±0.1%	2.0× (training, RTX 2080Ti)	Channel-wise dynamic for gradients
Block-Fallback (Zhang et al., 11 Mar 2025)	Overlay w/ BF16	1.4–1.6× (training, 4090)	Blockwise fallback to 16-bit on outliers
ZeroQuant-HERO (Yao et al., 2023)	<0.2 pt (M1)	Projected 4–5× (inference)	Token-/feature-wise dynamic, hardware-aware

5. Hardware and Software Optimizations

Dynamic INT8 quantization methodologies are increasingly designed to exploit hardware characteristics:

Kernel Fusion: Quantization, dequantization, and scaling operations are fused into GEMM or memory-bound kernels, maximizing locality and SIMD/Tensor Core throughput (Yao et al., 2023).
On-Chip Accumulator Strategies: Scales and zero-points are broadcast into shared memory/register files, eliminating expensive per-element divisions in favor of register-level products (Yao et al., 2023).
Block/Tile Tiling: All state-of-the-art systems leverage block- or tile-wise microkernels (e.g., in FlashAttention or modern CNNs) to exploit fast on-chip memory transposition and minimize DRAM accesses (Chen et al., 25 Sep 2024, Zhao et al., 2021, Zhang et al., 11 Mar 2025).
Memory-Bound Operator Fusion: For transformers, per-token dynamic quantization is concealed within LayerNorm or Softmax passes, ensuring that quantization overhead does not impact critical path latency (Yao et al., 2023).

These techniques collectively maximize throughput and minimize additional latency incurred by runtime scale calculations or outlier detection.

6. Extensions, Generalizations, and Limitations

Dynamic INT8 quantization pipelines generalize naturally to other bit-widths (e.g., INT4, INT16) as long as corresponding integer GEMMs exist (Chen et al., 25 Sep 2024). Selective per-block, per-token, or per-channel quantization allows balancing between accuracy and throughput/memory benefits.

However, several challenges remain:

Fallback/mixed-precision logic: The tuning of fallback thresholds (in block-fallback schemes) and module-level toggling of INT8/FP16 can require heuristic or empirical tuning (Yao et al., 2023, Zhang et al., 11 Mar 2025).
Coverage: TM-IQR, for example, applies only to FF2 activations in transformers for tractability; extension to all layers increases overhead (El-Kurdi et al., 2022).
Outlier Sensitivity: Heavy-tailed or GLU activations may require increased fallback or sophisticated outlier processing to maintain stability (Zhang et al., 11 Mar 2025).
Calibration vs. Dynamic Trade-offs: Per-tensor dynamic quantization may be suboptimal in highly non-uniform activation distributions, suggesting further research on hybrid per-channel/per-token dynamic schemes (El-Kurdi et al., 2022).
Hardware Dependence: Effectiveness and achievable speedup scale with support for fast INT8 GEMMs (e.g., via NVIDIA Tensor Cores, cublasLt, custom kernels) (Yao et al., 2023, Chen et al., 25 Sep 2024).

This suggests that future advances in dynamic INT8 quantization will be shaped by continued co-design between quantization algorithms, model architectures, and hardware capabilities.