Low-Bit Post-Training Quantization

• Low-bit post-training quantization is a process that converts full-precision neural networks into low-bit representations, achieving efficient, high-throughput inference with minimal performance loss.
• It employs advanced rounding optimization, vector quantization, and mixed-precision strategies to mitigate quantization errors in ultra-low-bit regimes.
• The technique facilitates robust hardware integration and empirical gains in transformers, LLMs, and vision models through adaptive calibration and blockwise reconstruction.

Low-bit post-training quantization (PTQ) encompasses a set of methodologies for converting pre-trained, full-precision neural networks into efficient, low-memory, and high-throughput quantized models without retraining. This process targets extremely low weight/activation precision—often 2–4 bits, but sometimes sub-2-bit or hybrid schemes—aiming to minimize the resulting loss in predictive performance. State-of-the-art low-bit PTQ frameworks now integrate advanced rounding optimization, vector quantization, blockwise loss objectives, adaptive precision, and calibration-centric optimization, enabling robust deployment across modern transformers, LLMs, vision transformers (ViTs), and generative architectures. This article synthesizes modern advancements, architectural strategies, and empirical insights in low-bit PTQ, emphasizing recent results in both natural language and vision domains.

1. Quantization Formulations and Loss Measures

Low-bit PTQ frameworks model the quantization process as a mapping from floating-point weights $\theta$ (or activations $a$) to quantized values $\widehat{\theta}$ using low-bit codes and associated dequantization parameters (scales, zero points, centroids). The canonical formulation uses an asymmetric (or sometimes symmetric) uniform affine quantizer:

$$X^q = \mathrm{clamp}\bigl(\lfloor X / s \rceil + z,\; 0, 2^N-1 \bigr), \qquad \widehat X = s\,(X^q - z)$$

where $s$ is a scale, $z$ is a zero point, and $N$ is the bit-width (Li et al., 2024). Non-uniform and vector quantization strategies generalize this operator by allowing per-column K-Means codebooks (Wang et al., 2024), residual vector quantizers (Liu et al., 2024), or even full blockwise codebook assignments.

The quantization loss is most commonly measured as the mean-square reconstruction error (MSE) between the outputs of floating-point and quantized subnetworks, typically at the block or layer level:

$$\min_\epsilon \sum_b \| \mathrm{block}(\widehat{\theta}^{(b)}, x^{(b)}) - \mathrm{block}(\theta^{(b)}, x^{(b)}) \|_F^2$$

where each block corresponds to an architectural module (e.g., decoder, attention+FFN, or transformer block) (Li et al., 2024, Liu et al., 2024). Newer works use prediction difference metrics such as Kullback-Leibler divergence in output logits (Liu et al., 2022), or proxy cross-entropy gradients to guide bit allocation (Cheng et al., 4 Dec 2025).

2. Rounding Optimization and Blockwise Reconstruction

Greedy or naive round-to-nearest (RTN) schemes perform poorly in ultra-low-bit scenarios due to the quantization of rounding errors across entire layers. Consequently, modern frameworks implement advanced rounding optimization, including:

• Blockwise Rounding with Progressive Adaptive Rounding (PAR): Binary rounding variables are relaxed as $\alpha = \sigma(\nu)$ (sigmoid), and a progressive schedule hardens variables from soft ($0 < \alpha < 1$) to hard ($\alpha \in \{0,1\}$) while minimizing blockwise reconstruction loss. The process alternates hardening (based on “hardness score”) and local Adam updates, finalizing all rounding at the end (Li et al., 2024).
• Gradient-based Layer or Column Rounding: Some methods apply small-scale stochastic search on rounding offsets per scalar or vector to optimize local objectives, typically using straight-through estimators when gradients cross the quantization boundary (Cheng et al., 4 Dec 2025).
• Weighted Block Losses via BatchNorm Statistics: Rounding errors are regularized per layer using robust L-P losses parameterized by BatchNorm scales, adapting robustness to outliers per block (Yao et al., 2022).

Blockwise or groupwise reconstruction—where quantization variables are learned per architectural block—substantially improves approximation accuracy compared to pure layerwise calibration, especially in transformers and LLMs (Li et al., 2024, Liu et al., 2024).

3. Vector Quantization and Mixed-Granularity Methods

Scalar quantization fails in the extreme low-bit regime due to codebook limitations. Vector quantization (VQ) strategies assign entire columns or small groups of weights to centroids in a shared or per-column codebook. VPTQ, for example:

• Decomposes weight matrices into blockwise vectors and assigns each to a codebook index via second-order loss minimization:

$\min_{\{i_j\}} \sum_j \| v_j - C_{i_j} \|^2$

where $v_j$ are vector partitions and $C_{i_j}$ are codebook centroids (Liu et al., 2024).

• Supports multi-stage schemes such as residual VQ (applying VQ to the residual after first quantization) and “outlier quantization” using a larger codebook for rare, high-magnitude vectors.

Hybrid precision schemes further refine accuracy by adaptively assigning higher bit-width or even floating-point retention to “outlier” entries or columns identified via magnitude or sensitivity heuristics (Zhao et al., 18 Feb 2025, Wang et al., 2024). The use of structured (1D) masks enables sub-2-bit average bit-widths with negligible metadata cost (Zhao et al., 18 Feb 2025).

4. Auxiliary Optimizations: Scale Tuning, Preprocessing, and Distillation

PTQ accuracy is highly sensitive to dequantization parameters; several methods provide:

• Learnable Dequantization Scale Tuning: Simultaneous optimization of a small trainable factor $\delta$, such that dequantization is $\widehat\theta = 2\sigma(\delta) s(\theta^q - z)$, is often performed with rounding optimization and can reduce error by up to 30% (Li et al., 2024).
• Preprocessing via Restorative Fine-Tuning: A brief LoRA adaptation on pre-training data can concentrate saliency into structured subsets of weights, producing distributions more amenable to sub-2-bit quantization (Zhao et al., 18 Feb 2025).
• Distillation-Aided Calibration: A brief calibration step distills the outputs or features of the FP model into the quantized model, refining quantization parameters via end-to-end losses such as LPIPS, MSE, or logit cross-entropy (Liu et al., 2024, Li et al., 1 Jun 2025).

5. Adaptivity and Mixed-Precision Strategies

Modern low-bit PTQ frameworks frequently employ adaptive bit allocation across layers or columns:

• Sensitivity Metrics and Dynamic Programming Assignment: Layerwise sensitivity (e.g., “DeltaLoss”) combines gradient information and quantization deviation, enabling integer-programming-based assignment of bits under a global memory or compute budget (Cheng et al., 4 Dec 2025).
• Column-wise Adaptive Precision: By computing column-wise outlier ratios, methods dynamically allocate higher bit-width to “sensitive” columns while using the minimum for others to match overall bit budgets (Wang et al., 2024).
• Dynamic Outlier Reservation: Columns or entries with high outlier ratios receive partial or full floating-point retention, implemented via a sparse reservation mask (Wang et al., 2024).

Vector/mixed-precision models thus approach or surpass fixed-precision baselines at extreme compression ratios, with customizable accuracy/rate trade-off.

6. Hardware Implementation and Scalability

Practical PTQ is constrained by available integer compute, memory bandwidth, and the efficiency of the quantization pipeline.

• Blockwise and Vector Quantization Support: Block- and vector-based PTQ methods are structured to be compatible with group-wise or matrix-based quantized kernel primitives in modern hardware, facilitating INT2–INT4 inference when supported (Li et al., 2024, Liu et al., 2024).
• Power-of-Two Quantization: To support accelerators restricted to bit-shifts, Po2 quantization constrains scales to powers of two, requiring global joint optimization of exponents to minimize cumulative rounding and clipping error (Yao et al., 2022).
• Memory and Throughput Gains: Extreme low-bit PTQ (e.g., W2A16 with group size 128) can reduce LLM model weight memory from 756 GB to 114 GB (LLaMA-3.1-405B) and, with mature kernels, recover or exceed FP16 throughput (Li et al., 2024).
• Calibration and Runtime Costs: Modern frameworks demonstrate sublinear runtime and memory scaling, e.g., 2–6 hours total PTQ on a single A100 GPU for 7B–13B parameter models for state-of-the-art accuracy at 2–3 bits (Li et al., 2024, Liu et al., 2024, Cheng et al., 4 Dec 2025).

7. Empirical Performance, Scaling Laws, and Future Challenges

Recent low-bit PTQ methods attain near-lossless performance under aggressive quantization regimes, but empirical limits are emerging:

Model	Bits	SOTA PPL/Top-1 Acc	Reference
LLaMA-2-7B (W2A16)	2-bit	6.82 (TesseraQ)	(Li et al., 2024)
LLaMA-2-7B (VPTQ)	2.02-bit	6.13	(Liu et al., 2024)
LLaMA-1-7B (CLAQ)	2.24-bit	6.93	(Wang et al., 2024)
LLaMA-1-7B (PTQ1.61)	1.61-bit	12.5	(Zhao et al., 18 Feb 2025)
ViT-B (PFCR+POS)	3-bit	75.61%	(Ding et al., 2024)
ResNet-50 (FP=xINT)	4-bit	77.0%	(Zhang et al., 2024)

Extensive scaling analyses reveal that quantization-induced degradation (QiD) increases super-linearly with both training token count and model precision: for a unified scaling law on LLMs,

$$\Delta_{\!q}\mathrm{Loss}(N,D,P)=0.017\,N^{-0.2261}\,D^{0.5251}\,P^{-5.4967}$$

indicating qualitative collapse of 2–3 bit PTQ on fully trained, trillion-token-scale models unless mitigated by quantization-aware retraining or advanced mixed-precision PTQ (Ouyang et al., 2024).

• "TesseraQ: Ultra Low-Bit LLM Post-Training Quantization with Block Reconstruction" (Li et al., 2024)
• "VPTQ: Extreme Low-bit Vector Post-Training Quantization for LLMs" (Liu et al., 2024)
• "PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for LLMs" (Zhao et al., 18 Feb 2025)
• "SignRoundV2: Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs" (Cheng et al., 4 Dec 2025)
• "CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs" (Wang et al., 2024)
• "Progressive Fine-to-Coarse Reconstruction for Accurate Low-Bit Post-Training Quantization in Vision Transformers" (Ding et al., 2024)
• "FP=xINT:A Low-Bit Series Expansion Algorithm for Post-Training Quantization" (Zhang et al., 2024)
• "Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens" (Ouyang et al., 2024)
• "RAPQ: Rescuing Accuracy for Power-of-Two Low-bit Post-training Quantization" (Yao et al., 2022)
• "COMQ: A Backpropagation-Free Algorithm for Post-Training Quantization" (Zhang et al., 2024)
• "Post-Training Quantization for Neural Networks with Provable Guarantees" (Zhang et al., 2022)

Low-bit PTQ is now a mature area with rich methodology and empirically validated procedures for reliably mapping large-scale pre-trained networks to extreme compression, while exposing intricate new challenges for future high-capacity, fully trained models.