---
title: INT4 Weight-Only Quantization
url: https://www.emergentmind.com/topics/int4-weight-only-quantization
type: topic
---

# INT4 Weight-Only Quantization

INT4 weight-only quantization refers to the post-training reduction of neural network weight tensors to 4-bit integer representations, with activations retained in higher-precision formats (typically FP16/BF16). This approach enables aggressive model compression and bandwidth reduction, facilitating deployment and inference acceleration, particularly in large language models (LLMs) and vision architectures. Recent research demonstrates that, when combined with advanced calibration and compensation schemes, INT4 weight-only quantization can preserve nearly all baseline accuracy across a wide range of architectures, while incurring minimal computational and integration overhead.

## 1. Quantizer Formulation and Mathematical Framework

Weight-only quantization maps each weight tensor \( W \) of a neural network to a grid of signed 4-bit integer values using a uniform quantizer. Let \( w \) denote a full-precision (FP32/FP16) weight, and \( q \) its quantized representation.

Commonly, symmetric, per-channel or per-group approaches are used:

- **Symmetric quantization (per-channel):**
  \[
  \Delta_c = \frac{\max_{i} |W_{c,i}|}{2^{3}-1}
  \]
  \[
  q = \text{clip}\left(\text{round}\left(\frac{w}{\Delta_c}\right), -7, 7 \right)
  \]
  \[
  w' = q \Delta_c
  \]
  Here, \( W \in \mathbb{R}^{C \times K} \), and \( c \) is an output channel [2305.12356].

- **Asymmetric quantization (per-row, unsigned):**
  \[
  S_i = \frac{w_{max}^{(i)} - w_{min}^{(i)}}{15}, \quad Z_i = w_{min}^{(i)}
  \]
  \[
  q_{ij} = \text{clip}\left(\text{round}\left(\frac{w_{ij} - Z_i}{S_i}\right),0,15\right)
  \]
  \[
  \hat w_{ij} = S_i q_{ij} + Z_i
  \]
  [2303.08302].

Many practical systems use per-group scales, where blocks of weights (e.g., 64–256 elements) share their scale parameter, maximizing hardware efficiency [2308.09723, 2411.02355].

## 2. Optimization Algorithms and Calibration Procedures

The reduction in representational bandwidth at INT4 introduces significant quantization noise. Addressing this requires loss-minimizing parameter search rather than naive range-based quantization.

**1. Powell's Joint Optimization (LAPQ):**  
The quantizer step sizes \(\{\Delta_i\}\) are optimized by directly minimizing the post-quantization loss:
\[
\min_{\Delta_1, \dots, \Delta_n} \; \mathcal{L}(\Delta_1,\dots,\Delta_n)
\]
where \(\mathcal{L}\) is averaged over a small calibration set. LAPQ constructs a trajectory of \(L_p\)-optimal scales, fits a quadratic, then applies joint Powell's method for global minimization [1911.07190].

**2. Error-Minimizing Coordinate Descent (COMQ):**  
For each column (per-channel) of \( W \), coordinate-wise updates alternately optimize 4-bit codes \( z_{ij} \) and scaling factors \(\alpha_j\) in closed form:
\[
z_{ij} \leftarrow \text{clip}\left(\text{round}\left(\frac{w_{ij}}{\alpha_j}\right),-8,7\right)
\]
\[
\alpha_j \leftarrow \frac{\langle W_{:j},z_j \rangle}{\|z_j\|_2^2}
\]
Greedy coordinate update order on large-magnitude weights accelerates convergence [2403.07134].

**3. Second-Order, Hessian-Aware Quantization (GPTQ):**  
Weights are grouped (e.g., G=128) and quantized with blockwise MSE minimization, leveraging gradients and Hessians of the layer output w.r.t. weights, computed over a small calibration set [2411.02355, 2303.08302].

**4. Fine-Grained Adaptive Quantization (FineQuant):**  
Columns are split into blocks (typically B=64), adaptively refining group granularity when dynamic ranges shrink excessively (threshold α≈0.8–0.9) [2308.09723].

## 3. Loss Landscape and Non-Separability at Low Bitwidth

Empirical and theoretical analyses show that INT4 quantization noise induces pronounced layer-to-layer coupling, rendering the loss landscape non-separable. A second-order Taylor expansion quantifies this:
\[
\Delta\mathcal{L} \approx \nabla_{\mathbf{v}}\mathcal{L} \cdot \bm{\varepsilon} + \frac{1}{2}\bm{\varepsilon}^T \nabla^2 \mathcal{L} \bm{\varepsilon}
\]
For INT8, the quadratic coupling term is negligible. At INT4, high curvature and sharp valleys necessitate multi-layer (joint) optimization [1911.07190]. Gaussian curvature metrics (e.g., \(K\approx 6.7\times10^{-25}\) for 4 b, \(K \approx 0.58\) for 2 b in ResNet-18) confirm this regime shift, justifying the more sophisticated search methods employed in modern PTQ frameworks.

## 4. Practical Algorithms, Hardware Implementation, and Inference

A variety of pipelines have demonstrated practical near-lossless INT4 weight-only quantization:

- **Per-group quantization:** Per-column or blockwise scales (B=64 or 128) stored as FP16, int4 codes packed at 2 values/byte [2308.09723].
- **Fused GPU kernels:** On-the-fly dequantization with tensor-core GEMM—matrix multiplication accumulates activations (FP16/BF16) with int4 weights, avoiding explicit full-precision materialization [2308.09723].
- **No retraining:** Approaches such as FineQuant and COMQ function without fine-tuning or even calibration data, though minimal data can further improve accuracy [2308.09723, 2403.07134].
- **Hardware support:** Modern accelerators (NVIDIA H100) natively support int4 x fp16 GEMMs. Memory savings are 4× vs. FP16 for weights and up to 3.65× throughput improvement observed in edge cases [2308.09723, 2411.02355].

Typical end-to-end steps:

1. Extract static weights from pretrained model.
2. Determine scaling factor(s) (max-abs or MSE-optimal).
3. Quantize to int4 via rounding/clipping.
4. Store packed codes.
5. Inference via fused kernel and optional bias correction.

## 5. Empirical Results and Model Recovery

Recent empirical studies present a consistent pattern: INT4 weight-only quantization can achieve accuracy recovery ≈98–99% relative to full precision (FP16/BF16) across a range of model scales and tasks.

**Representative empirical data:**

| Model/Task         | FP16 PPL | INT4 (GPTQ) PPL | INT4 Recovery |
|--------------------|----------|-----------------|--------------|
| Llama-3.1-8B (V1)  | —        | —               | 98.7%        |
| Llama-3.1-70B (V1) | —        | —               | 99.5%        |
| OPT-30B (WikiText) | 10.70    | 10.78           | Class I      |
| OPT-175B           | 9.08     | 9.84 (B=64)     | 26% of FP16 size, <0.8 PPL gap |
| ResNet-18 (Top1)   | 69.7     | 60.3 (LAPQ)     | —            |
| ViT-B/16 (Top1)    | 84.53    | 83.86 (COMQ)    | Δ=−0.67%     |

*Adding low-rank error compensation (LoRC, rank 4–8) can entirely close the accuracy gap for many tasks, with <0.5% parameter and negligible runtime increase [2303.08302, 2307.09782].

## 6. Format Comparisons, Mixed Quantizers, and Guidelines

**INT4 vs FP4:**  
INT4 and FP4 formats show complementary performance. INT4 is preferred for uniform, small-range weights, while FP4 is better for layers with outlier-heavy distributions. Mixture-of-Formats approaches (MoFQ)—which select INT4 or FP4 per layer based on MSE—match or slightly exceed the best single-format baselines, with negligible overhead and SOTA efficiency [2305.12356, 2307.09782].

**Block size and granularity:**  
Per-column INT4 may be sufficient for most matrices, but “catastrophic” collapse can occur in rare cases. Adaptive block splitting (α≈0.8–0.9) and block size B=64 effectively prevent these failures [2308.09723].

**Deployment:**
- INT4 weight-only is the top choice for synchronous, latency-sensitive workloads (chat APIs, per-query decode), providing up to 2–7× cost-efficiency compared to FP8/INT8 [2411.02355].
- For large-batch asynchronous tasks, 8-bit (INT8 or FP8) may outperform INT4, depending on hardware.
- Calibration: 256–512 high-quality tokens suffice for GPTQ-based pipelines.
- Kernel support: Ensure inference engines provide optimized INT4 support for best performance.

## 7. Best Practices and Future Directions

**Best Practices:**
- Use per-group (blockwise, B∼64–256) symmetric quantization for most LLMs.
- Apply loss-aware or Hessian-aware optimization (GPTQ/LAPQ) when accuracy must be maximized.
- Monitor for catastrophic failures with per-column-only INT4, especially on matrices with large outliers. Adapt block granularity as needed.
- Consider mixture quantization (MoFQ) if both INT4 and FP4 are natively supported on deployment hardware.
- Employ low-rank correction if ultra-high fidelity is needed and a tiny parameter overhead (<0.5%) is tolerable.

**Research Directions:**
- Further improving loss-minimizing quantization in the non-separable regime characteristic of INT4.
- Enhanced hardware support for fine-grained INT4 kernels and mixed-format inference.
- Unification of quantization strategies across both dense and MoE architectures.
- Exploration of dynamic quantization (e.g., at runtime based on input distribution) for further gains.

References:  
[1911.07190]  
[2305.12356]  
[2303.08302]  
[2307.09782]  
[2411.02355]  
[2403.07134]  
[2308.09723]

Source: https://www.emergentmind.com/topics/int4-weight-only-quantization