---
title: 'SmoothQuant+: Efficient LLM Quantization'
url: https://www.emergentmind.com/topics/smoothquant-3df5d987-9e75-4152-9bd9-c85e8133444a
type: topic
---

# SmoothQuant+: Efficient LLM Quantization

SmoothQuant+ refers to a family of advanced post-training quantization (PTQ) techniques that extend the original SmoothQuant framework, targeting highly accurate, hardware-efficient quantization of large language models (LLMs) and Transformers. Initialized by the need to reduce inference memory and compute overheads without sacrificing accuracy, especially at 4-bit or lower precision, SmoothQuant+ generalizes and refines channel-wise activation smoothing, adapts to diverse quantization errors (including token-wise "activation spikes"), and incorporates group-wise and data-driven optimizations for accurate, efficient deployment in practical systems. Implementations of SmoothQuant+ achieve lossless or near-lossless accuracy while demanding significantly less hardware, and several state-of-the-art variants have been integrated into frameworks such as vLLM [2312.03788], with extensions for activation quantization and fully arbitrary-precision workflows in both NLP and vision settings.

## 1. Foundations and Motivation

SmoothQuant+ is motivated by the distinct challenges that arise in quantizing LLMs and Transformers: 
- **Activation outliers:** In large models (e.g., LLaMA-34B), a small subset of hidden-state channels exhibit extreme values, amplifying quantization error in subsequent weight-only PTQ schemes [2312.03788].
- **Error accumulation:** Quantization error can compound layer-wise, especially in deep models, making naive quantization strategies (RTN, AWQ, GPTQ) inadequate at very low bit-widths [2312.03788, 2211.10438].
- **Hardware alignment:** Most available accelerators support INT8/INT4 arithmetic but not mixed-precision or per-channel quantization, mandating block/group-wise parameterization for practical deployment [2408.08554].

SmoothQuant+ builds directly on the core transformation introduced in SmoothQuant [2211.10438]: for a linear transformation $Y = XW$, one applies a mathematically equivalent rescaling, $Y = (X \,\mathrm{diag}(s)^{-1})(\mathrm{diag}(s) W)$, where $s$ is a channel-wise scaling vector. This migrates activation outliers into the corresponding weights, allowing for more effective quantization.

## 2. Methodological Advances in SmoothQuant+

SmoothQuant+ introduces several critical algorithmic and workflow improvements over classic SmoothQuant:

- **Mathematically-exact channel-wise smoothing:** For each linear layer, per-input channel scales are computed by interpolating between the maximum absolute value of activations ($\max_t |X_{t,j}|$) and weights ($\max |W_{j,\cdot}|$) via a hyperparameter $\alpha\in[0,1]$:
  \[
  s_j = [\max_t |X_{t,j}|]^\alpha / [\max |W_{j,·}|]^{1-\alpha}
  \]
  $\alpha$ is grid-searched on a calibration dataset to minimize global quantization loss [2312.03788].

- **Group-wise 4-bit quantization:** Weights are quantized in groups (e.g., group size $G=128$) along the channel axis, applying per-group min-max range scaling and zero-point adjustment [2312.03788]. This reduces outlier impact compared to per-tensor schemes and is hardware-friendly.

- **Error-minimizing calibration:** The smoothing strength $\alpha$ and scale vectors $s$ are optimized jointly to directly minimize the quantization-induced MSE on the calibration set. Dequantization remains mathematically exact after fusion into preceding LayerNorms or residual streams [2312.03788, 2211.10438].

- **Empirical token-wise spike handling:** In Transformer variants with GLUs (e.g., SwiGLU, GeGLU), extremely sparse but large-magnitude "activation spikes" persist after standard channel-wise smoothing. SmoothQuant+ integrates strategies such as Quantization-free Module (QFeM) and Quantization-free Prefix (QFeP) to selectively exempt modules or positions associated with spikes from activation quantization, operating in W8A16 (weights INT8, activations FP16) for those modules, and precomputing FP16 caches for problematic tokens [2405.14428].

- **Seamless hardware integration:** Custom CUDA kernels (e.g., W4A16 in vLLM) enable high-throughput, low-latency fused dequantization-GEMM operations. Weight buffers are stored as INT4 with 16-bit groupwise scales and zero points [2312.03788].

## 3. Empirical Performance and Implementations

Empirical evaluations demonstrate that SmoothQuant+ matches or surpasses full-precision accuracy on canonical LLM benchmarks, with orders-of-magnitude improvements in inference efficiency:

| Model            | Method       | Precision   | Pass@1 (HumanEval) | Throughput (vs FP16 ×2GPU) | Latency (vs FP16 ×2GPU) |
|------------------|-------------|-------------|--------------------|----------------------------|-------------------------|
| Code Llama-34B   | FP16        | FP16        | 51.22%             | 1×                         | 1×                      |
| Code Llama-34B   | RTN         | W4          | 46.34%             |                            |                         |
| Code Llama-34B   | AWQ         | W4          | 50.61%             | 0.8×                       | 1.5×                    |
| Code Llama-34B   | SmoothQuant+| W4          | 53.05%             | 1.9–4.0×                   | 0.68×                   |

On Code Llama-34B, SmoothQuant+ enables deployment on a single A100 40GB (previously requiring 2×A100 for FP16), achieving "lossless" or even improved accuracy. Multilingual evaluation (BabelCode) shows improvement in average pass@1 from 40.45% to 41.05% [2312.03788]. Throughput increases by up to $4\times$ and token latency drops to $0.68\times$ that of the FP16 baseline.

## 4. Robustness Extensions and Hybridization

SmoothQuant+ has been extended to address settings previously challenging for channel-wise scale migration alone:

- **Handling GLU activation spikes:** QFeM identifies layers where token-wise scales demonstrate extreme outlier ratios $r^{(m)}$ (max-to-median), and disables activation quantization only in these modules, keeping all others quantized [2405.14428]. QFeP further precomputes FP16 KV caches for spike-inducing token prefixes, preventing recurrent quantization failures.
  
- **Vision Transformer adaptation:** In MPTQ-ViT, SmoothQuant with bias (SQ-b) introduces a per-channel bias subtraction, centering activation distributions to reduce asymmetric clamping loss. An OPT-m scaling search further partitions post-GELU activation distributions to three hardware-friendly quantized regions, selected by minimizing second-order error metrics [2401.14895].

- **Symmetry and low-bit quantization:** To mitigate performance collapse in the 2-bit regime due to the loss of histogram symmetry, a "bit-balance" strategy corrects symmetric quantizer codebooks and rescaling, incorporated into arbitrary-bit frameworks such as ABQ-LLM [2408.08554].

- **Learned rotations:** SpinQuant parameterizes and learns orthonormal rotations $R$ applied to weights and/or activations, minimizing end-to-end PTQ loss under 4-bit quantization. This generalizes channel-wise scaling to full-dimensional mixing, achieving large performance gains over SmoothQuant, especially in low-bit scenarios [2405.16406].

## 5. Workflow, Calibration, and Practical Guidelines

The canonical SmoothQuant+ workflow comprises the following steps [2312.03788, 2405.14428]:
1. **Calibration Set Selection:** Select a small, representative calibration set (e.g., 100–200 prompts), ensuring distributional similarity to downstream tasks.
2. **Activation and Weight Statistics:** Record per-channel activation maxima and per-channel weight maxima.
3. **Smoothing Parameter Optimization:** Search $\alpha\in [0,1]$ (step size ≈0.05) to minimize quantization loss.
4. **Smoothing and Weight Fusion:** Apply channel-wise (optionally, data-driven or bias/rotation-augmented) smoothing; fuse scales into the model.
5. **Group-wise Quantization:** Partition weight matrices into groups (G=128 recommended), quantize to 4 bits/group using per-group scale and zero-point.
6. **Outlier Module Diagnosis (QFeM/QFeP):** Analyze module-wise activation spike ratios and exempt problematic modules/prefixes.
7. **Inference Kernel Integration:** Deploy in vLLM or analogous frameworks using optimized fused kernels (e.g., W4A16), with no further user code changes required.

Recommended choices:
- **Group size**: 128 for weight-group quantization (balances fidelity and metadata)
- **Calibration set size**: ~150
- **Spike threshold $\alpha$**: optimized per model (typ. $>$16 for LLaMA-2-13B)
- **Bias and region partitioning**: for ViTs, use mean subtraction and OPT-m as per [2401.14895].

Limitations:
- Calibration set distributional mismatch may cause degradation.
- Activation quantization is not performed in vanilla SmoothQuant+; out-of-distribution activations can still stress dynamic range.
- For INT8 PTQ with severe spike behavior, direct per-token methods (QFeM/QFeP) are required [2405.14428].

## 6. Impact, Extensions, and Comparative Results

SmoothQuant+ and its derivatives have established state-of-the-art accuracy/efficiency trade-offs for low-bit quantization in both NLP and vision, with a range of methods targeting specific aspects:

| Method/Variant    | Precision    | Target Domain   | Notable Features            | Key Results                                      |
|-------------------|-------------|-----------------|-----------------------------|--------------------------------------------------|
| SmoothQuant+      | W4 (only)   | LLMs            | Lossless accuracy, group-wise, channel smoothing | Up to 4× throughput, 0 accuracy loss [2312.03788]|
| SQ-b/OPT-m        | 4/5/8 bits  | Vision Transf.  | Per-channel bias, region scaling | +23.35% accuracy improvement [2401.14895]        |
| QFeM, QFeP        | W8A8        | GLU-LLMs        | Outlier-spike mitigation, module- and prefix-level exemption | Restores near-FP16 quality [2405.14428]         |
| ABQ-LLM           | Arbitrary   | LLMs            | Distribution correction, bit balance, arbitrary precision | 1.6× speedup, 2.7× compression [2408.08554]      |
| SpinQuant         | W4A4KV4     | LLMs            | Learned rotations, Stiefel manifold opt. | Reduces quantization gap by 25–27 pts [2405.16406]|

SmoothQuant+ serves as a foundation for recent innovations in quantization-aware inference, influencing techniques for learned rotations, asymmetric range reduction, and mixed-precision assignment.

## 7. Open Problems and Future Directions

Several avenues remain for further advancement:
- **Fine-grained, data-adaptive smoothing:** Dynamic, per-layer or per-channel adaptation of smoothing parameters during inference or online calibration.
- **Extremely low bit-width quantization:** Robust methods capable of W2A2 or lower, potentially combining bit-balance with learned rotations and hardware-specific grouping [2408.08554].
- **Generalization to new architectures:** Broader adaptation to emerging model variants (hybrid architectures, non-GLU activations, multimodal encoders).
- **Hardware-agnostic implementation:** Further generalization of arbitrary-precision kernels to diverse accelerator backends and exploitation of new hardware primitives.
- **Unified treatment of outliers:** Joint approaches addressing both channel- and token-wise outliers, hybridizing ideas from SmoothQuant+, QFeM/QFeP, and rotation-based methods.

The SmoothQuant+ research line continues to inform the state of the art in quantized model deployment, setting methodological and empirical benchmarks for efficient, accurate transformer inference across domains [2312.03788, 2401.14895, 2405.14428, 2408.08554, 2405.16406, 2211.10438].

Source: https://www.emergentmind.com/topics/smoothquant-3df5d987-9e75-4152-9bd9-c85e8133444a