---
title: Weight-Only Quantization Techniques
url: https://www.emergentmind.com/topics/weight-only-quantization
type: topic
---

# Weight-Only Quantization Techniques

Weight-only quantization is a class of parameter compression techniques in deep neural networks that reduces the storage and memory-bandwidth requirements by mapping full-precision weight tensors to low-bitwidth representations, while leaving all computational activations (and often key–value caches in LLMs) in high precision. This approach targets memory- and bandwidth-bound inference regimes, especially in modern large language models (LLMs), vision transformers, and efficient edge deployments. Across deep learning, weight-only quantization enables substantial reductions in model size and inference latency with minimal accuracy loss, provided the quantizer is adapted to the complex distributions of weight tensors encountered in state-of-the-art architectures.

## 1. Theoretical Foundations and Problem Formulation

In canonical weight-only quantization, for a neural network with $L$ linear or convolutional layers, each with weights $W_\ell \in \mathbb{R}^{d_\mathrm{out} \times d_\mathrm{in}}$, a quantization function $Q_b(w)$ replaces full-precision weights with low-precision representations as follows:
\[
Q_b(w) = \mathrm{round}\left(\frac{w}{\Delta_b}\right)\Delta_b
\]
where $b$ denotes the bit-width and $\Delta_b$ is a step size, commonly chosen per layer or per group, to cover the dynamic range of the data [2509.12019].

Mixed-precision quantization extends this to assign different bit-widths $b_\ell$ to each layer, solving:
\[
\min_{\mathbf{b} \in \mathcal{B}^L} E(\mathbf{b}) \qquad 
\text{s.t.}\quad \sum_{\ell=1}^L \mathrm{Size}(b_\ell) \leq M_{\max},
\]
with $E(\mathbf{b})$ the model quality degradation metric and $M_{\max}$ the memory budget. This is a combinatorial, NP-complete optimization due to the discrete nature of $\mathcal{B}$ and layerwise assignment [2509.12019].

Rate–distortion theory offers a lower bound for Gaussianized weights:
\[
D(R) = \sigma^2 2^{-2R}
\]
where $R$ is the effective bitwidth and $D$ is the minimal achievable mean squared error [2509.20214]. Schemes that approach or exploit this bound through fractional-bit quantizers are optimal in the information-theoretic sense for normalized (rotated or Gaussianized) weight distributions.

## 2. Quantization Schemes and Algorithms

Several families of weight-only quantization algorithms have been developed:

**Symmetric/Uniform Quantization:** Assigns uniform grid levels to weights. Typically quantizes each block, channel, or entire tensor with parameters:
\[
s = \frac{\max(W) - \min(W)}{2^b-1}, \quad z = -\min(W)/s
\]
and applies
\[
Q(w) = s \cdot \mathrm{clip}(\mathrm{round}(w/s) + z, 0, 2^b-1)
\]
Widely used in “RTN” (round-to-nearest), basic QAT, and PTQ flows [2309.05516, 2306.00978, 2010.15979].

**Power-of-Two (PoT) and Additive PoT (APoT):** Weights are mapped to the nearest power-of-two value (or sum of several powers-of-two in APoT):
\[
Q_{\mathrm{PoT}}(w) = \operatorname{sign}(w) \cdot 2^{\mathrm{round}(\log_2|w|)}
\]
This approach matches hardware requirements of bitshift-based multipliers and improves accuracy at $b\leq4$ compared to uniform quantization [2203.05025].

**Activation-Aware and Outlier-Aware Quantization:** Methods such as AWQ and GWQ identify and protect salient weight channels either via activation statistics or by direct gradient sensitivity analysis:
- AWQ upweights the scale of the most activation-important input channels, preserving their representational capacity during low-bit quantization [2306.00978].
- GWQ computes gradients $\nabla_W \mathcal{L}(W; D_c)$ on one or a few calibration samples and retains the top $1\%$ of weights with the highest gradient magnitude in higher precision, quantizing the rest [2411.00850].

**Density-Aware Quantization (DAQ):** Aligns dynamic range to the densest 95% of weights (excluding outliers) via quantile estimation, then fine-tunes quantization parameters $(s, z)$ directly to minimize layerwise reconstruction loss with sign-SGD optimization [2410.12187].

**Low-Rank and Flexible Scaling:** LRQ learns a low-rank scaling matrix $S = UV^T$ to tailor individual weight scaling with dramatically fewer parameters than full-rank scaling, reducing overfitting and enhancing generalization in low-bit regimes [2407.11534].

**Blockwise, Groupwise, and Per-Channel Quantization:** FineQuant adaptively determines group size $G$ by a heuristic that balances range expansion with quantization error, storing blockwise scales per column or channel. Per-input-channel (per-IC) quantization isolates activation outliers, essential below $4$ bits [2309.15531, 2308.09723].

**Probabilistic and Bayesian Approaches:** Probabilistic Weight Fixing models each weight as a Gaussian with learned mean and uncertainty, iteratively fixing subsets of weights to cluster centers based on relative distance in Mahalanobis space. This encourages entropy-minimized, robust weight clustering with global codebook assignments [2309.13575].

**Mixed-Precision and Fractional Bitwidths:** AMQ [2509.12019] and Q-Palette [2509.20214] construct configurations across per-layer bitwidths and quantizer types, using genetic search (e.g., NSGA-II), surrogate predictors, and information-theoretic assignment to target the Pareto frontier in memory–accuracy and latency–accuracy spaces.

## 3. Optimization Workflows, Search, and Calibration

The complex search space for optimal quantization assignments in modern LLMs (e.g., $3^{224}$ for Llama-2 7B) prohibits brute-force enumeration. Recent systems develop:

- **Combinatorial Search with NSGA-II:** AMQ prunes inert layers via per-layer sensitivity, builds proxy quantized models ($Q^{\rm HQQ}$), uses a surrogate (RBF) quality predictor, and applies multi-objective evolutionary algorithms to sample and converge on optimal configurations [2509.12019].
- **Integer Programming for Mixed Scheme Selection:** Q-Palette defines quantizer/bitwidth selection as a multiple-choice knapsack problem, optionally incorporating group fusion constraints for hardware kernel fusion [2509.20214].
- **Heuristic Group Size Selection:** FineQuant iteratively shrinks the quantization group while monitoring the blow-up in dynamic range to achieve over 94% BLEU recovery compared to fixed granularity [2308.09723].
- **Sensitivity and Hessian-Based Scoring:** AdaDim algorithmically allocates bits and grouping axes where quantization loss is highest, coupling with blockwise PTQ (RTN, GPTQ) [2309.15531].
- **SignSGD for Rounding and Clipping:** SignRound employs a low-cost 200-step signed gradient optimization of the quantization offsets and clipping parameters, minimizing direct reconstruction loss without introducing inference overhead [2309.05516].

Most methods utilize a small calibration dataset (1–100 samples) and minimize layer/block output deviation to preserve output distribution or logits (DAQ, LRQ, SignRound, GWQ), often with straight-through estimator or surrogate gradient methods to overcome quantizer non-differentiability.

## 4. Hardware, Inference Acceleration, and Kernel Implementation

Weight-only quantization directly reduces memory traffic and enables custom hardware acceleration schemes:

**Kernel Fusion and GEMM Optimization:** Methods such as TinyChat (AWQ) and FineQuant develop fused int4/int8-to-FP16 GEMM kernels that perform on-the-fly dequantization using block or per-channel scales, minimizing DRAM round-trips and matching arithmetic intensity for memory-bound LLM decoding [2306.00978, 2308.09723].

**Shift-Based Accumulation:** Power-of-two and APoT quantization replace multipliers in MAC units with barrel shifters and adders, reducing both area and energy by up to $6\times$ and $2\times$ respectively compared to uniform 8×8 bit multipliers [2203.05025].

**Batch- and Fusion-Aware Quantization:** Q-Palette extends QTIP-style TCQ and VQ fractional-bit quantization to batch sizes up to 16 and enables kernel-level layer fusion for fused multi-head attention and FFN GEMMs [2509.20214].

**Deployment Metrics:** AWQ with TinyChat and FineQuant achieve $3.2$–$3.9\times$ throughput increases across Llama-2, MPT, and Falcon families, with 4× reduction in weight storage [2306.00978, 2308.09723]. Quantization reduces model sizes proportionally: a 70B-parameter model can be compressed from $\approx280$GB (FP32) to $\approx17.5$ GB at 2 bits [2309.05516].

## 5. Empirical Performance and Benchmarks

Comprehensive studies across architectures and tasks demonstrate that modern weight-only quantization can nearly reach full-precision accuracy at 3–4 bits, and with advanced schemes, even at 2 bits in selected settings.

| Method             | Model       | Bitwidth   | Perplexity (WikiText2) | Zero-Shot Acc (%) | Notes            |
|--------------------|-------------|------------|-----------------------|-------------------|------------------|
| FP16               | LLaMA-2-7B  | 16         | 5.46                  | 70.49             | Baseline         |
| AMQ                | LLaMA-2-7B  | 3          | –                     | 65.59             | +1.96 over BitStack, [2509.12019]|
| DAQ                | LLaMA-2-7B  | 4 (NF4)    | 5.60                  | –                 | 19.6%-22.8% lower perplexity loss vs. AWQ, [2410.12187] |
| AWQ                | LLaMA-2-7B  | 4          | 5.60                  | 70.13             | $≈$FP16, [2306.00978] |
| SignRound          | Mistral-7B  | 4          | –                     | 62.33             | W4G-1, [2309.05516] |
| FineQuant          | OPT-175B    | 4          | –                     | –                 | $3.65\times$ throughput, $<0.5$ BLEU drop, [2308.09723] |
| GWQ                | LLaMA-2-7B  | $\sim$4    | 5.53                  | 60.58             | 1.2$\times$ speedup, [2411.00850] |
| AdaDim+GPTQ        | LLaMA-7B    | 3          | 9.5                   | 49.5              | [2309.15531]     |
| LRQ                | LLaMA-2-7B  | 3          | 6.48                  | 59.07             | State-of-the-art weight-only, [2407.11534] |

Results confirm that post-training, QAT, low-rank, and mixed-precision methods can all achieve near-baseline accuracy for mainstream LLMs and vision models in the 3–4 bit regime. AMQ consistently outperforms previous any-size (BitStack, PB-LLM) and uniform baselines under matched memory constraints [2509.12019]. Q-Palette’s mixed-scheme allocation with fractional bitwidth pushes the Pareto frontier in both memory/perplexity and latency/perplexity [2509.20214].

## 6. Best Practices, Limitations, and Open Challenges

**Practical recommendations:**
- For 4-bit quantization, per-input-channel grouping and adaptive or sensitivity-driven allocation (AdaDim, GWQ) are recommended on all current LLMs [2309.15531, 2411.00850].
- At 3 or fewer bits, advanced or hybrid approaches (probabilistic, activation- or gradient-aware, low-rank scaling) are critical to avoid accuracy collapse.
- Hardware deployment favors block-/groupwise quantization (group size 64–128), with dequantization fused to GEMM kernels [2308.09723, 2306.00978].
- For resource-constrained edge or batch-1 inference, develop and deploy kernels supporting non-integer or fractional bitwidths per layer for maximal throughput at fixed accuracy [2509.20214].

**Limitations and open issues:**
- The combinatorial explosion of mixed-precision assignments requires advanced AutoML, genetic search, or integer programming (e.g. in AMQ, Q-Palette).
- Sensitivity analysis is costly for very large models, motivating faster proxies (e.g. activation-independent simulated quantization).
- Calibration sets remain a bottleneck for ultra-low bitwidths; methods that minimize data requirements (GWQ) are advancing.
- Hardware support for truly arbitrary per-layer or per-block bitwidths and non-uniform quantization (APoT, VQ, TCQ) still lags, especially on FPGAs and custom ASICs [2203.05025, 2509.20214].

## 7. Outlook and Related Research Directions

Weight-only quantization continues to be a dominant axis for achieving efficient inference and deployment in large-scale neural models. Recent directions include:
- Automated Pareto-frontier search for hardware–accuracy trade-offs at deployment-time [2509.12019, 2509.20214].
- Bayesian and probabilistic quantization for entropy-minimized, robust, extremely low-bitweight models [2309.13575].
- Integration with pruning, clustering, and kernel fusion for multi-level compression [1811.01907, 2509.20214].
- Extensions to SNNs and time-adaptive quantization with temporal resource allocation [2511.17567].

For advanced LLMs, emerging empirical and theoretical results demonstrate that with proper axis grouping, sensitivity control, and cross-layer optimization, weight-only quantization below 4 bits can match or approach full-precision performance, enabling resource-efficient deployment at massive scale.

---

**Key references:**  
[2509.12019] (AMQ), [2410.12187] (DAQ), [2306.00978] (AWQ), [2411.00850] (GWQ), [2309.15531] (AdaDim), [2407.11534] (LRQ), [2308.09723] (FineQuant), [2309.05516] (SignRound), [2509.20214] (Q-Palette), [1811.01907] (ADMM compression), [2203.05025] (PoT/APoT), [2309.13575] (PWFN), [2511.17567] (TaWQ).

Source: https://www.emergentmind.com/topics/weight-only-quantization