Dissecting Quantization Error: A Concentration-Alignment Perspective

Abstract: Quantization can drastically increase the efficiency of large language and vision models, but typically incurs an accuracy drop. Recently, function-preserving transforms (e.g. rotations, Hadamard transform, channel-wise scaling) have been successfully applied to reduce post-training quantization error, yet a principled explanation remains elusive. We analyze linear-layer quantization via the signal-to-quantization-noise ratio (SQNR), showing that for uniform integer quantization at a fixed bit width, SQNR decomposes into (i) the concentration of weights and activations (capturing spread and outliers), and (ii) the alignment of their dominant variation directions. This reveals an actionable insight: beyond concentration - the focus of most prior transforms (e.g. rotations or Hadamard) - improving alignment between weight and activation can further reduce quantization error. Motivated by this, we introduce block Concentration-Alignment Transforms (CAT), a lightweight linear transformation that uses a covariance estimate from a small calibration set to jointly improve concentration and alignment, approximately maximizing SQNR. Experiments across several LLMs show that CAT consistently matches or outperforms prior transform-based quantization methods at 4-bit precision, confirming the insights gained in our framework.

• The paper decomposes linear-layer quantization error into bit width, concentration, and weight–activation alignment, showing that orthogonal rotations improve concentration but cannot change alignment.
• The paper introduces calibration-based Concentration–Alignment Transforms (CAT) that combine block-diagonal alignment with Hadamard rotation, delivering up to 10 dB per-layer SQNR gains and often making W4A4 approach or exceed W6A6.
• The paper demonstrates that CAT improves practical LLM quantization, reducing Llama 2 7B WikiText perplexity to 6.11 versus 9.31 for QuaRot and reaching 66.22% average zero-shot accuracy without transform training.

Overview

This paper develops a principled account of quantization error in the linear layers of LLMs, decomposing the signal-to-quantization-noise ratio (SQNR) into interpretable factors and using the resulting theory to design a new family of function-preserving transforms. The authors, from Qualcomm AI Research, identify a component of quantization error—weight–activation alignment—that is invisible to rotation-based transforms such as Hadamard matrices, and propose Concentration–Alignment Transforms (CAT) that jointly optimize both components. The practical outcome is a block-diagonal, calibration-based transform that matches or outperforms trained rotation-based methods at W4A4, often approaching W6A6-level SQNR.

The concentration–alignment framework

Starting from the SQNR of a quantized linear layer $y = Wx$, and invoking classical decorrelation assumptions on quantization noise (Widrow and Kollár's statistical theory of quantization, Gersho's analysis), the paper derives an approximation in which the joint SQNR factors into a bit-width term and two scale-invariant statistics:

$$\text{SQNR}(Q(W)Q(x)) \approx 12\left(N(b_x)^2 C(x) \parallel N(b_w)^2 C(W)\right) A(W, x)$$

where $\parallel$ denotes the parallel (harmonic) combination, $C(\cdot)$ is a concentration term capturing the spread of activations or weight rows relative to their quantization range (a kurtosis-like ratio of squared $\ell_2$ norm to squared range, sensitive to outliers), and $A(W,x)$ is an alignment term measuring the similarity between the dominant variation directions of weights and activations, via the ratio of expected output energy $\mathbb{E}\|Wx\|^2$ to the Frobenius-norm upper bound $\|W\|_F^2 \mathbb{E}\|x\|^2$.

Two structural properties of this decomposition drive the rest of the paper. First, the joint SQNR is governed by the harmonic sum of the activation and weight terms, so it is dominated by the worse of the two components; empirically, activation SQNR is typically lower than weight SQNR, which explains why activation-side interventions (e.g., SmoothQuant-style scaling) are disproportionately effective. Second, and more consequential, alignment is invariant under orthogonal transformations: for any orthogonal $Q$, $A(WQ^T, Qx) = A(W,x)$. Since Hadamard and learned-rotation methods (QuaRot, SpinQuant) operate exclusively through orthogonal maps, they can improve concentration but provably cannot improve alignment. The paper validates the SQNR approximation empirically on Llama v3.2 1B and Qwen v3 layers across W4A4–W8A8, showing close agreement for nearly all layers in the 5–50 dB regime; the main deviations are layers dominated by massive activations (e.g., the [BOS] token outlier in down_proj), where the approximation underestimates the true SQNR.

What existing transforms do and do not fix

The empirical analysis across LLM layers yields a clear taxonomy. Without transforms, activation distributions are heavier-tailed than Laplace (poor concentration due to outliers), while weights sit between Laplace and Gaussian. Hadamard rotations push both weights and activations toward Gaussian concentration, with gains exceeding 10 dB in activation concentration for large layers such as down_proj in Qwen v3 8B. SmoothQuant-style channel scaling trades activation concentration for weight concentration—beneficial overall because the joint SQNR is bottlenecked by the worse term—and has a mildly positive but limited effect on alignment.

The critical finding is on alignment: layers such as down_proj, o_proj, and v_proj exhibit substantially suboptimal alignment, with more than 10 dB of headroom relative to the achievable optimum. Because rotations cannot touch this term, QuaRot and SpinQuant leave this entire error component unaddressed. Since alignment acts as a multiplicative factor on both weight and activation SQNR, improving alignment by a factor $$\text{SQNR}(Q(W)Q(x)) \approx 12\left(N(b_x)^2 C(x) \parallel N(b_w)^2 C(W)\right) A(W, x)$$0 is equivalent to increasing both bit widths by approximately $$\text{SQNR}(Q(W)Q(x)) \approx 12\left(N(b_x)^2 C(x) \parallel N(b_w)^2 C(W)\right) A(W, x)$$1 bits—so a 10 dB alignment gain is roughly worth two extra bits on each side.

The CAT transform

The alignment-optimal transform admits a closed-form solution via the matrix geometric mean of the inverse activation autocorrelation and the weight autocorrelation:

$$\text{SQNR}(Q(W)Q(x)) \approx 12\left(N(b_x)^2 C(x) \parallel N(b_w)^2 C(W)\right) A(W, x)$$2

which maps the principal variation directions of weights and activations into a common space. The maximum attainable alignment is determined by the eigenvalue distribution of the output covariance, $$\text{SQNR}(Q(W)Q(x)) \approx 12\left(N(b_x)^2 C(x) \parallel N(b_w)^2 C(W)\right) A(W, x)$$3. The full-rank $$\text{SQNR}(Q(W)Q(x)) \approx 12\left(N(b_x)^2 C(x) \parallel N(b_w)^2 C(W)\right) A(W, x)$$4 is impractical online, so the paper constructs CAT(block): a two-step transform that first applies a block-diagonal approximation of $$\text{SQNR}(Q(W)Q(x)) \approx 12\left(N(b_x)^2 C(x) \parallel N(b_w)^2 C(W)\right) A(W, x)$$5 (blocks of size $$\text{SQNR}(Q(W)Q(x)) \approx 12\left(N(b_x)^2 C(x) \parallel N(b_w)^2 C(W)\right) A(W, x)$$6, estimated from a small calibration set of 128 DCLM-edu sequences) and then composes with a Hadamard matrix to maximize concentration, exploiting rotation invariance of alignment. Notably, for $$\text{SQNR}(Q(W)Q(x)) \approx 12\left(N(b_x)^2 C(x) \parallel N(b_w)^2 C(W)\right) A(W, x)$$7 the construction reduces to a diagonal scaling with entries $$\text{SQNR}(Q(W)Q(x)) \approx 12\left(N(b_x)^2 C(x) \parallel N(b_w)^2 C(W)\right) A(W, x)$$8, a principled variant of SmoothQuant.

Per-layer SQNR analysis shows CAT(block) yields gains of up to 10 dB on gate_proj and down_proj, and transformed W4A4 SQNR exceeds even W6A6 SQNR on all Qwen v3 8B layers except o_proj.

Experimental results

The evaluation covers Llama 2 7B, Llama 3 8B, Llama 3.2 1B instruct, Ministral 8B instruct, and Qwen 3 8B, with dynamic per-token asymmetric activation and KV-cache quantization, symmetric per-channel weight quantization, and both RTN and GPTQ weight quantizers. Calibration uses DCLM-edu rather than WikiText to avoid biasing the WikiText-2 perplexity evaluation.

The headline results are striking for a training-free method. Without any transform training, CAT(block) with RTN reduces Llama 2 7B WikiText perplexity to 6.11 versus 9.31 for QuaRot and 6.96 for SpinQuant, and achieves 66.22% average zero-shot accuracy versus 57.54% for QuaRot—already competitive with trained FlatQuant (6.05 / 66.67%). With training, CAT(block) generally outperforms FlatQuant on zero-shot accuracy across models (e.g., 67.34% vs. 66.67% on Llama 2 7B; 69.66% vs. 68.34% on Llama 3 8B), though FlatQuant retains an edge in perplexity on several settings. Under GPTQ, trained CAT(block) and FlatQuant are essentially on par. A secondary observation is that GPTQ adds little to CAT or FlatQuant, plausibly because their learnable weight clipping already captures much of GPTQ's benefit.

Limitations and open questions

The paper is explicit that the alignment-optimal transform is a full-rank matrix whose online application would negate the efficiency benefits of quantization; the block-diagonal approximation is presented as a case study rather than a proven optimum, and the authors concede they do not establish which structured approximation (diagonal, Kronecker, block-diagonal, or permutations) offers the best speed–accuracy trade-off. The SQNR framework itself relies on negligible clipping error and decorrelated, uniformly distributed quantization noise, and breaks down for layers dominated by massive activations. The theory also addresses uniform integer quantization; the concurrent WUSH work derives analogous optimal transforms for floating-point formats, suggesting the framework's extension to other number formats remains open.

Conclusion

By factoring linear-layer SQNR into bit width, concentration, and alignment, this paper reframes transform-based quantization: outlier suppression (concentration) addresses only part of the error, and the alignment term—which rotations cannot modify—offers quantifiable headroom of roughly two bits' worth of SQNR in the worst layers. CAT operationalizes this insight with a calibration-based block-diagonal transform that, with or without training, matches or exceeds the accuracy of state-of-the-art trained methods at W4A4. The main open problem left by the paper is the design of structured, hardware-friendly approximations to the full-rank alignment-optimal transform.