High-Rate Quantized Matrix Multiplication: Theory and Practice

Abstract: This work investigates the problem of quantized matrix multiplication (MatMul), which has become crucial for the efficient deployment of LLMs. We consider two settings: 1) Generic MatMul, where both matrices must be quantized (weight+activation quantization); and 2) weight-only quantization, where the second matrix is only known through covariance matrix $Σ_X$ of its columns. For each setting, we first review the fundamental information-theoretic tradeoff between quantization rate and distortion (high-rate theory), and then analyze the performance of several popular quantization schemes, comparing them to these fundamental limits. Specifically, we discuss rate loss (compared to information theoretic optima) of absmax INT and floating-point (FP) quantization, for which we also derive remarkably accurate heuristic approximations. Weight-only quantization is related to the problem of weighted mean squared error (WMSE) source coding, whose classical (reverse) waterfilling solution dictates how one should distribute rate between coordinates of the vector. We show how waterfilling can be used to improve practical LLM quantization algorithms (GPTQ), which at present allocate rate equally. This new scheme (termed ``WaterSIC'') only uses scalar INT quantizers, but its high-rate performance is basis free (it depends only on the determinant of $Σ_X$ and, thus, unlike existing schemes, is immune to applying random rotations) and is within a multiplicative factor of $\frac{2πe}{12}$ (or 0.25 bit/entry) of the information-theoretic distortion limit (!). GPTQ's performance is affected by the choice of basis, but for a random rotation and actual $Σ_X$ from Llama-3-8B we find GPTQ to be within 0.1 bit (depending on the layer type) of WaterSIC, suggesting that GPTQ with random rotation is also near optimal (for high-rate quantization).

• The paper presents high-rate quantization benchmarks for matrix multiplication addressing both generic and weight-only quantization settings.
• Generic MatMul quantization benchmarks are achieved through random rotation and nested lattice quantization, showing INT and FP format performance with specific effective rates.
• A close connection between GPTQ and WaterfIC is demonstrated, with WaterSIC achieving near-optimal distortion and proving rotation-invariant in theory and practice.

Overview

This survey by Ordentlich and Polyanskiy develops information-theoretic benchmarks for quantized matrix multiplication (MatMul) and uses them to audit widely deployed LLM quantization methods. The paper treats two settings. In generic MatMul (weight-plus-activation quantization), no statistics of the matrices are available at quantizer design time; here the high-rate fundamental limit is $$D^*_{ij} = \frac{\|a_i\|^2\|b_j\|^2}{n}\cdot 2\cdot 2^{-2R}$$ for the $(i,j)$ entry of $A^\top B$. In weight-only quantization, only one factor is compressed and its quantizer may exploit the activation covariance $\Sigma_X$ estimated from a calibration set; here the limit is governed by reverse waterfilling over $\Sigma_X$, yielding $D \approx |\Sigma_X|^{1/n} 2^{-2R}$ in the high-rate regime. Throughout, "rate" is used in the information-theoretic sense ($NR$ bits jointly dependent on all entries), in contrast to entrywise scalar formats such as INT8/FP8, and the analysis restricts attention to the high-resolution regime where quantization error behaves like additive uniform noise.

Fundamental limits for generic MatMul

For rate-$R$ schemes with encoders, decoder, and shared randomness $\omega$, the worst-case distortion over matrices with unit-norm columns admits a scheme based on random rotation plus nested lattice quantization achieving, uniformly over all inputs,

$$E\big((A^\top B)_{ij}-(\widehat{A^\top B})_{ij}\big)^2 \le \frac{\|a_i\|^2\|b_j\|^2}{n}\left(2\cdot 2^{-2R}+o(1)\right).$$

Two negative results motivate the need for randomization: any non-randomized deterministic scheme suffers $(i,j)$0 distortion due to adversarial alignment between one matrix and the other's quantization noise, and single-sketch Johnson–Lindenstrauss estimators also incur $(i,j)$1 distortion from estimator variance. For iid Gaussian inputs, the exact finite-rate limit was previously established: distortion $(i,j)$2 with a phase transition at critical rate $(i,j)$3, below which optimal schemes must combine sketching with vector quantization. In the high-rate regime this collapses to $(i,j)$4, matching the universal achievability bound — so the benchmark above is essentially tight and cannot be improved beyond lower-order terms.

Effective rates of INT and FP multipliers

The paper derives closed-form high-rate approximations for absmax-scaled scalar constellations, validated empirically on activations of Llama-3-8B layer 15 as "remarkably accurate." Under an additive uniform-noise model, INT$(i,j)$5 with absmax scaling achieves distortion

$(i,j)$6

where $(i,j)$7 involves per-vector $(i,j)$8 ratios and can be as large as $(i,j)$9 for aligned data — a multiplicative gap to optimality that random rotation reduces to at most $A^\top B$0. Consequently, under Hadamard rotation the effective rate is

$A^\top B$1

e.g., $A^\top B$2 bits for $A^\top B$3: roughly 1.25 of every 8 bits are lost to the per-vector scale.

The FP analysis appears to be new. Introducing an "infinite-exponent independent noise" (IEIN) model with dithered absmax scaling, the authors obtain $A^\top B$4 — FP buys about 2.24 free bits relative to INT at equal bit count, because its geometric spacing matches the signal's scale-invariance. Notably, the FP penalty factor $A^\top B$5 has expectation exactly 1 for iid data and at most 3 under random rotation, but can be below 1 on real Llama-3-8B matrices without rotation. This yields a somewhat contrarian practical conclusion: full-vector Hadamard rotation is unnecessary and actually harmful for FP8 quantization of these layers, whereas it is crucial for INT — providing what the authors argue is the first rigorous explanation of prior empirical observations that rotations hurt FP quantization.

For 4-bit formats, both absmax INT4 and FP4 degrade severely (overload errors dominate for E2M1), motivating multi-scaling: NVINT4 and NVFP4 use block scales of 8/16 bits per block of 16 entries (effective rate 4.5 bits). A heuristic argument shows NVFP4's distortion is typically bounded by the IEIN prediction for $A^\top B$6. Experiments at $A^\top B$7 bits show NestQuant with $A^\top B$8 scales outperforming NVINT4/NVFP4 by roughly 0.6 effective bits, attributed to better lattice shaping (replacing the cubic cells of $A^\top B$9) and multi-scale overload control.

Weight-only quantization: waterfilling limits

Weight-only quantization with objective $\Sigma_X$0 is a weighted-MSE source coding problem. For Gaussian weights, a data-processing argument gives the classical reverse-waterfilling solution, whose high-rate form is $\Sigma_X$1. An isotropic codebook with $\Sigma_X$2-oblivious encoding achieves only $\Sigma_X$3 (arithmetic mean of eigenvalues); the gain from optimal allocation equals half the AM–GM log-gap, illustrated across all Llama-3-8B layers. Two structural observations sharpen the picture:

• Informed vs. uninformed decoding: since dequantization must run online without access to $\Sigma_X$4, the practically relevant constraint is a $\Sigma_X$5-oblivious decoder with adaptive encoding. Citing forthcoming work, the paper reports that even an isotropic Gaussian codebook with $\Sigma_X$6-aware nearest-codeword rounding attains the waterfilling distortion up to higher-order terms, universally over $\Sigma_X$7 (within 0.1 bit at all rates). The dominant loss in naive schemes therefore comes from suboptimal rounding, not suboptimal codebooks — but solving that rounding exactly is a closest vector problem, known to be NP-hard.
• Cholesky interpretation: writing $\Sigma_X$8 with upper-triangular $\Sigma_X$9, the diagonal entries $\Sigma_X$0 are the variances of orthogonal innovations of the activation process, and they replace eigenvalues as the objects controlling achievable distortion.

GPTQ as successive cancellation and the WaterSIC algorithm

The paper recasts GPTQ/LDLQ as successive interference cancellation (equivalently Babai's nearest-plane algorithm, or V-BLAST-style filtering): quantize coordinates sequentially from last to first, feeding back partially cancelled contributions. With uniform spacing $\Sigma_X$1, SIC's decision region is a rectangular box with side lengths $\Sigma_X$2, giving distortion proportional to the arithmetic mean of squared Cholesky diagonals,

$\Sigma_X$3

which is basis-dependent. The proposed WaterSIC variant instead chooses per-coordinate spacings $\Sigma_X$4, equalizing each coordinate's contribution. Its distortion becomes

$\Sigma_X$5

since $\Sigma_X$6. Because this expression depends on $\Sigma_X$7 only through its determinant, WaterSIC is provably immune to random rotations, unlike GPTQ. It uses only scalar integer quantizers plus entropy coding or shaping, yet sits within a multiplicative factor $\Sigma_X$8 of the information-theoretic optimum — equivalently within Koshelev's celebrated 0.25-bit constant. This is the paper's central theoretical claim: a trivial modification of GPTQ (per-coordinate scales derived from Cholesky diagonals, amortizable over the many output neurons of a layer) closes essentially the entire gap between current practice and the waterfilling limit.

Empirical findings on Llama-3-8B

Three experimental results stand out. First, normalized-error tables confirm the INT8 and FP8 distortion formulas to within hundredths of a bit of predicted effective rate. Second, comparing GPTQ and WaterSIC with entropy coding against the exact finite-$\Sigma_X$9 waterfilling curve confirms the asymptotic $D \approx |\Sigma_X|^{1/n} 2^{-2R}$0 ratio. Third, and most consequential, when GPTQ is applied after a random rotation its actual distortion comes within roughly 0.1 bit of WaterSIC on real Llama-3-8B covariances — implying that GPTQ with random rotation is already near-optimal for high-rate weight quantization, even though its guarantee is basis-dependent. The residual basis dependence is explained via the innovation-energy identity: GPTQ's cost is the arithmetic mean of $D \approx |\Sigma_X|^{1/n} 2^{-2R}$1, minimized in the PCA basis but improved by rotation because the natural (Adam-induced, axis-aligned-outlier) basis is "privileged" relative to random bases. The paper further derives an accurate spectrum-only approximation to rotated Cholesky diagonals, interpolating smoothly from arithmetic to harmonic mean of the eigenvalues, enabling performance prediction of rotated GPTQ without computing decompositions.

Limitations and open questions

The authors are explicit about several caveats. All analyses assume the high-resolution regime and become unreliable below roughly 2 bits per entry, where sketching, shrinkage, and weight structure matter; extending waterfilling-based guarantees to low rates is left open. Entropy coding, required to realize WaterSIC's guarantee, decompresses about an order of magnitude slower than raw loads on GPUs, so its deployment value is uncertain, while GPU-friendly shaping (rectangular or coset-based) is either suboptimal or computationally hard. The $D \approx |\Sigma_X|^{1/n} 2^{-2R}$2 gap itself could be shrunk by joint quantization over higher-dimensional lattices (down to $D \approx |\Sigma_X|^{1/n} 2^{-2R}$3), at the cost of harder entropy coding. WaterSIC's per-coordinate scales must be communicated and amortize only when $D \approx |\Sigma_X|^{1/n} 2^{-2R}$4; behavior for narrow layers is unaddressed. Finally, the near-equivalence of rotated GPTQ and WaterSIC is established only empirically for Llama-3-8B covariances, and bounding the worst-case gap over admissible spectra remains an open problem.

Conclusion

The paper supplies a unified high-rate theory connecting information-theoretic optima for quantized MatMul to the practical formats and algorithms used in LLM inference. Its main deliverables are: tight benchmarks showing INT loses ~1.24 bits to scaling and FP gains ~2.24 bits relative to INT; a demonstration that rotation helps INT but hurts FP on realistic activations; proof that a waterfilled-spacing variant of GPTQ (WaterSIC) is within 0.25 bits of the optimal weight-only distortion while being rotation-invariant; and evidence that vanilla GPTQ with random rotation is already nearly optimal in practice. The remaining open territory lies chiefly in low-rate regimes, GPU-compatible shaping, and closing the final sub-bit gaps identified by the theory.