Randomized Hadamard Quantization

Updated 6 July 2026

Randomized Hadamard-based quantization is a technique that applies fast orthogonal mixing using normalized Hadamard matrices and random sign matrices to precondition data for low-bit quantization.
It leverages multiple compositions of Hadamard transforms to reshape distributions towards Gaussianity, ensuring unbiased reconstruction and provable error guarantees.
Recent methods integrate these transforms with blockwise, vector, and lattice quantization to enhance compression in large language models and related applications.

Searching arXiv for relevant papers on randomized Hadamard-based quantization and closely related Hadamard-preconditioned quantization methods. Found the following potentially relevant arXiv papers:

(Ben-Basat et al., 7 May 2026) — "Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven"
(Feng et al., 13 May 2026) — "Provable Quantization with Randomized Hadamard Transform"
(Tseng et al., 2024) — "QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks"
(Domb et al., 22 Jun 2026) — "HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models"
(Zagitov et al., 28 May 2026) — "HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization"
(Lin et al., 20 Apr 2026) — "DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization"
(Vicentino, 30 Mar 2026) — "PolarQuant: Optimal Gaussian Weight Quantization via Hadamard Rotation for LLM Compression"
(Federici et al., 11 Jun 2025) — "HadaNorm: Diffusion Transformer Quantization through Mean-Centered Transformations"
(Sharma et al., 25 May 2026) — "QAM-W: Joint 2D Codebook Quantization for LLM Weights via Hadamard Rotation and Activation-Aware Scaling"
(Pereira et al., 19 Jun 2026) — "Fast-TurboQuant: A Multiplier-Free Online Vector Quantization Approach" Randomized Hadamard-Based Quantization denotes a family of quantization schemes in which an orthogonal mixing stage—typically a randomized Hadamard transform of the form $x \mapsto HDx$ , or a closely related composition—is applied before scalar, blockwise, or lattice quantization. The purpose of the transform is to redistribute energy across coordinates, reduce axis-aligned outliers, and expose a representation whose marginals are closer to Gaussian or otherwise isotropic, while preserving the fast $O(d\log d)$ complexity of Walsh–Hadamard structure rather than the $\Theta(d^2)$ cost of dense random rotations. The paradigm now spans LLM weight PTQ, KV-cache compression, embedding quantization, and approximate nearest-neighbor pipelines, but recent work also makes a sharp distinction between genuinely randomized Hadamard methods and deterministic Hadamard-preconditioned quantizers that use the same orthogonal-mixing intuition without stochastic preprocessing (Ben-Basat et al., 7 May 2026, Feng et al., 13 May 2026, Tseng et al., 2024).

1. Core transform formulations

The canonical randomized Hadamard transform uses a normalized Hadamard matrix $H$ together with a random diagonal Rademacher sign matrix $D$ . In the simplest case, one applies

$y = HDx,$

with $H$ orthogonal and $D=\operatorname{diag}(\pm 1)$ . More elaborate constructions compose multiple such stages,

$R_k = H D_k H D_{k-1}\cdots H D_1,$

or apply them two-sidedly around a weight matrix. QuIP# uses exactly this latter pattern: for a weight matrix $W$ and proxy Hessian $O(d\log d)$ 0, the transformed objects are

$O(d\log d)$ 1

where $O(d\log d)$ 2 are orthogonal scaled Hadamard matrices and $O(d\log d)$ 3 are random diagonal sign matrices (Tseng et al., 2024).

In modern model-compression pipelines, the transform is rarely global over an entire tensor. HyperQuant applies a per-tile RHT along the contraction axis, with

$O(d\log d)$ 4

and chooses tile sizes matched to MMA structure, with default tile size $O(d\log d)$ 5 on H100/Blackwell (Domb et al., 22 Jun 2026). This tiling convention reflects a recurring systems constraint: Hadamard preprocessing is most useful when it can be fused with blockwise quantization, dequantization, or matrix-multiply scheduling rather than treated as an independent dense linear operator.

The same structural idea also appears outside dense weight PTQ. In online embedding compression, Fast-TurboQuant replaces a dense Gaussian rotation by

$O(d\log d)$ 6

where $O(d\log d)$ 7 is zero-padded to the next power-of-two dimension $O(d\log d)$ 8 so that FWHT is applicable (Pereira et al., 19 Jun 2026). This illustrates a broad design pattern: when randomized orthogonal preprocessing is needed but dense random matrices are too expensive, Hadamard structure supplies a fast surrogate that preserves orthogonality and admits matrix-free implementation.

2. Distribution shaping, incoherence, and Gaussianization

The central statistical role of randomized Hadamard preprocessing is distribution shaping. Under a dense uniform random rotation, each coordinate of a rotated unit vector follows the spherical-coordinate law and becomes asymptotically Gaussian after $O(d\log d)$ 9 scaling. That is the ideal model behind many quantizer analyses. A single RHT, however, is not sufficient in the worst case. For

$\Theta(d^2)$ 0

one RHT produces coordinates in $\Theta(d^2)$ 1, which is far from Gaussian (Ben-Basat et al., 7 May 2026). This failure mode is why the recent theory distinguishes carefully between one, two, and three RHT compositions.

For scalar or coordinatewise quantization, two RHTs suffice. After two RHTs, every fixed coordinate of the normalized rotated vector is within $\Theta(d^2)$ 2 of a standard Gaussian in both Kolmogorov distance and $\Theta(d^2)$ 3-Wasserstein distance, uniformly over all inputs. For vector quantization over fixed-size blocks, two RHTs may still fail because marginal Gaussianity does not control within-block conditional correlation; three RHTs are required to obtain decaying coordinate covariance and thereby recover Gaussian/URR-style VQ guarantees up to a vanishing additive term (Ben-Basat et al., 7 May 2026).

This probabilistic picture aligns with the incoherence perspective of QuIP#. QuIP# shows that randomized Hadamard preprocessing produces transformed weights that are “roughly ball-shaped sub-Gaussian,” and it proves sharper incoherence guarantees than the older Kronecker-factorized preprocessing used in QuIP. Specifically, the transformed Hessian and weights are $\Theta(d^2)$ 4-incoherent with

$\Theta(d^2)$ 5

which feeds directly into better Hessian-aware quantization bounds (Tseng et al., 2024).

A distinct, deterministic route to Gaussianization appears in PolarQuant. There the argument is not algorithmic randomness but a spherical heuristic: normalize each $\Theta(d^2)$ 6-weight block to the unit sphere, apply a normalized Walsh–Hadamard transform, rescale by $\Theta(d^2)$ 7, and treat the resulting coordinates as approximately $\Theta(d^2)$ 8. The paper’s Proposition 1 states that if the normalized block is uniformly distributed on $\Theta(d^2)$ 9, then each coordinate satisfies

$H$ 0

and for $H$ 1 the reported Kolmogorov–Smirnov statistic between empirical rotated weight coordinates and $H$ 2 is typically below $H$ 3 (Vicentino, 30 Mar 2026). This is a different justification from classical RHT analysis, but it serves the same end: transform the source into a regime where low-bit quantization is far easier.

3. Scalar quantization, dither, and provable guarantees

The most explicit scalar-quantization theory for randomized Hadamard preprocessing is given by “Provable Quantization with Randomized Hadamard Transform” (Feng et al., 13 May 2026). That work studies worst-case vector quantization on the sphere under the transform

$H$ 4

followed by coordinatewise scalar quantization in a Gaussian-adapted quantile domain. The scalar design is based on

$H$ 5

with the identity $H$ 6. The paper then introduces a shared scalar dither $H$ 7 and constructs an unbiased reconstruction map $H$ 8 such that the scalar quantizer satisfies

$H$ 9

The main theorem states that the full vector quantizer is unbiased,

$D$ 0

and achieves

$D$ 1

uniformly over all dimensions and all unit vectors as $D$ 2. The same paper also proves an inner-product error guarantee,

$D$ 3

which is directly relevant to similarity search, federated aggregation, and KV-style inner-product workloads (Feng et al., 13 May 2026).

A hardware-oriented scalar extreme appears in Fast-TurboQuant. There the dense Gaussian rotation of TurboQuant is replaced by the multiplier-free FWHT pipeline

$D$ 4

followed in the experiments by 1-bit sign quantization. On DBpedia OpenAI-3 Large embeddings, the reported sequential execution times are $D$ 5 ms for TurboQuant and $D$ 6 ms for Fast-TurboQuant, corresponding to a $D$ 7 algorithmic speedup. The same experiment reports MSE $D$ 8 versus $D$ 9 and Recall@10 $y = HDx,$ 0 versus $y = HDx,$ 1, with the paper explicitly noting that part of the quality gain is due to zero-padding from $y = HDx,$ 2 to $y = HDx,$ 3 and the associated dimension expansion (Pereira et al., 19 Jun 2026).

4. Block, vector, and lattice quantization in large-model PTQ

Once the transform has reduced coordinate concentration, scalar quantization is no longer the only natural choice. QuIP# exploits the fact that incoherence-processed weights become “roughly ball-shaped sub-Gaussian” and therefore better matched to vector quantization than to independent scalar rounding. Its main contribution is to pair randomized Hadamard incoherence processing with BlockLDLQ and hardware-efficient $y = HDx,$ 4-based lattice codebooks, achieving state-of-the-art weight-only PTQ at $y = HDx,$ 5 bits per weight (Tseng et al., 2024). In this setting, the Hadamard stage is not an isolated heuristic; it is the mechanism that makes a spherical 8-dimensional codebook geometrically sensible.

HyperQuant extends the same logic into a more explicit rate–distortion pipeline. It combines four ingredients: per-tile RHT, low-dimensional lattice quantization ( $y = HDx,$ 6, $y = HDx,$ 7, $y = HDx,$ 8, or $y = HDx,$ 9), lossless bit stripping, and near-entropy-optimal Rice coding, with additional bias-correction methods for the KV cache (Domb et al., 22 Jun 2026). The paper reports that HyperQuant outperforms HIGGS at every operating point from $H$ 0 to $H$ 1 bits per scalar on weights, beats both TurboQuant and OCTOPUS on KV quantization down to $H$ 2 bps, and at $H$ 3 bps compresses the linear weights $H$ 4 and the KV cache $H$ 5 at near-lossless quality. An especially notable systems observation is that, on the post-RHT lattice output, int8 outperforms fp8 because the transformed distribution is already light-tailed and nearly bounded, so logarithmic fp8 dynamic range is no longer advantageous (Domb et al., 22 Jun 2026).

HARP takes a different position in the design space. It treats fixed RHT as a strong baseline for extreme PTQ but replaces it by a learnable structured two-sided orthogonal processor that is initialized exactly to the corresponding RHT up to a fixed permutation convention. The adaptive processor preserves orthogonality, supports non-power-of-two dimensions through Mixed-Radix schedules, and improves perplexity and zero-shot accuracy over fixed RHT across $H$ 6- to $H$ 7-bit settings, while preserving high deployment efficiency; one reported comparison gives $H$ 8 tok/s for HARP versus $H$ 9 tok/s for FP16 (Zagitov et al., 28 May 2026). In effect, HARP generalizes randomized Hadamard preprocessing from a fixed incoherence processor into an adaptive, layer-aware family.

5. Deterministic Hadamard-preconditioned variants

The phrase “randomized Hadamard-based quantization” is often used too broadly. A substantial fraction of recent Hadamard-centered quantizers are not randomized in the strict algorithmic sense at all; they use fixed Walsh–Hadamard mixing, sometimes with normalization, centering, or a fixed sign mask, but no random sign resampling, no random permutation, and no subsampling.

Method	Transform	Randomization status
PolarQuant (Vicentino, 30 Mar 2026)	$D=\operatorname{diag}(\pm 1)$ 0 after block normalization	Deterministic; no $D=\operatorname{diag}(\pm 1)$ 1, $D=\operatorname{diag}(\pm 1)$ 2, or subsampling
HadaNorm (Federici et al., 11 Jun 2025)	$D=\operatorname{diag}(\pm 1)$ 3 before activation quantization	Deterministic Hadamard with dynamic centering and static scaling
KVLinC (Saxena et al., 6 Oct 2025)	Value-side post-multiplication $D=\operatorname{diag}(\pm 1)$ 4; keys kept channel-wise	Deterministic; fixed Hadamard, no randomized transform
QAM-W (Sharma et al., 25 May 2026)	$D=\operatorname{diag}(\pm 1)$ 5, block-Hadamard with fixed-seed sign mask	Fixed sign mask from a 64-bit seed; not resampled

This distinction is substantive rather than terminological. PolarQuant explicitly states that “The Hadamard matrix is deterministic and self-inverse,” and its relation to classical randomized Hadamard transforms is only conceptual (Vicentino, 30 Mar 2026). HadaNorm likewise inserts a fixed Hadamard transform only after mean-centering and channel normalization, arguing that Hadamard mixing alone is weakened when channels have mismatched means and scales (Federici et al., 11 Jun 2025). KVLinC uses fixed post-multiplication Hadamard rotation on the value cache only, concluding that rotating keys is not robust at $D=\operatorname{diag}(\pm 1)$ 6 bits and that the best design is

$D=\operatorname{diag}(\pm 1)$ 7

with separate linear correction for key-induced attention errors (Saxena et al., 6 Oct 2025). QAM-W adopts a block-Hadamard transform with a deterministic sign mask, then performs joint 2D Lloyd coding of adjacent rotated coordinates; its activation-aware variant at $D=\operatorname{diag}(\pm 1)$ 8 bpw stays within $D=\operatorname{diag}(\pm 1)$ 9 of BF16 WikiText-2 perplexity on every evaluated model, whereas at strict $R_k = H D_k H D_{k-1}\cdots H D_1,$ 0 bpw the rotated-codebook frontier method QTIP remains superior (Sharma et al., 25 May 2026).

The broader implication is that Hadamard mixing has become a generic distribution-shaping primitive. Some papers rely on explicit randomization and worst-case guarantees; others rely on data geometry, normalization, or calibration statistics and use Hadamard structure only as a cheap orthogonal mixer.

6. Performance regimes, applications, and limits

In LLM weight PTQ, moderate-bit Hadamard preprocessing can be nearly lossless. PolarQuant’s ablation on Qwen3.5-9B reports FP16 perplexity $R_k = H D_k H D_{k-1}\cdots H D_1,$ 1, absmax Q5 $R_k = H D_k H D_{k-1}\cdots H D_1,$ 2, Hadamard rotation only $R_k = H D_k H D_{k-1}\cdots H D_1,$ 3, Lloyd–Max centroids only $R_k = H D_k H D_{k-1}\cdots H D_1,$ 4, and full PolarQuant Q5 $R_k = H D_k H D_{k-1}\cdots H D_1,$ 5. From $R_k = H D_k H D_{k-1}\cdots H D_1,$ 6 to $R_k = H D_k H D_{k-1}\cdots H D_1,$ 7, about $R_k = H D_k H D_{k-1}\cdots H D_1,$ 8 of the gain is attributed to Hadamard rotation alone. The same paper also reports that PolarQuant Q5 dequantized and re-quantized by torchao INT4 yields perplexity $R_k = H D_k H D_{k-1}\cdots H D_1,$ 9 versus $W$ 0 for direct absmax INT4, at $W$ 1 tok/s and $W$ 2 GB VRAM (Vicentino, 30 Mar 2026). This is strong evidence that orthogonal mixing can function not only as a direct quantizer front end but also as a denoising or distribution-regularizing preprocessing stage for downstream native INT4 backends.

In microscaling FP4, however, fixed randomized Hadamard mixing is not always the end of the story. DuQuant++ studies MXFP4, where tensors are partitioned into 32-element groups that share a block scale, making within-group outliers the dominant error source. The paper identifies block-wise randomized Hadamard rotation as a strong baseline, but shows that an outlier-aware fine-grained rotation aligned with the MXFP4 group size $W$ 3 is better matched to the objective. On LLaMA3-8B, the reported WikiText2 perplexities are $W$ 4 for MR-GPTQ and $W$ 5 for DuQuant++* (Lin et al., 20 Apr 2026). The result suggests that randomized Hadamard is effective as generic smoothing, but data-aware rotation can dominate when the quantizer is highly sensitive to specific within-block peaks.

KV-cache compression exposes another boundary. KVLinC shows that at $W$ 6-bit precision, blanket Hadamard rotation of both keys and values can be actively harmful for keys because key errors perturb the attention logits themselves. The paper reports that QuaRot perplexity reaches $W$ 7 on Qwen2.5-1.5B and $W$ 8 on Qwen3-1.7B under 2-bit KV-cache quantization, while KVLinC remains stable by combining value-side Hadamard rotation with linear correction adapters for key errors (Saxena et al., 6 Oct 2025). This suggests that Hadamard equalization alone does not solve attention-logit sensitivity in the extreme low-bit KV regime.

The main theoretical limitations are now comparatively well delineated. One RHT is not worst-case sufficient for coordinatewise Gaussianity; two RHTs recover URR-style scalar-quantization behavior with $W$ 9 error; vector quantization requires three RHTs unless an $O(d\log d)$ 00 runtime moment check certifies sufficient input flatness (Ben-Basat et al., 7 May 2026). Practical limitations remain architecture- and backend-dependent: power-of-two dimensions may force padding, adaptive alternatives such as HARP introduce calibration cost, and fixed randomized mixing can be suboptimal when the real objective is not generic incoherence but quantizer-aligned block geometry or attention-preserving bias control (Zagitov et al., 28 May 2026, Lin et al., 20 Apr 2026, Saxena et al., 6 Oct 2025).

Taken together, the literature now supports a fairly precise taxonomy. Randomized Hadamard transforms are the mathematically controlled fast substitute for dense random rotations; deterministic Hadamard-preconditioned methods are their systems-oriented relatives; and adaptive or outlier-aware rotations are increasingly preferred when backend structure, calibration data, or tensor-specific error models make generic random mixing too blunt an instrument.