Channel-wise Hadamard Rotation in Quantized Neural Networks

• Channel-wise Hadamard rotation is a structured orthogonal transformation that independently rotates neural network channels using Hadamard matrices to mitigate quantization artifacts.
• It integrates channel-wise scaling with blockwise 2D quantization to reduce rate-distortion and achieve lower quantization errors in extreme low-bit regimes.
• The approach leverages the Fast Walsh-Hadamard Transform for efficient computation and is adopted in codecs like QAM-W, SmoothRot, KVLinC, and HARP for robust LLM inference.

Channel-wise Hadamard rotation is a structured orthogonal transformation, applied independently along channels or rows of neural network weights or activations, to enhance robustness in extreme low-bit quantization. Leveraging the mathematical properties of the Hadamard matrix, this rotation is combined with channel-wise scaling and subsequent blockwise or 2D quantization, enabling more efficient inference, lower rate-distortion, and suppression of quantization artifacts such as outliers and mode collapse. Recent quantization codecs and PTQ (post-training quantization) algorithms—QAM-W, SmoothRot, KVLinC, and HARP—systematically utilize channel-wise Hadamard rotation for both weights and activations, ranging from LLM weights to attention caches and linear projections (Sharma et al., 25 May 2026, Czakó et al., 4 Jun 2025, Saxena et al., 6 Oct 2025, Zagitov et al., 28 May 2026).

1. Mathematical Foundations of the Hadamard Matrix and Channel-wise Rotation

An $n \times n$ Hadamard matrix $H_n$ comprises only $\{\pm 1\}$ entries and satisfies $H_n H_n^\top = n I_n$, making it (after normalization) an orthogonal matrix. The matrix admits a recursive construction: $$H_1 = [1], \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}.$$ Each row is orthogonal to every other, and the normalized Hadamard, $H_n/\sqrt{n}$, is orthonormal. Channel-wise Hadamard rotation refers to applying a block-diagonal matrix with $H_b/\sqrt{b}$ blocks independently to each channel (row) or block of a larger tensor. In the QAM-W codec, each weight row $w \in \mathbb{R}^{d_{\rm in}}$ is $L_2$-normalized, partitioned into blocks of size $b$ (largest power-of-two divisor $H_n$0), and each block is mapped as $H_n$1 (Sharma et al., 25 May 2026). This operation is efficiently implemented via the Fast Walsh-Hadamard Transform in $H_n$2.

2. Quantization Pipeline: Block-wise Rotation, Pairing, and Lloyd-Max Coding

The typical workflow leveraging channel-wise Hadamard rotation is as follows:

1. Row $H_n$3-Normalization: Each $H_n$4 transformed to $H_n$5.
2. Channel-wise Activation-aware Scaling: Columns are rescaled by $H_n$6, typically $H_n$7 for RMS $H_n$8, where $H_n$9 is the channel's activation (Sharma et al., 25 May 2026, Czakó et al., 4 Jun 2025).
3. Block Hadamard Rotation: Block-diagonal $\{\pm 1\}$0 applied per row: $\{\pm 1\}$1, with $\{\pm 1\}$2 scaled and normalized.
4. 2D Pairing: Adjacent entries in $\{\pm 1\}$3 grouped into $\{\pm 1\}$4.
5. Joint Codebook Quantization: Each 2D pair quantized using a $\{\pm 1\}$5-entry codebook $\{\pm 1\}$6, trained via Lloyd–Max on the standard 2D Gaussian (Sharma et al., 25 May 2026). Scalar quantization and polar representation are disfavored due to higher distortion.

Decoding inverts these steps: codebook lookup, inverse Hadamard rotation via $\{\pm 1\}$7, unscaling, renormalization.

A summary table, as exemplified in QAM-W (Sharma et al., 25 May 2026):

Stage	Operation	Mathematical Formulation
Norm/Scaling	$\{\pm 1\}$8	$\{\pm 1\}$9
Rotation	block Hadamard	$H_n H_n^\top = n I_n$0
2D Pairing	$H_n H_n^\top = n I_n$1	$H_n H_n^\top = n I_n$2
Quantization	per-pair Lloyd–Max	$H_n H_n^\top = n I_n$3
Storage	indices and $H_n H_n^\top = n I_n$4	$H_n H_n^\top = n I_n$5

This joint 2D quantization regime leverages the isotropic, approximately Gaussian post-rotation distribution, making codebook methods optimal in low bit-width settings.

3. Application in LLMs: Outlier Smoothing, Weight/Activation Quantization, and KV-cache Compression

Activation Outliers: 4-bit and lower quantization is challenged by rare, large outliers in activations and weights, yielding suboptimal quantizer utilization. Channel-wise scaling mitigates channelwise amplitude variance, while Hadamard rotation spreads concentrated energy, transforming channel-local outliers into many low-amplitude, quantization-friendly components (Czakó et al., 4 Jun 2025). In SmoothRot, the combined scaling–rotation transform $H_n H_n^\top = n I_n$6 achieves up to $H_n H_n^\top = n I_n$7 smaller quantization interval and closes $H_n H_n^\top = n I_n$8–$H_n H_n^\top = n I_n$9 of the accuracy gap to FP16 versus prior rigid orthogonalization methods (Czakó et al., 4 Jun 2025).

Weight Quantization: For LLM layers, QAM-W and HARP protocols perform block-diagonalized (channel-wise) rotations, scaling, and pairwise quantization, reducing rate-distortion and matching the accuracy of reference methods such as SmoothQuant W8A8 at $$H_1 = [1], \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}.$$0 fewer bits in the 5–6 bpw regime (Sharma et al., 25 May 2026). HARP further adapts the rotated basis to layer curvature and calibration distribution, superseding fixed randomized Hadamard mixing (Zagitov et al., 28 May 2026).

KV Cache Quantization: In the KVLinC protocol, channel-wise Hadamard rotation is applied post-token to value matrices, followed by aggressive (2-bit) quantization and linear correction adapters for keys (Saxena et al., 6 Oct 2025). This yields state-of-the-art perplexity and 2.5$$H_1 = [1], \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}.$$1 throughput gain on Llama and Qwen models at 2-bit KV precision.

4. Adaptive and Structured Extensions: From Fixed Hadamard to Learnable Rotations

Classic methods employ randomized diagonal sign-flips and Hadamard mixing (Randomized Hadamard Transform, RHT) for incoherence (Zagitov et al., 28 May 2026). HARP introduces a learnable, structured generalization: per-channel block-orthogonal transforms parameterized via sparse “stride” stages and mixed-radix schedules. Each stage applies a block with a base mixer (Hadamard for power-of-two blocks), modulated by a small learnable orthogonal matrix $$H_1 = [1], \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}.$$2. Initialization recovers the RHT up to a permutation. Calibration data guides adaptation, driving the rotation to simultaneously reduce quantization residual in high-curvature directions and block-diagonalize the empirical Hessian (Zagitov et al., 28 May 2026).

This adaptivity yields improved rate-distortion under layer-specific curvature and quantizer structure, as quantified by Hessian-weighted reconstruction loss: $$H_1 = [1], \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}.$$3 which, after two-sided channel-wise rotation, is minimized in the transformed basis.

Mixed-radix scheduling ensures efficient support for non-power-of-two channel dimensions, avoiding zero-padding and preserving fast transform cost.

5. Empirical Assessment and Regimes of Effectiveness

• 5–6 bpw range: Channel-wise Hadamard rotation with 2D codebook coding (in QAM-W) remains within $$H_1 = [1], \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}.$$4 of BF16 perplexity on WikiText-2 across multiple LLMs (1.1B–13B parameters), at $$H_1 = [1], \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}.$$5 fewer bits compared to SmoothQuant W8A8 (Sharma et al., 25 May 2026).
• Extreme low-bit quantization: In 2–4 bit settings, Hadamard-preconditioned and adaptive rotations (HARP) yield improved perplexity and zero-shot accuracy over fixed basis, with throughput up to 128 tokens/s (vs 61 tok/s for FP16) (Zagitov et al., 28 May 2026).
• Activation quantization: Combined channel-wise scaling and rotation reduces maximal activation range by up to $$H_1 = [1], \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}.$$6, enabling substantially lower quantization errors without added inference latency (Czakó et al., 4 Jun 2025).
• KV cache compression: Channel-wise post-Hadamard transformation on values, with per-token quantization, achieves best-in-class performance for LLMs at 2 bits (Saxena et al., 6 Oct 2025).
• Ablation and sensitivity: Channel-wise rotation plus codebook quantization consistently outperforms scalar or polar coordinate quantization, except in extreme bandwidth-limited (strict 4 bpw) regimes, where alternative rotated-codebook frontiers (QTIP) may dominate (Sharma et al., 25 May 2026).

6. Limitations, Implementation, and Ongoing Developments

Channel-wise Hadamard rotation preserves deployment efficiency due to $$H_1 = [1], \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}.$$7 operations and is efficiently fused in GPU kernels (Czakó et al., 4 Jun 2025, Sharma et al., 25 May 2026); however, further optimizations include

• Learnable post-Hadamard rotation (e.g., HARP): Requires one-time calibration and tuning per model, with associated GPU-hours (Zagitov et al., 28 May 2026).
• Diminishing gains in presence of advanced quantization backends: When combined with highly expressive quantizers (GPTQ) or after activation smoothing, the marginal utility of Hadamard rotation lessens, motivating further rotation basis optimization.
• Non-activation-aware extensions: HARP, for example, currently focuses on weight-only PTQ; activation-aware principled extensions are suggested as a future direction (Zagitov et al., 28 May 2026).

A plausible implication is that channel-wise Hadamard rotation will remain a standard baseline for orthogonalization in extreme quantization pipelines, with the field pushing towards increasingly data-adaptive, hardware-efficient, and activation-aware augmentations.

7. Comparative Table of Channel-wise Hadamard Rotation Across Protocols

Protocol	Rotation Scope	Quantization	Scaling	Bit Regime	Reported Gains
QAM-W (Sharma et al., 25 May 2026)	Block-diagonal, per-row	Joint 2D codebook	Per-channel, activation-aware	3.5–6 bpw	Within $$H_1 = [1], \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}.$$8 BF16 PPL at $$H_1 = [1], \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}.$$9 bpw
SmoothRot (Czakó et al., 4 Jun 2025)	Full, per-downproj layer	Per-channel (act+weights)	SmoothQuant-style	4 bits	$H_n/\sqrt{n}$0–$H_n/\sqrt{n}$1 FP16 gap closed
KVLinC (Saxena et al., 6 Oct 2025)	Post, per-token value	2-bit, asymmetric	—	2 bits	Best 2-bit KV PPL, $H_n/\sqrt{n}$2 throughput
HARP (Zagitov et al., 28 May 2026)	Two-sided, per-channel	PTQ backend, block	Data-adaptive	2–4 bits	$H_n/\sqrt{n}$3 speed, improved PPL/acc

• "QAM-W: Joint 2D Codebook Quantization for LLM Weights via Hadamard Rotation and Activation-Aware Scaling" (Sharma et al., 25 May 2026)
• "SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs" (Czakó et al., 4 Jun 2025)
• "KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction" (Saxena et al., 6 Oct 2025)
• "HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization" (Zagitov et al., 28 May 2026)