---
title: Channel-wise Hadamard Rotation in Quantized Neural Networks
url: https://www.emergentmind.com/topics/channel-wise-hadamard-rotation
type: topic
---

# Channel-wise Hadamard Rotation in Quantized Neural Networks

Channel-wise Hadamard rotation is a structured orthogonal transformation, applied independently along channels or rows of neural network weights or activations, to enhance robustness in extreme low-bit quantization. Leveraging the mathematical properties of the Hadamard matrix, this rotation is combined with channel-wise scaling and subsequent blockwise or 2D quantization, enabling more efficient inference, lower rate-distortion, and suppression of quantization artifacts such as outliers and mode collapse. Recent quantization codecs and PTQ (post-training quantization) algorithms—QAM-W, SmoothRot, KVLinC, and HARP—systematically utilize channel-wise Hadamard rotation for both weights and activations, ranging from LLM weights to attention caches and linear projections [2605.26339][2506.05413][2510.05373][2605.29843].

## 1. Mathematical Foundations of the Hadamard Matrix and Channel-wise Rotation

An $n \times n$ Hadamard matrix $H_n$ comprises only $\{\pm 1\}$ entries and satisfies $H_n H_n^\top = n I_n$, making it (after normalization) an orthogonal matrix. The matrix admits a recursive construction:
\[
H_1 = [1], \quad H_{2n} = 
\begin{pmatrix}
H_n & H_n \\
H_n & -H_n
\end{pmatrix}.
\]
Each row is orthogonal to every other, and the normalized Hadamard, $H_n/\sqrt{n}$, is orthonormal. Channel-wise Hadamard rotation refers to applying a block-diagonal matrix with $H_b/\sqrt{b}$ blocks independently to each channel (row) or block of a larger tensor. In the QAM-W codec, each weight row $w \in \mathbb{R}^{d_{\rm in}}$ is $L_2$-normalized, partitioned into blocks of size $b$ (largest power-of-two divisor $\leq 1024$), and each block is mapped as $y^{(k)} = H_b \hat{w}^{(k)} / \sqrt{b}$ [2605.26339]. This operation is efficiently implemented via the Fast Walsh-Hadamard Transform in $O(n \log n)$.

## 2. Quantization Pipeline: Block-wise Rotation, Pairing, and Lloyd-Max Coding

The typical workflow leveraging channel-wise Hadamard rotation is as follows:

1. **Row $L_2$-Normalization**: Each $w$ transformed to $\hat{w} = w / \|w\|_2$.
2. **Channel-wise Activation-aware Scaling**: Columns are rescaled by $\alpha_j$, typically $\alpha_j = r_j^\alpha$ for RMS $r_j = \sqrt{\mathbb{E}[x_j^2]}$, where $x_j$ is the channel's activation [2605.26339][2506.05413].
3. **Block Hadamard Rotation**: Block-diagonal $F = \oplus H_b / \sqrt{b}$ applied per row: $y = F \tilde{w}$, with $\tilde{w}$ scaled and normalized.
4. **2D Pairing**: Adjacent entries in $y$ grouped into $z_k = (y_{2k-1}, y_{2k})$.
5. **Joint Codebook Quantization**: Each 2D pair quantized using a $2^B$-entry codebook $\mathcal{C} \subset \mathbb{R}^2$, trained via Lloyd–Max on the standard 2D Gaussian [2605.26339]. Scalar quantization and polar representation are disfavored due to higher distortion.

Decoding inverts these steps: codebook lookup, inverse Hadamard rotation via $F^\top$, unscaling, renormalization.

A summary table, as exemplified in QAM-W [2605.26339]:

| Stage             | Operation                           | Mathematical Formulation                |
|-------------------|-------------------------------------|-----------------------------------------|
| Norm/Scaling      | $w \to \tilde{w}$                  | $\tilde{w} = S \hat{w}$                |
| Rotation          | block Hadamard                     | $y = F \tilde{w}$                       |
| 2D Pairing        | $y \to \{z_k\}$                    | $z_k = (y_{2k-1}, y_{2k})$             |
| Quantization      | per-pair Lloyd–Max                 | $q_k = Q(z_k)$                         |
| Storage           | indices and $r_{\rm row}$          | $\{q_k\}, r_{\rm row}$                 |

This joint 2D quantization regime leverages the isotropic, approximately Gaussian post-rotation distribution, making codebook methods optimal in low bit-width settings.

## 3. Application in LLMs: Outlier Smoothing, Weight/Activation Quantization, and KV-cache Compression

**Activation Outliers**: 4-bit and lower quantization is challenged by rare, large outliers in activations and weights, yielding suboptimal quantizer utilization. Channel-wise scaling mitigates channelwise amplitude variance, while Hadamard rotation spreads concentrated energy, transforming channel-local outliers into many low-amplitude, quantization-friendly components [2506.05413]. In SmoothRot, the combined scaling–rotation transform $x' = H (s \odot x)$ achieves up to $30\times$ smaller quantization interval and closes $10$–$30\%$ of the accuracy gap to FP16 versus prior rigid orthogonalization methods [2506.05413].

**Weight Quantization**: For LLM layers, QAM-W and HARP protocols perform block-diagonalized (channel-wise) rotations, scaling, and pairwise quantization, reducing rate-distortion and matching the accuracy of reference methods such as SmoothQuant W8A8 at $32\%$ fewer bits in the 5–6 bpw regime [2605.26339]. HARP further adapts the rotated basis to layer curvature and calibration distribution, superseding fixed randomized Hadamard mixing [2605.29843].

**KV Cache Quantization**: In the KVLinC protocol, channel-wise Hadamard rotation is applied post-token to value matrices, followed by aggressive (2-bit) quantization and linear correction adapters for keys [2510.05373]. This yields state-of-the-art perplexity and 2.5$\times$ throughput gain on Llama and Qwen models at 2-bit KV precision.

## 4. Adaptive and Structured Extensions: From Fixed Hadamard to Learnable Rotations

Classic methods employ randomized diagonal sign-flips and Hadamard mixing (Randomized Hadamard Transform, RHT) for incoherence [2605.29843]. HARP introduces a learnable, structured generalization: per-channel block-orthogonal transforms parameterized via sparse “stride” stages and mixed-radix schedules. Each stage applies a block with a base mixer (Hadamard for power-of-two blocks), modulated by a small learnable orthogonal matrix $Q_{t,c}$. Initialization recovers the RHT up to a permutation. Calibration data guides adaptation, driving the rotation to simultaneously reduce quantization residual in high-curvature directions and block-diagonalize the empirical Hessian [2605.29843].

This adaptivity yields improved rate-distortion under layer-specific curvature and quantizer structure, as quantified by Hessian-weighted reconstruction loss:
\[
L(W,\hat{W}) = \mathrm{Tr}\left( (W-\hat{W}) H (W-\hat{W})^\top \right)
\]
which, after two-sided channel-wise rotation, is minimized in the transformed basis.

Mixed-radix scheduling ensures efficient support for non-power-of-two channel dimensions, avoiding zero-padding and preserving fast transform cost.

## 5. Empirical Assessment and Regimes of Effectiveness

- **5–6 bpw range**: Channel-wise Hadamard rotation with 2D codebook coding (in QAM-W) remains within $\pm0.4\%$ of BF16 perplexity on WikiText-2 across multiple LLMs (1.1B–13B parameters), at $32\%$ fewer bits compared to SmoothQuant W8A8 [2605.26339].
- **Extreme low-bit quantization**: In 2–4 bit settings, Hadamard-preconditioned and adaptive rotations (HARP) yield improved perplexity and zero-shot accuracy over fixed basis, with throughput up to 128 tokens/s (vs 61 tok/s for FP16) [2605.29843].
- **Activation quantization**: Combined channel-wise scaling and rotation reduces maximal activation range by up to $10^4$, enabling substantially lower quantization errors without added inference latency [2506.05413].
- **KV cache compression**: Channel-wise post-Hadamard transformation on values, with per-token quantization, achieves best-in-class performance for LLMs at 2 bits [2510.05373].
- **Ablation and sensitivity**: Channel-wise rotation plus codebook quantization consistently outperforms scalar or polar coordinate quantization, except in extreme bandwidth-limited (strict 4 bpw) regimes, where alternative rotated-codebook frontiers (QTIP) may dominate [2605.26339].

## 6. Limitations, Implementation, and Ongoing Developments

Channel-wise Hadamard rotation preserves deployment efficiency due to $O(n\log n)$ operations and is efficiently fused in GPU kernels [2506.05413][2605.26339]; however, further optimizations include
- **Learnable post-Hadamard rotation (e.g., HARP)**: Requires one-time calibration and tuning per model, with associated GPU-hours [2605.29843].
- **Diminishing gains in presence of advanced quantization backends**: When combined with highly expressive quantizers (GPTQ) or after activation smoothing, the marginal utility of Hadamard rotation lessens, motivating further rotation basis optimization.
- **Non-activation-aware extensions**: HARP, for example, currently focuses on weight-only PTQ; activation-aware principled extensions are suggested as a future direction [2605.29843].

A plausible implication is that channel-wise Hadamard rotation will remain a standard baseline for orthogonalization in extreme quantization pipelines, with the field pushing towards increasingly data-adaptive, hardware-efficient, and activation-aware augmentations.

## 7. Comparative Table of Channel-wise Hadamard Rotation Across Protocols

| Protocol     | Rotation Scope     | Quantization         | Scaling       | Bit Regime      | Reported Gains                      |
|--------------|--------------------|----------------------|--------------|-----------------|-------------------------------------|
| QAM-W [2605.26339]     | Block-diagonal, per-row | Joint 2D codebook | Per-channel, activation-aware | 3.5–6 bpw        | Within $\pm0.4\%$ BF16 PPL at $5.5$ bpw |
| SmoothRot [2506.05413]| Full, per-downproj layer| Per-channel (act+weights) | SmoothQuant-style | 4 bits          | $10$–$30\%$ FP16 gap closed         |
| KVLinC [2510.05373]     | Post, per-token value  | 2-bit, asymmetric | —              | 2 bits           | Best 2-bit KV PPL, $2.5\times$ throughput |
| HARP [2605.29843]      | Two-sided, per-channel  | PTQ backend, block | Data-adaptive   | 2–4 bits          | $2\times$ speed, improved PPL/acc    |

## References

- "QAM-W: Joint 2D Codebook Quantization for LLM Weights via Hadamard Rotation and Activation-Aware Scaling" [2605.26339]
- "SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs" [2506.05413]
- "KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction" [2510.05373]
- "HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization" [2605.29843]

Source: https://www.emergentmind.com/topics/channel-wise-hadamard-rotation