Block-Diagonal Hadamard Rotation

• The paper introduces block-diagonal Hadamard rotations that partition a vector space and apply independent Hadamard transforms for efficient high-dimensional computations.
• Empirical results show significant quantization improvements, reducing LLM perplexity from 6.90 to 6.40 using token-wise INT4 quantization.
• Fast Walsh–Hadamard Transforms and the block-diagonal structure enable scalable O(N log n) implementations for randomized algorithms and subspace embeddings.

A block-diagonal Hadamard rotation is a structured orthogonal transformation constructed by partitioning a vector space into blocks and applying independent Hadamard (or Walsh–Hadamard) transforms to each block. Such rotations are central in modern randomized algorithms, signal processing, and quantization pipelines where they provide fast, scalable surrogates for dense random orthogonal maps, notably in high-dimensional and hardware-constrained settings. The mathematical, algorithmic, and empirical properties of block-diagonal Hadamard rotations are well-characterized in the context of dimensionality reduction, low-bit quantization for LLMs, and distributed sketching.

1. Mathematical Definition and Construction

Let $d$ be a power of two, $d=2^m$. The (unnormalized) Hadamard matrix $H\in\{-1,+1\}^{d\times d}$ is defined recursively via the Sylvester construction:

$$H_1 = (1), \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}$$

Normalizing gives $Q = d^{-1/2}H$, so that $Q$ is orthogonal: $QQ^\top = I_d$.

Block-diagonal assembly: Partition the vector space $\mathbb{R}^N$ into $k$ blocks of size $n_i$ (typically $d=2^m$0 for some $d=2^m$1). Define the block-diagonal Hadamard operator as:

$d=2^m$2

Each block $d=2^m$3 acts independently on its corresponding subvector, and the overall map is orthogonal:

$d=2^m$4

More broadly, block-diagonal structure is preserved under random sign-flips (Rademacher diagonals) and additional filtering, enabling applications in both stochastic rotation and deterministic frequency orderings (e.g., sequency-ordered Walsh blocks).

2. Marginal and Global Approximation Properties

Block-diagonal (or two-block) Hadamard rotations are widely used as surrogates for Haar-uniform random rotations. Their approximation properties are sharply characterized:

• Marginal distributions: In high dimensions ($d=2^m$5), the distribution of each coordinate of a rotated unit vector converges to that of a coordinate of a Haar-uniform rotation. The Kolmogorov distance between the marginal distributions satisfies the bound:

$d=2^m$6

for an explicit constant $d=2^m$7, where $d=2^m$8 denotes the two-block Hadamard rotation and $d=2^m$9 the Haar-uniform rotation. This shows uniform convergence in all one-dimensional marginals (Zilca et al., 25 Apr 2026).

• Global joint law: In contrast, the Wasserstein-$H\in\{-1,+1\}^{d\times d}$0 distance between the full vector distributions of the two-block Hadamard and Haar-uniform rotations is sharply bounded below. For $H\in\{-1,+1\}^{d\times d}$1, the worst-case Wasserstein distance exceeds $H\in\{-1,+1\}^{d\times d}$2, and as $H\in\{-1,+1\}^{d\times d}$3, the limit is $H\in\{-1,+1\}^{d\times d}$4. This gap persists regardless of $H\in\{-1,+1\}^{d\times d}$5 (Zilca et al., 25 Apr 2026).

Aspect	Marginal (Kolmogorov)	Global (Wasserstein-1)
Error decay	$H\in\{-1,+1\}^{d\times d}$6	Constant, $H\in\{-1,+1\}^{d\times d}$7
Asymptotic behavior	$H\in\{-1,+1\}^{d\times d}$8 0	Does not vanish

While block-diagonal Hadamard rotations provide accurate approximations for one-dimensional marginals, they cannot globally match the joint distribution of a Haar rotation due to their restriction to a discrete subset (hypercube image) after intermediate transforms.

3. Algorithmic Efficiency and Implementation

Block-diagonal Hadamard rotations admit efficient implementations, critical for practical deployment in high-dimensional applications:

• Fast Walsh–Hadamard Transform (FWHT): Each $H\in\{-1,+1\}^{d\times d}$9-dimensional Hadamard block can be applied in $$H_1 = (1), \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}$$0 operations. The full block-diagonal map thus costs $$H_1 = (1), \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}$$1 computations for $$H_1 = (1), \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}$$2-dimensional input split into $$H_1 = (1), \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}$$3 blocks.
• Randomization: Random sign-flip diagonals can be integrated before each block for additional mixing with $$H_1 = (1), \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}$$4 random bits.
• Memory and deployment: The orthogonality and block-diagonal structure enable in-place computation and parallelization, with only $$H_1 = (1), \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}$$5 storage required per block for $$H_1 = (1), \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}$$6-by-$$H_1 = (1), \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}$$7 Hadamard matrices (Vicentino, 30 Mar 2026, Jia et al., 21 Apr 2026).

This computational profile far outpaces dense Haar rotations, which require $$H_1 = (1), \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}$$8 time and storage, and supports fused kernels in token-wise quantization pipelines for LLMs.

4. Applications in Quantization and Compression

Block-diagonal Hadamard rotations are a cornerstone of state-of-the-art quantization schemes for neural network compression:

• PolarQuant: The block-diagonal Hadamard operation "Gaussianizes" weight vectors, transforming block-normalized weights into approximately independent normal random variables prior to quantization. Ablation demonstrates that this rotation accounts for 98% of quantization improvement, reducing perplexity in LLMs from 6.90 (absmax Q5, no rotation) to 6.40, nearly reaching full-precision baselines, and enabling lossless integration with downstream INT4 quantizers (Vicentino, 30 Mar 2026).
• GSR (Grouped Sequency-Arranged Rotation): Block-diagonal Walsh–Hadamard rotations with sequency ordering further improve low-bit quantization, isolating outlier impacts and clustering similar frequencies. GSR achieves comparable performance to learned rotations for 2-bit PTQ without retraining, providing large gains in perplexity and reasoning benchmarks on LLaMA-2-7B (Choi et al., 2 May 2025).
• Token-wise INT4 Quantization: In system-aware pipelines for KV-cache compression, block-diagonal Hadamard rotation with small block sizes (e.g., $$H_1 = (1), \quad H_{2n} = \begin{pmatrix} H_n & H_n \ H_n & -H_n \end{pmatrix}$$9) embedded within fused GPU kernels eliminates nearly all accuracy drop associated with naive INT4, while matching the throughput of non-rotated implementations (Jia et al., 21 Apr 2026).

The key mechanism in quantization contexts is dynamic-range smoothing: the rotation redistributes local outliers, shrinking the per-coordinate dynamic range, compressing the requisite scale, and minimizing quantization error.

5. Extensions: Learnable Block-Orthogonal Processors

Recent work generalizes block-diagonal Hadamard rotations by introducing learnable block-orthogonal processors:

• HARP (Hadamard-Preconditioned Adaptive Rotation Processor): Each orthogonal mixer is constructed as a product of sparse, block-diagonal stages, each block preconditioned by a fixed Hadamard but modulated by a learnable rotation (Givens or Cayley). This structure permits adaptation to layer-specific statistics and non-power-of-two dimensions via Mixed-Radix scheduling. HARP is initialized to exactly recover the randomized Hadamard transform and is tuned using calibration data to minimize quantization proxy losses and block-diagonalization regularizers (Zagitov et al., 28 May 2026).
• Empirical improvements: HARP consistently improves perplexity and zero-shot accuracy in 2–4 bit quantization regimes compared to fixed Hadamard transforms while preserving FWHt-level inference throughput.

A plausible implication is that learnable block-diagonal Hadamard-style architectures will supersede fixed-structure rotations in applications where statistical adaptivity confers tangible gains within hardware and memory budgets.

6. Theoretical Guarantees and Distributed Embedding

Block-diagonal Hadamard rotations are central to distributed oblivious subspace embeddings and randomized numerical linear algebra:

• Block SRHT (Subsampled Randomized Hadamard Transform): In distributed settings, sketches constructed from blockwise SRHTs provide nearly optimal Johnson–Lindenstrauss embedding properties. For sketch dimension $Q = d^{-1/2}H$0, block SRHT matches the distortion of classical SRHT or Gaussian sketches, while enabling efficient parallelization and communication reduction (Balabanov et al., 2022).
• OSE property: The block-diagonal structure admits local $Q = d^{-1/2}H$1 application per node, with only a single global all-reduce operation, offering superior speed and scalability to both dense Gaussian and butterfly-structured global Hadamard approaches.

7. Practical Considerations and Limitations

Block-diagonal Hadamard rotations excel in scenarios where only one-dimensional or low-dimensional marginals determine algorithmic guarantees, such as coordinate-wise quantization and subspace projection. However, they cannot be treated as full substitutes for Haar-uniform rotations in applications sensitive to high-dimensional isotropy or higher-order geometric properties; a non-vanishing Wasserstein discrepancy is intrinsic (Zilca et al., 25 Apr 2026).

Further considerations include the block size tradeoff: smaller blocks increase locality and outlier isolation (beneficial for aggressive group-wise quantization), while larger blocks approach global mixing for high-rank data at higher computational cost (Choi et al., 2 May 2025, Jia et al., 21 Apr 2026). In hardware-bound environments (e.g., LLM KV-cache), block sizes are often tuned to system widths or cache layouts for fused execution.

In summary, block-diagonal Hadamard rotations are fundamental building blocks for modern fast orthogonal transformations, underlie highly effective practical quantization strategies in high-dimensional ML models, and possess a precise theoretical characterization that shapes both their utility and limitations.