DuQuant++: Efficient 4-bit Quantization

• The paper introduces DuQuant++, a PTQ method that optimizes outlier suppression via a fine-grained, block-diagonal orthogonal rotation aligned with MXFP4 groups to enhance quantization accuracy.
• It reduces computational cost by applying a single 32×32 rotation per group, halving the rotation overhead compared to dual-rotation methods.
• Empirical evaluations on LLaMA-3 variants demonstrate state-of-the-art performance with lower perplexity and improved QA accuracy among 4-bit quantization approaches.

DuQuant++ is a post-training quantization (PTQ) method that enhances the efficiency and accuracy of neural network inference under the MXFP4 microscaling format, which is natively supported by NVIDIA Blackwell Tensor Cores for LLMs. By adapting outlier-aware, fine-grained rotation mechanisms to precisely match the microscaling group structure of MXFP4, DuQuant++ delivers state-of-the-art accuracy and a substantial reduction in online computational overhead for 4-bit weight/activation quantization (W4A4) (Lin et al., 20 Apr 2026).

1. Background: Microscaling Quantization and Outlier-Induced Error

NVIDIA’s MXFP4 (“E8M0”) format partitions weight or activation tensors into contiguous groups of $B = 32$ values (microscaling groups), with each group sharing a single 8-bit exponent (scale) and a zero-bit mantissa. Given $X\in\mathbb{R}^{m\times n}$, the tensor is divided into blocks $\{X_j\}_{j=1}^N$ ($N = mn/32$), and each element $x_i \in X_j$ is quantized as

$$Q(x_i) = \operatorname{nearest}\left(\left\lfloor\frac{x_i}{s_j}\right\rceil, q_{\min}, q_{\max}\right),$$

where $$s_j = 2^{\left\lfloor\log_2(\max|X_j|)\right\rfloor - b}$$ is the block scale, and $[q_{\min}, q_{\max}]$ span the FP4 dynamic range.

The microscaling mechanism achieves high throughput and hardware efficiency but is highly susceptible to group-local outliers—single elements with unusually large magnitude—which inflate $s_j$ and compress the quantization range for all non-outlier elements in the block. This effect is especially pronounced in LLM activations and weights, which routinely display both “normal” (broadly distributed) and “massive” (rare, extreme) outliers. Conventional rotation-based quantization stabilization methods (randomized Hadamard, learnable data-agnostic rotations) fail to target channels with concentrated outlier activity (Lin et al., 20 Apr 2026).

2. DuQuant++ Methodology: Outlier-Aware Rotation Aligned with Microscaling Groups

DuQuant++ extends the core technique of DuQuant, optimizing outlier suppression by aligning the rotation block size directly with the MXFP4 microscaling group ($B = 32$). The method proceeds as follows:

1. Affine Rebalancing (SmoothQuant-style): Preceding rotation, an affine transformation rebalances the contributions of weights and activations using

$X\in\mathbb{R}^{m\times n}$0

where $X\in\mathbb{R}^{m\times n}$1 trades off outlier handling between activations and weights.

2. Block-Diagonal Orthogonal Rotation: A single block-diagonal orthogonal rotation $X\in\mathbb{R}^{m\times n}$2, with $X\in\mathbb{R}^{m\times n}$3, is applied. Each rotation block is specifically optimized (greedily over Givens rotations) to minimize MXFP4 quantization error:

$X\in\mathbb{R}^{m\times n}$4

where $X\in\mathbb{R}^{m\times n}$5 is the $X\in\mathbb{R}^{m\times n}$6-th column and $X\in\mathbb{R}^{m\times n}$7 is its quantized reconstruction.

A key distinction from the original DuQuant is the removal of the second rotation and permutation: since each MXFP4 group now independently scales, the need to redistribute block-level variance using permutations is eliminated.

The composite quantized linear layer thus takes the form

$X\in\mathbb{R}^{m\times n}$8

with $X\in\mathbb{R}^{m\times n}$9 and $\{X_j\}_{j=1}^N$0 fully quantized to MXFP4, and only a single inference-time matrix multiply $\{X_j\}_{j=1}^N$1 required (as $\{X_j\}_{j=1}^N$2 is absorbed offline into the weights).

3. Computational Efficiency and Rotation Cost Analysis

DuQuant++ substantially reduces the online cost of quantization stabilization in comparison to previous methods:

• Original DuQuant (integer quantization): Two $\{X_j\}_{j=1}^N$3 rotations plus a permutation per block: $\{X_j\}_{j=1}^N$4.
• DuQuant++ (MXFP4): Single $\{X_j\}_{j=1}^N$5 rotation per block: $\{X_j\}_{j=1}^N$6.

Given $\{X_j\}_{j=1}^N$7 FLOPs for a $\{X_j\}_{j=1}^N$8 multiply, DuQuant++ approximately halves the constant factor in rotation cost and entirely eliminates the permutation overhead. Empirical analysis shows a 2× reduction in total rotation overhead for GPU kernels (Lin et al., 20 Apr 2026).

Efficient implementation is enabled by reusing the same $\{X_j\}_{j=1}^N$9 rotation $N = mn/32$0 across all activation groups, allowing the matrix to reside in shared memory ($N = mn/32$1 KiB) and minimizing branching or kernel launch costs.

4. Implementation Details and Algorithmic Procedure

The method is deployed atop NVIDIA Blackwell Tensor Cores, which offer native support for MXFP4 GEMM operations and scaling. The quantization pipeline is:

• For each linear layer, compute outlier-aware $N = mn/32$2 using a greedy Givens rotation algorithm:

1. Initialize $N = mn/32$3.
2. For $N = mn/32$4 to $N = mn/32$5 (e.g., $N = mn/32$6):
   • Find index pair $N = mn/32$7 maximizing $N = mn/32$8 over all blocks/indices.
   • Apply a Givens rotation $N = mn/32$9 in plane $x_i \in X_j$0 to reduce peak magnitude.
   • Update $x_i \in X_j$1.
3. Return $x_i \in X_j$2.

• Fuse $x_i \in X_j$3 offline into quantized weights, so only activations undergo online transformation.
• All $x_i \in X_j$4 rotation matrices are reused per group to economize on-memory and computation.

The design ensures that no additional CUDA kernel calls are needed at runtime for weights, and online rotation is localized to a single efficient kernel invocation (Lin et al., 20 Apr 2026).

5. Experimental Evaluation and Comparative Performance

DuQuant++ was benchmarked on four LLaMA-3 variants, both pre-trained (8B, 3B) and instruction-tuned (8B, 8B.1), under MXFP4 W4A4 quantization. The evaluation employed 128 WikiText2 calibration sequences and assessed both language modeling perplexity (WikiText2, C4) and aggregate QA accuracy on seven tasks. In all tests, DuQuant++ (including its GPTQ-enhanced form, "DuQuant++*") achieved state-of-the-art performance among MXFP4 baselines:

Method	WikiText2 ↓	C4 ↓	Avg QA ↑
FP16	6.14	9.46	69.1
QuaRot	9.46	15.06	62.9
MR-GPTQ	7.29	11.41	66.1
DuQuant++	7.07	11.14	66.5
DuQuant++*	6.88	11.06	67.1

Method	WikiText2 ↓	C4 ↓	Avg QA ↑
FP16	8.31	13.03	67.8
FlatQuant	9.25	15.27	64.9
MR-GPTQ	9.25	14.62	65.2
DuQuant++	8.91	14.30	65.9
DuQuant++*	8.75	14.12	65.9

Figure 1 in the source publication demonstrates that DuQuant++ yields the lowest per-group $x_i \in X_j$5 quantization error $x_i \in X_j$6 across all transformer layers at key positions, outperforming both original activations and Hadamard-rotated activations. Histogram analysis confirms that $x_i \in X_j$7 consistently redistributes outlier mass and smooths long-tailed distributions.

6. Technical Significance, Limitations, and Future Directions

The alignment of rotation blocks ($x_i \in X_j$8) with MXFP4 groups is critical for both error mitigation and computational efficiency. This design eliminates inter-block leakage and obviates global permutations. The single outlier-aware rotation suffices under MXFP4’s independent per-block scaling, differing fundamentally from the dual-rotation pipeline required for integer quantization with a global scale.

A core sensitivity of DuQuant++ is its strict alignment with hardware microscaling group size; non-matching block sizes would undermine both hardware compatibility and error control. Extension of the approach to other formats such as NVFP4 ($x_i \in X_j$9, E4M3) is plausible, as is integration into hybrid or future microscaling variants.

Current limitations include the exploration of per-block, data-dependent rotations—which would introduce additional memory overhead—and the lack of direct quantization-aware training for rotation matrices $$Q(x_i) = \operatorname{nearest}\left(\left\lfloor\frac{x_i}{s_j}\right\rceil, q_{\min}, q_{\max}\right),$$0. Avenues for further work include jointly optimizing $$Q(x_i) = \operatorname{nearest}\left(\left\lfloor\frac{x_i}{s_j}\right\rceil, q_{\min}, q_{\max}\right),$$1 with fine-tuning and generalizing the method to novel microscaling quantization schemas (Lin et al., 20 Apr 2026).