NVFP4 Quantization: Methods & Impact

• NVFP4 quantization is a 4-bit floating point technique that uses dual-level scaling and block-wise Hadamard rotations to reduce quantization errors.
• It leverages stochastic rounding and fine-grained mixed precision to stabilize training while achieving near full-precision accuracy with significant memory savings.
• NVFP4 is optimized for modern NVIDIA architectures, delivering up to 6× speedup and energy efficiency, making it vital for efficient inference, reinforcement learning, and diffusion models.

NVFP4 quantization is a 4-bit floating-point post-training and training quantization technique, supported in recent NVIDIA GPU architectures, designed to maximize throughput and memory efficiency during both inference and training of large-scale neural networks. NVFP4 integrates advanced microscaling, dual-level scaling schemes, Hadamard-based rotation for outlier dispersion, and stochastic rounding for unbiased gradient estimation. Its distinctive approach—pairing small block sizes and more expressive scaling formats—yields competitive accuracy compared to full-precision and FP8, unlocks hardware-level speedups, and supports stable, large-batch training and RL rollouts in LLMs and diffusion architectures.

1. NVFP4 Format: Structure and Scaling

NVFP4 represents tensors by partitioning data into blocks of 16 elements, each accompanied by a local scale factor. Unlike the MXFP4 format (which employs larger block sizes and power-of-two scaling in UE8M0), NVFP4 uses E4M3 format (1 sign bit, 4 exponent bits, 3 mantissa bits) for the block-level scale. Quantized values are stored as 4-bit FP4 codes. A global per-tensor scale (FP32) is multilplied prior to local scaling, leading to a “two-level scaling” scheme: $W_{dec} = S_{FP32} \cdot (S_{E4M3} \odot W_{FP4})$ where $W_{FP4}$ is the blockwise quantized weight, $S_{E4M3}$ is the block scale vector, and $S_{FP32}$ the global scale.

This scheme increases local dynamic range adaptation and reduces quantization errors for both typical and outlier values (NVIDIA et al., 29 Sep 2025). NVFP4 is designed specifically to handle blockwise quantization where the quantizer block size is matched to the hardware kernel, ensuring optimal utilization on NVIDIA Blackwell, Hopper, and similar architectures.

2. Quantization Workflow and Error Mitigation

NVFP4 leverages several key techniques for minimizing quantization-induced reconstruction error:

• Random Hadamard Transform (RHT): Applied to blocks prior to quantization, it redistributes outlier values (large activations/weights) into an approximately Gaussian distribution, reducing their dominance in scale calculation. For matrix multiplication, the Hadamard matrix $H$ satisfies $H H^{\mathsf{T}} = I$ and preserves the result, $C = (A H)\cdot(H^{\mathsf{T}} B) = AB$, so rotation is data-preserving and compatible with existing pipeline (NVIDIA et al., 29 Sep 2025, Egiazarian et al., 27 Sep 2025).
• Two-Dimensional Block Quantization: Weights are quantized in $16 \times 16$ tiles, aligning block scaling for both forward and backward passes (i.e., under tensor transposition). Activations and gradients use $1 \times 16$ scaling axes, since their quantization-induced errors are less damaging (NVIDIA et al., 29 Sep 2025).
• Stochastic Rounding: Gradients and activations use probabilistic rounding, where a value $x$ in $[a, b]$ is rounded to $a$ or $b$ with probabilities proportional to $(b-x)$ and $(x-a)$. This reduces systematic rounding bias and stabilizes stochastic training (NVIDIA et al., 29 Sep 2025, Hu et al., 22 Sep 2025).
• Selective High-Precision Retention: Numerically sensitive layers (e.g., final blocks, sometimes first layers) remain in higher precision (BF16 or FP8). In practice, about 15–16% of the model may be kept at high precision, balancing stability and efficiency (NVIDIA et al., 29 Sep 2025).
• Fine-Grained Mixed Precision (FGMP) Assignment: In post-training quantization, impact scores are computed for blocks using sensitivity-weighted errors (diagonal Fisher information). Blocks with high impact are preserved in FP8; all others use NVFP4. This selection is governed by a global threshold (Hooper et al., 19 Apr 2025).

3. Outlier Dispersion and Small Block Size Effects

NVFP4’s block size of 16 elements makes traditional outlier mitigation ineffective, as a single outlier within the block can disproportionately scale the group and degrade quantization granularity for non-outliers. To counteract this, block-wise Hadamard rotations are fused into both weights and activations before quantization (Micro-Rotated-GPTQ, MR-GPTQ). The rotation spreads energy across all coordinates, equalizing Mean Squared Error (MSE) between per-element and outlier elements: $$\mathrm{MSE}_{top}(G) = \mathrm{MSE}(G) = \frac{1}{G}\mathbb{E}[\|\epsilon_x\|_2^2]$$ where $G$ is the group size and $\epsilon_x$ are errors in rotated domain (Egiazarian et al., 27 Sep 2025).

Empirical evaluation confirms that MR-GPTQ recovers 98–99% of FP16 accuracy in LLMs even with NVFP4 quantization, outperforming previous GPTQ or round-to-nearest methods, particularly in tasks where small group sizes exacerbate quantization error.

4. Hardware Support and Throughput

NVFP4 is natively supported on NVIDIA H100, B200, and related hardware, often achieving up to 3.6× layer-wise and 2.2× end-to-end inference speedup vs. FP16 on B200, and up to 6×/4× speedup respectively on RTX5090 GPUs. Specialized GPU kernels (QuTLASS) fuse micro-rotation and quantization, further minimizing overhead. NVFP4 described in FGMP achieves a 14% energy savings and a 30% reduction in memory footprint compared to FP8 (Hooper et al., 19 Apr 2025, Egiazarian et al., 27 Sep 2025).

5. Training Stability and Optimization

Stable training at FP4 precision is challenging due to limited dynamic range and rounding bias. NVFP4 integrates:

• Gradient-Based Microscaling Framework: Quantization gradients are explicitly calculated; for input $x_{ij}$, the derivative is: $$\frac{\partial f_{ij}}{\partial X_{ij}} = Q'(s_q X_{ij}) + \frac{\partial s}{\partial X_{ij}} \Bigg[\frac{q'(s)}{s_q} (X_{ij} Q'(s_q X_{ij}) - \frac{1}{s_q} Q(s_q X_{ij})) \Bigg]$$ highlighting the interplay of scale quantization and signal propagation (Hu et al., 22 Sep 2025).
• Global+Local Scaling: UE5M3 format for scale factors is identified as optimal for balancing precision and dynamic range, outperforming traditional E4M3 in very large models (Hu et al., 22 Sep 2025).
• Stochastic Rounding and Hadamard Transformations: Used throughout forward and backward passes to minimize quantization-induced bias and concentrate outliers (Hu et al., 22 Sep 2025).

Training of a 12B-parameter LLM with NVFP4 on 10T tokens achieves less than 1.5% loss error vs. FP8, and similar downstream performance (e.g., MMLU-pro 5-shot: 62.58% NVFP4 vs. 62.62% FP8) (NVIDIA et al., 29 Sep 2025).

6. Applications: Efficient Inference, RL, and Diffusion

NVFP4 is leveraged in multiple domains:

• LLM Inference and Pretraining: Enables memory- and throughput-constrained deployment, with minimal degradation in perplexity or downstream accuracy. FGMP quantization maintains $<$1% perplexity degradation on Llama-2-7B with up to 14% energy and 30% memory savings (Hooper et al., 19 Apr 2025).
• Reinforcement Learning (RL): QeRL integrates NVFP4 quantization with Adaptive Quantization Noise (AQN), injecting dynamic Gaussian noise during training. This noise increases policy entropy $\mathcal{H}(\pi(\cdot|q))$ and accelerates exploration. QeRL achieves 1.5× rollout speedup and matches full fine-tuning accuracy (GSM8K 90.8%, MATH 500 77.4% at 7B) (Huang et al., 13 Oct 2025).
• Diffusion Models: FP4 quantization (inc. NVFP4-style scaling) outperforms integer-based approaches at W4A6 and W4A8 (weights/activations), showing reduced reconstruction noise and preservation of fine-grained details in tasks like PixArt-$\alpha$ synthesis (Chen et al., 19 Mar 2025).

7. Limitations and Future Research

NVFP4 is not universally optimal; limited dynamic range in E4M3 scaling may hinder convergence in extremely LLMs, suggesting UE5M3 or dynamic formats may be preferable (Hu et al., 22 Sep 2025). Small group sizes render outlier handling difficult; rotation-based methods such as MR-GPTQ are essential. Deterministic rounding introduces training bias; stochastic rounding is advisable.

A plausible implication is that future NVFP4-like quantization schemes will adopt Pareto-optimal scaling, more dynamic format selection, and even more aggressive block-wise transformation strategies to further improve the efficiency-accuracy frontier.

Table: NVFP4 Quantization Components and Effects

Component	Technical Feature	Impact
Block size	16 (vs. 32 in MXFP4)	Higher local adaptation
Local scale format	E4M3 (option: UE5M3)	Wider dynamic range
RHT/Rotation	Hadamard transform (fused)	Outlier dispersion, lower MSE
Mixed precision	Selective FP8/BF16 layers	Training stability
Rounding	Stochastic (vs. deterministic)	Lower bias, better gradients
Kernel support	QuTLASS (GPU, native FP4)	Speedup and throughput

NVFP4 quantization, through microscaling, block-wise rotation, dual-level scaling, and selective high precision retention, establishes a modern framework for ultra-low precision computation in deep neural networks. Empirical evidence shows near-baseline accuracy across tasks and marked efficiency improvements, while limitations in group size and scale format highlight directions for further research and practical tuning.