---
title: NVFP4 Low-Precision Pretraining
url: https://www.emergentmind.com/topics/nvfp4-low-precision-pretraining
type: topic
---

# NVFP4 Low-Precision Pretraining

NVFP4 Low-Precision Pretraining is a hardware-accelerated, block-microscaled 4-bit floating-point (FP4) training paradigm for large Transformer and Mixture-of-Experts models, designed to maximize training throughput and memory efficiency while minimizing loss and accuracy degradation relative to standard 16-bit floating-point workflows. NVFP4 builds on per-block E2M1 quantization with FP8-scale factors, uses multi-pronged suppression of quantization artifacts (including stochastic rounding, outlier management, and oscillation control), is supported natively on NVIDIA Blackwell GPUs, and has been validated in multi-trillion-token pretraining runs up to the 120B parameter scale [2509.25149][2510.27527][2604.12374][2601.22813][2505.19115].

## 1. NVFP4 Format and Quantization Principles

NVFP4 is a mixed-precision, block-microscaled FP4 number format. Every micro-block of 16 floating-point E2M1 values (${±0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6}$) shares a single signed block scale in E4M3 (FP8) format, plus (optionally) a global FP32 scale for large tensors [2509.25149][2505.19115][2604.12374]. This multi-stage scaling enables an effective block dynamic range of $[-6 \cdot 448, +6 \cdot 448]$ and improved resistance to outlier-induced clipping. Quantization proceeds as follows for a block $x_{1:16}$:
- Compute scale $s = Q_{\mathrm{E4M3}}(\max_j |x_j|)$.
- Each element is quantized: $\hat{x}_j = s \cdot Q_{\mathrm{E2M1}}(x_j / s)$.
- Dequantization is simply $x_j \approx \hat{x}_j$.

Hybrid workflows include two-dimensional (row/column) block scaling for weights (enabling exact forward/backward GEMM consistency), and one-dimensional (contiguous axis) block scaling for activations and gradients (to maximize hardware GEMM kernel efficiency) [2509.25149][2604.12374].

## 2. Core Algorithmic Advances for Stable Training

Low-precision training introduces unique numerical challenges: dynamic range underflow/overflow, quantized-gradient bias, and block-level representational collapse driven by outliers. NVFP4 pretraining addresses these through a suite of techniques:
- **Random Hadamard Transform (RHT)**: Applied before quantization (especially weight gradients), RHT spreads isolated outlier values across a block, reducing maximal quantization error and block kurtosis [2509.25149][2510.27527][2602.02047].
- **2D Quantization**: For weights, simultaneous row- and column-block scaling ensures consistent quantized values for the same block, in both forward and backward passes, avoiding chain-rule and transpose inconsistencies [2509.25149].
- **Stochastic Rounding (SR)**: Gradients are quantized via probabilistic selection between adjacent FP4 levels, yielding an unbiased estimator $\mathbb{E}[\hat{g}] = g$ and controlled variance [2509.25149][2505.19115][2510.27527].
- **Selective High-Precision Layers**: Final and sensitive layers (typically last 10–20%, embeddings, output projections) are retained in BF16, as their quantization error most affects downstream accuracy [2509.25149][2604.12374].
- **Double-Block Quantization, Outlier Clamping/Compensation**: Additional error suppression and representational uniformity via clamping activations to high quantiles and computing residual corrections only on the subspace of “hot” persistent outlier channels [2505.19115][2510.27527][2602.02047].

The integration of these techniques enables end-to-end NVFP4 quantization in all GEMM-heavy linear modules within large language models and hybrid MoE architectures [2604.12374][2510.27527].

## 3. Advanced Outlier and Oscillation Mitigation

Two dominant sources of accuracy degradation are (i) persistent outlier channels (driven by softmax, gating/nonlinearities, or SwiGLU) and (ii) quantization-induced weight oscillation. NVFP4 pipelines have developed specific extensions:
- **Hot-Channel Patch (HCP) & CHON Recipe**: Identify persistent hot channels (per-channel quantization residual norms), reinject residuals in a hardware-fused second-order correction during GEMM, and maintain higher-precision quantization (usually BF16) for “post-QK” and other outlier-sensitive projections [2602.02047].
- **OutControl**: Static masking of a small fraction (e.g., 10%) of channel dimensions—selected by running $\|\cdot\|_2$ norm—in higher precision, with all remaining compute in NVFP4. This is implemented in both forward and backward linear passes [2510.27527].
- **Oscillation Control (OsciReset, Q-EMA, Q-Ramping)**: Periodically detect and reset weights that oscillate at FP4 quantization thresholds, forcing the master weights to bin center and thus reducing the frequency of precision-induced output jumps. Q-EMA uses an exponential moving average of master weights for forward quantizer selection, Q-Ramping delays updates for high-oscillation weights and amplifies gradient steps to push weights off quantization midpoints [2510.27527][2502.20853].

Empirically, this unified set of mitigation techniques closes more than half the loss/accuracy gap between naive FP4 and full precision, and in large runs brings NVFP4 accuracy to within 1% of BF16 baselines [2510.27527][2602.02047].

## 4. Key Innovations and Algorithmic Recipes

Notable algorithmic contributions include:
- **Four-Over-Six (4/6) Adaptive Block Scaling**: Each block is quantized twice (scaling min/max to ±4 and ±6), selecting the scale with lowest quantization mean-squared error. This approach directly targets the elevated error on near-maximal FP4 values, avoids divergence in models where standard NVFP4 is unstable, and is highly efficient on NVIDIA Blackwell GPUs [2512.02010][2601.22813].
- **MS-EDEN (Micro-Scaling EDEN) Gradient Quantizer**: Provides unbiased, blockwise, and post-RHT quantization for gradients, with more than 2$\times$ lower MSE than stochastic rounding at 4 bits [2601.22813].
- **Progressive Learning and MoR (Mixture of Representations)**: Progressive learning decays the influence of an ancillary BF16 or FP16 branch during training, guiding the low-precision network into a good solution manifold before “dropping” the full-precision branch [1905.11781]. MoR dynamically analyzes numerical statistics per-tensor or block and assigns the lowest-precision format (NVFP4, FP8, BF16) meeting a per-block error threshold, maximizing efficiency while controlling rare high-error blocks [2512.22804].

These recipes are compatible with and often synergize with the standard NVFP4 convergence- and stability-improving measures. For example, “Quartet II” combines (4/6)-enhanced forward quantization with MS-EDEN unbiased gradient estimation and achieves state-of-the-art fully-NVFP4 LLM pretraining [2601.22813].

## 5. Large-Scale Experimental Validations

NVFP4 low-precision pretraining has been validated in models up to 120B parameters and sequence lengths up to 1M, with total training tokens exceeding 25T [2604.12374]. Experiments on 7B-, 12B-, and 13B-parameter Llama, OLMo, and Nemotron family models confirm:
- Downstream evaluation loss (validation and task accuracy) remains within $\approx$1% of FP8 or BF16 baselines [2509.25149][2604.12374][2510.27527].
- Task accuracy for representative LLM benchmarks (MMLU, GSM8K-CoT, HumanEval, HellaSwag, Winograd) is indistinguishable in most cases [2509.25149][2510.27527][2604.12374].
- State-of-the-art FQT (fully quantized training) with NVFP4 can match or exceed BF16 efficiency (~2–4$\times$ speedup, ~2$\times$ memory reduction) [2601.22813][2509.25149][2604.12374][2505.19115].
- In vision and smaller language models, Q-EMA/Q-Ramping, OutControl, and hybrid MoR further enhance loss convergence and stability, with up to 1.3$\times$ memory traffic reduction compared to FP8 [2502.20853][2512.22804][2509.25149].

## 6. Hardware and Implementation Aspects

NVFP4 is natively supported on NVIDIA Blackwell GPUs, including fused block quantization/dequantization, tensor-core GEMMs operating directly on packed FP4/E4M3 pairs, and fast stochastic rounding/PTX instructions for unbiased quantization [2604.12374][2509.25149]. Custom CUDA kernels implement RHT, hot-channel compensation, and blockwise scale selection (e.g., 4/6). The underlying software typically extends NVIDIA’s Transformer Engine, cuBLAS, or PyTorch/TensorFlow AMP for fine-grained, mixed-precision graph annotation.

Best-practice reproducibility recommendations include:
- Do not increase learning rate solely due to NVFP4 regularization; adjust only if empirical validation suggests benefit [2604.12374].
- Monitor per-layer and per-block underflows and fallback to higher precision if excessive zero gradients are observed [2604.12374][2510.27527].
- Use conservative error thresholds (e.g., 1% mean relative error for NVFP4 blocks in MoR) to preserve accuracy [2512.22804].

## 7. Impact, Limitations, and Future Directions

NVFP4-enabled pretraining enables scaling to massive model and data regimes with substantially reduced hardware cost and memory footprint [2505.19115][2604.12374][2601.22813]. Persistent limitations include:
- Block-level outlier resistance is not perfect; rare pathological blocks continue to require higher-precision fallback, and fine-grained control remains an open challenge [2602.02047][2512.22804].
- Some oscillation-mitigation strategies (e.g., Q-Ramping) imply tracking per-weight statistics, introducing minor implementation overhead [2502.20853][2510.27527].
- The 4/6 grid scaling is currently incompatible with MXFP4 (E8M0-scaled FP4) [2512.02010].

Future research is directed at (i) eliminating all remaining high-precision blocks/layers, (ii) generalizing block-adaptive quantization, (iii) combining learnable block rotations and latent exponent sharing, and (iv) extending robust NVFP4 recipes to mixture-of-experts, retrieval-augmented, and extremely long-context LLMs [2512.02010][2601.22813][2512.22804][2602.02047].

---

**Key References**: [2509.25149], [2510.27527], [2604.12374], [2601.22813], [2505.19115], [2512.02010], [2512.22804], [2502.20853], [1905.11781], [2602.02047]

Source: https://www.emergentmind.com/topics/nvfp4-low-precision-pretraining