NVFP4 Pre-training: 4-bit Scaling & Stability

• NVFP4 pre-training is a 4-bit floating-point regime that uses hierarchical microscaling and native Blackwell Tensor Core support to stabilize long-horizon training.
• Hybrid strategies—combining selective high-precision layers, 2D block scaling, Random Hadamard transforms, and stochastic rounding—achieve convergence close to full-precision baselines.
• Adaptive techniques such as Four Over Six, MS-EDEN, and outlier control further reduce quantization error and improve performance compared to traditional floating-point methods.

NVFP4 pre-training denotes training workflows in which the numerically dominant linear GEMMs of a model are executed in NVIDIA’s 4-bit microscaling floating-point format during optimization rather than only at deployment. In the current literature, the term covers both hybrid recipes that retain selected layers or operations in higher precision and fully-NVFP4 schemes for linear layers, but the shared objective is stable long-horizon training under FP4 arithmetic by exploiting Blackwell Tensor Core support, hierarchical microscaling, and the reduced cost of dense matrix multiplication relative to BF16 and FP8 (NVIDIA et al., 29 Sep 2025, Panferov et al., 30 Jan 2026).

1. Numerical representation and computational scope

NVFP4 is a block-scaled FP4 format built around an E2M1 payload, a per-block FP8 E4M3 scale, and a tensor-level FP32 scale. A standard dequantization form used in the literature is

$$\hat{\mathbf{X}} = \hat{\mathbf{X}}^{\mathrm{FP4}} \cdot \alpha^{\mathrm{FP8}} \cdot \alpha^{\mathrm{FP32}}, \qquad \hat{\mathbf{X}}^{\mathrm{FP4}} \in \mathbb{F}_{\mathrm{E2M1}},$$

with the E2M1 value set given as $\{0,\pm0.5,\pm1,\pm1.5,\pm2,\pm3,\pm4,\pm6\}$. In the LLM and video-training papers, the standard granularity is a 16-element block scale stored in FP8, together with a tensor-global FP32 scale for range extension (Chen et al., 18 May 2026, NVIDIA et al., 29 Sep 2025).

The format is treated as attractive for pre-training because Blackwell GPUs support NVFP4 natively in tensor cores and because the microscaling hierarchy preserves substantially more dynamic range than earlier FP4 schemes. The original large-scale LLM study emphasizes three differences from MXFP4: NVFP4 uses 16-element blocks rather than 32-element blocks, stores block scales in E4M3 rather than UE8M0, and uses two-level scaling with both per-tensor FP32 and per-block FP8 factors. In an 8B-model comparison trained on 1T tokens, NVFP4 converges to better loss than MXFP4; MXFP4’s relative error is about 2.5%, NVFP4’s is about 1.5%, and MXFP4 needs about 36% more tokens to match NVFP4’s final loss (NVIDIA et al., 29 Sep 2025).

The later fully-quantized analyses sharpen this framing. One paper explicitly argues that NVFP4 is the first 4-bit floating-point format that is plausibly practical for end-to-end LLM pre-training because Blackwell hardware supports it natively and microscaling preserves dynamic range through per-block FP8 scales. Another systems paper extends the same logic beyond LLMs, presenting NVFP4 as a training-time infrastructure choice for long video generation because the proportion of GEMM-heavy compute grows with video length, making low-precision matrix multiplication increasingly valuable (Panferov et al., 30 Jan 2026, Chen et al., 18 May 2026).

2. Baseline recipe for stable long-horizon LLM training

The first large public NVFP4 pre-training recipe is not a bare format substitution but a composite stabilization method. Its central ingredients are Random Hadamard transforms on weight-gradient inputs, 2D block scaling for weights, stochastic rounding on gradients, and selective high-precision layers. The study validates the recipe by training a 12-billion-parameter hybrid Mamba-Transformer on 10 trillion tokens, described as the longest publicly documented training run in 4-bit precision to date, and reports training loss and downstream task accuracies comparable to an FP8 baseline (NVIDIA et al., 29 Sep 2025).

In this baseline formulation, weights use $16\times16$ blocks, while activations and gradients use $1\times16$ blocks. The explicit purpose of 2D weight scaling is forward/backward consistency: the same weight tensor should not be quantized differently in the forward and backward pass merely because the transpose changes the quantization axis. Random Hadamard transforms are applied only to weight-gradient GEMMs, with $16\times16$ transforms and a single fixed random sign vector across training. Stochastic rounding is applied only to gradients, not to forward-pass tensors, because the study reports that using it on forward tensors is detrimental and using it on all backward tensors brings little benefit beyond gradients alone (NVIDIA et al., 29 Sep 2025).

The recipe is also explicitly hybrid. For the main 12B run, the first two blocks and the last eight blocks are kept in BF16, corresponding to about 16% of linear layers in higher precision. Embeddings, the output projection head, normalization layers, nonlinearities, attention softmax, certain attention GEMMs, optimizer states, and main weights also remain in higher precision. Under this regime, the NVFP4 validation loss closely tracks FP8 throughout training; during the stable phase, relative loss error stays below 1%, and it rises to slightly above 1.5% near the end as the learning rate decays (NVIDIA et al., 29 Sep 2025).

This baseline established two points that recur across later work. First, NVFP4 pre-training became empirically credible at scales where small quantization errors accumulate over trillions of tokens. Second, the dominant failure modes were not reduced to a single issue; the literature subsequently decomposes them into scale-selection error, unbiased gradient estimation, outlier management, and late-stage optimization pathologies.

3. Adaptive scaling and unbiased gradient estimation

A major line of subsequent work argues that early NVFP4 recipes left accuracy on the table because they handled block scaling and unbiasedness too rigidly. The most direct example is Four Over Six (4/6), which attributes much of NVFP4’s brittleness to standard scale-to-6 block normalization. Because FP4 E2M1 has sparse representable magnitudes near the top of the range, blocks with important values around 4–5 are poorly represented when the block maximum is always mapped to 6. The 4/6 method therefore evaluates two candidate scales per block—one mapping the maximum to 6 and one mapping it to 4—and chooses the lower-error candidate by MSE. In the reported pre-training experiments, plain NVFP4 diverges in every case tested, whereas NVFP4 + 4/6 trains normally and follows loss curves much closer to BF16 (Cook et al., 1 Dec 2025).

TetraJet-v2 extends the same concern from forward quantization to end-to-end fully-quantized training. It quantizes activations, weights, and gradients of all linear layers in both the forward and backward pass, and identifies two additional obstacles: weight oscillation and outliers. Its quantization core is an unbiased double-block design with an outer block of size $1\times128$ and an inner block of size $1\times16$, intended to be more hardware-friendly and more accurate than per-tensor second quantization scaling. Around this core it adds OsciReset, which resets oscillating master weights to the center of their current quantization bin, and OutControl, which keeps selected persistent activation outlier channels in higher precision. Across model sizes up to 370M and data sizes up to 200B tokens, the paper reports an average 51.3% reduction in the performance gap to full-precision training (Chen et al., 31 Oct 2025).

Quartet II makes the most explicit analytical separation between forward representation capacity and backward unbiasedness. Its central claim is that prior methods overused element-wise stochastic rounding as a universal solution for unbiased gradient estimation, even though stochastic rounding is very noisy at 4-bit precision. On standard normal data, the paper reports MSE $\times10^{-3}$ of 9.0 for RTN, 23.5 for SR, and 17.5 for SR + FourOverSix, whereas its proposed MS-EDEN quantizer achieves 9.8 while remaining unbiased. MS-EDEN adapts EDEN to NVFP4 microscaling by shifting the corrective randomness from each FP4 element to the FP8 group scales. The correction factor is

$$S=\frac{\langle \boldsymbol{x},\boldsymbol{x}\rangle}{\langle \operatorname{RHT}(\boldsymbol{x},\omega),Q(\operatorname{RHT}(\boldsymbol{x},\omega))\rangle},$$

after which the corrected group scales are stochastically rounded to preserve unbiasedness. The paper states the key property as

$$\mathbb{E}_{\omega_{\mathrm{RHT}},\omega_{\mathrm{SR}}}\big[\operatorname{RHT}^{-1}(\widehat{\boldsymbol{x}})\big]=\boldsymbol{x}.$$

Built on this primitive, Quartet II uses native $\{0,\pm0.5,\pm1,\pm1.5,\pm2,\pm3,\pm4,\pm6\}$0 forward groups with 4/6 on the forward path, and MS-EDEN in the backward GEMMs with re-quantized weights. In Llama-style pre-training ablations, replacing standard NVFP4 recipes with Quartet II reduces validation loss gaps relative to BF16 by at least 20%; in larger Nanochat runs, it also outperforms NVIDIA’s original NVFP4 recipe, TetraJet-v2, and a 4/6-forward baseline (Panferov et al., 30 Jan 2026).

Taken together, these methods shift the field from a single “use stochastic rounding and stabilize the rest” paradigm toward operation-specific quantization design: adaptive forward scaling, unbiased but low-variance backward estimation, and explicit handling of optimization artifacts induced by FP4 binning.

4. Outlier dynamics, hot channels, and block heterogeneity

Another major development is the treatment of outliers as a structured, time-varying phenomenon rather than a generic nuisance. The longitudinal study of NVFP4 pre-training compares Softmax Attention and Linear Attention and finds that Linear Attention reduces per-tensor heavy tails but still exhibits persistent block-level spikes under block quantization. It localizes outliers to specific architectural components: Softmax in SA, gating in LA, and SwiGLU in FFN, with post-QK operations described as especially sensitive. The study also reports that outliers evolve from transient spikes early in training to a small set of persistent hot channels in later stages (Dong et al., 2 Feb 2026).

This diagnosis leads to Hot-Channel Patch (HCP), an online compensation mechanism that identifies hot channels by residual-based importance scores and reinjects residuals for those channels during training. The paper motivates the approach through the exact decomposition

$\{0,\pm0.5,\pm1,\pm1.5,\pm2,\pm3,\pm4,\pm6\}$1

and then targets the dominant error terms on a small hot set. In experiments, roughly 9.09% of channels are designated as hot channels. Combined with higher-precision protection for the most fragile post-QK operations, this yields CHON, which on a GLA-1.3B model trained for 60B tokens reduces the loss gap to BF16 from 0.94% to 0.58% while maintaining downstream accuracy. The implementation is explicitly hardware-minded: fused Triton kernels reduce average HCP overhead from 16.15% in the pre-fuse version to 5.27% post-fuse (Dong et al., 2 Feb 2026).

A related but distinct response to heterogeneity appears in MixFP4. That work argues that a single FP4 micro-format, E2M1, mismatches heterogeneous block statistics in LLM tensors: outlier-heavy blocks benefit from exponent-rich spacing, while flatter blocks can be better served by more uniform, INT-like spacing. MixFP4 therefore selects between E2M1 and E1M2 per block by MSE, reuses NVFP4’s scale hierarchy, and stores the format choice with zero additional metadata by repurposing the sign bit of the FP8 E4M3 block scale. Both payloads are decoded into a unified internal E2M2 compute representation to avoid datapath duplication. In Qwen3-style pre-training runs at 114M parameters for 2.5B tokens and 476M parameters for 4B tokens, the NVFP4 baseline is described as having unstable optimization, whereas MixFP4 gives smoother loss and lower final loss; the hardware study reports 3.1% area overhead and 1.5% power overhead on the tensor-core compute datapath (Zou et al., 29 May 2026).

These results make two technical points clear. First, outlier control in NVFP4 is increasingly channel- and operation-specific rather than solely tensor-global. Second, block heterogeneity is not only a matter of scale selection; it can also be a matter of choosing the micro-format itself.

5. Empirical status and systems-level extensions

The major NVFP4 pre-training methods now span a continuum from hybrid stabilization recipes to fully-NVFP4 linear-layer schemes and block-adaptive extensions. The following table summarizes the main variants and their reported training evidence.

Method	Core mechanism	Reported training evidence
Baseline NVFP4 recipe (NVIDIA et al., 29 Sep 2025)	RHT on Wgrad, 2D weight scaling, SR on gradients, selective BF16 layers	12B model on 10T tokens; loss and downstream accuracy comparable to FP8
Four Over Six (Cook et al., 1 Dec 2025)	Per-block choice between scale-to-4 and scale-to-6 by MSE	Plain NVFP4 diverges in every pre-training case tested; NVFP4 + 4/6 trains normally
TetraJet-v2 (Chen et al., 31 Oct 2025)	Unbiased double-block quantization, OsciReset, OutControl	Average 51.3% reduction in the performance gap to full precision
Quartet II (Panferov et al., 30 Jan 2026)	Native $\{0,\pm0.5,\pm1,\pm1.5,\pm2,\pm3,\pm4,\pm6\}$2 forward + 4/6, MS-EDEN backward	Validation-loss gaps to BF16 reduced by at least 20% in Llama-style ablations
CHON (Dong et al., 2 Feb 2026)	HCP with post-QK protection	GLA-1.3B gap reduced from 0.94% to 0.58%
MixFP4 (Zou et al., 29 May 2026)	Per-block E2M1/E1M2 selection	Smoother loss and lower final loss than NVFP4 baseline in Qwen3-style runs

Reported hardware gains are correspondingly substantial, although they depend on how much of the end-to-end runtime remains outside the matmul path. Quartet II provides CUDA kernels for Blackwell/RTX 5090-class hardware and reports more than 4.2× speedup over BF16 for linear-layer execution, about 70% better performance than prior FP4 training kernels from Quartet, and 51k tokens/s versus 20.8k tokens/s for BF16 in a 1.1B-parameter single-RTX-5090 training setup, or 245% of BF16 throughput (Panferov et al., 30 Jan 2026).

NVFP4 pre-training has also moved beyond autoregressive LLMs. LongLive-2.0 presents an NVFP4-based parallel infrastructure for long video generation in both training and inference. On the training side, it couples NVFP4 with Balanced SP, a sequence-parallel autoregressive layout that pairs clean-history and noisy-target temporal chunks on each rank and makes VAE preprocessing sequence-parallel. For AR training it uses the standard NVFP4 recipe—2D block scaling for weights, 1D block scaling for activations and gradients, and higher precision for reductions, normalization statistics, and optimizer states—and additionally uses Random Hadamard Transform before quantization for the gradient-sensitive weight-gradient GEMM. The paper reports iteration time dropping from 1372.9 ms with BF16+SP to 639.5 ms with NVFP4+Balanced SP at 64 seconds, a 2.1× speedup, and summarizes overall results as up to 2.15× speedup in training and 1.84× in inference; LongLive-2.0-5B reaches 45.7 FPS inference (Chen et al., 18 May 2026).

The systems literature therefore no longer treats NVFP4 as a narrow inference format. It is increasingly presented as a precision regime spanning pre-training, distillation, KV-cache compression, and deployment.

6. Limitations, misconceptions, and open problems

A persistent misconception is that “NVFP4 pre-training” already means uniform end-to-end FP4 execution of every operation. The literature does not support that generalization. The initial 12B/10T-token demonstration still keeps the first two blocks and last eight blocks in BF16, and retains embeddings, the output projection head, normalization layers, nonlinearities, attention softmax, certain attention GEMMs, optimizer states, and main weights in higher precision. The authors explicitly identify reducing this remaining BF16 footprint, quantizing all linear layers without harming convergence, extending NVFP4 to attention and communication paths, and testing larger models, longer horizons, and MoE architectures as future work (NVIDIA et al., 29 Sep 2025).

A second misconception is that any low-error forward heuristic can automatically serve as a correct backward estimator. Quartet II directly argues against this: 4/6 is beneficial on the forward path, but it is not inherently unbiased, and if the scale branch is chosen deterministically by lower MSE it introduces bias. The same paper’s broader point is that unbiasedness and low variance must be designed together, not traded naively against one another (Panferov et al., 30 Jan 2026).

A third misconception is that the main barrier is merely the coarse scale format. Four Over Six shows that scale-factor error is minor relative to value quantization, and that the critical damage is concentrated in near-maximal values. It also notes a concrete format dependency: 4/6 relies on the precision of NVFP4’s FP8 E4M3 scales and does not transfer to MXFP4’s E8M0 scales (Cook et al., 1 Dec 2025).

Finally, the pre-training literature consistently distinguishes training-time alignment from post-training compression. ARCQuant, despite being highly relevant to NVFP4 deployment, is explicitly a PTQ method calibrated offline with 128 WikiText2 samples and does not alter training dynamics. LongLive-2.0 similarly reports that direct PTQ degrades quality more noticeably, whereas pre-trained W4A4 NVFP4 stays closer to BF16 and preserves sharper visual details than PTQ, especially in facial regions (Meng et al., 12 Jan 2026, Chen et al., 18 May 2026).

These limitations suggest a current consensus rather than a settled endpoint. NVFP4 pre-training is no longer treated as intrinsically unstable, but stable accuracy is achieved only when the precision regime is co-designed with block scaling, unbiased gradient estimation, outlier control, and hardware execution constraints.