Full-Stack FP4 in LLMs

• Full-Stack FP4 is an end-to-end design regime that combines specialized 4-bit numerical formats, quantization strategies, and hardware-aware rules to enable efficient LLM training and inference.
• It employs tailored quantization recipes and module-specific scaling to stabilize activations, optimizers, and attention mechanisms while minimizing rounding biases.
• The system advocates adaptive micro-format selection and mixed-precision techniques to optimize computational throughput, memory usage, and accuracy at model scale.

Searching arXiv for papers on FP4 training/inference and “Full-Stack FP4”. Full-Stack FP4 denotes an end-to-end approach to 4-bit floating-point computation for LLMs in which numerical formats, quantization and rounding rules, layer- and module-specific recipes, optimizer states and arithmetic, attention kernels, scaling behavior, and hardware support are treated as a single system. In recent literature, the term is used both as a general stack-wide perspective—covering “numerical formats and their geometry, quantization and rounding theory, algorithmic training recipes, large-scale empirical behavior and scaling laws, and hardware/system implications” (Zhao et al., 18 Jun 2026)—and as a concrete NVFP4 pretraining framework that jointly stabilizes linear projections, optimizers, and attention (Ding et al., 5 Jul 2026). Across this literature, Full-Stack FP4 is not a single format or kernel, but a design regime for making 4-bit floating-point training and inference viable at model and system scale.

1. Scope and historical emergence

Early FP4 work in transformers was primarily post-training quantization. “LLM-FP4” showed that both weights and activations could be quantized to 4-bit floating-point in a post-training manner, and reported an average score of 63.1 on common-sense zero-shot reasoning tasks for LLaMA-13B, only 5.8 below the full-precision model (Liu et al., 2023). Training-oriented work then moved from isolated low-bit components toward whole-stack recipes. “Optimizing LLM Training Using FP4 Quantization” introduced what it describes as the first FP4 training framework for LLMs, combining a differentiable quantization estimator, outlier clamping and compensation, mixed precision, and vector-wise quantization, and scaling to 13B-parameter models trained on up to 100B tokens (Wang et al., 28 Jan 2025). “Towards Efficient Pre-training: Exploring FP4 Precision in LLMs” framed FP4 as a pretraining scheme with mixed-precision quantization strategies tailored for different modules and training stages, reporting accuracy comparable to BF16 and FP8 with smaller theoretical computational cost (Zhou et al., 17 Feb 2025).

Subsequent work broadened the meaning of FP4 training. “Quartet” argued that native FP4 training can be optimal when the heavy linear algebra runs natively in MXFP4 on Blackwell, with a thin FP16/FP32 shell for optimizer and numerically sensitive operations (Castro et al., 20 May 2025). “FP4 All the Way” then described fully quantized training in which forward, backward, and update GEMMs use predominantly FP4 for weights, activations, and gradients on datasets up to 200 billion tokens (Chmiel et al., 25 May 2025). By 2026, “Full-Stack FP4” had become an explicit programmatic goal: stable pretraining with quantized projections, optimizers, and attention in one NVFP4 pipeline (Ding et al., 5 Jul 2026).

Dimension	Representative work	Illustrative result
Post-training FP4	(Liu et al., 2023)	LLaMA-13B W4A4 average score 63.1
FP4 pretraining recipes	(Zhou et al., 17 Feb 2025)	Accuracy comparable to BF16 and FP8
Fully quantized training	(Chmiel et al., 25 May 2025)	7B model trained on 256 Gaudi2 accelerators
Complete NVFP4 stack	(Ding et al., 5 Jul 2026)	3B/64B-token pretraining with 1.47% loss gap
Platform-specific realization	(Zhang et al., 3 Mar 2026)	671B MoE with 14.8% memory reduction

2. Numerical formats, microscaling, and quantization geometry

A central question in Full-Stack FP4 is not merely whether 4-bit floating point is used, but which 4-bit grid is used and how it is scaled. Current hardware paths are dominated by non-uniform E2M1 data elements. In the notation used across several works, E2M1 denotes 1 sign bit, 2 exponent bits, and 1 mantissa bit, whereas E1M2 shifts one bit from exponent to mantissa, yielding a more uniform local grid (Zhao et al., 18 Jun 2026). The geometry of these grids matters directly to rounding bias. “Rethinking Shrinkage Bias in LLM FP4 Pretraining” defines the expected RTNE error in normalized magnitude space as

$$b_G(\mathcal P) = \mathbb{E}_{t \sim \mathcal P}[\rho_G(t) - t],$$

and shows that for non-uniform grids such as E2M1, asymmetric bins can induce systematic negative error, or “Shrinkage Bias” (Zhao et al., 18 Jun 2026). In contrast, uniform grids such as E1M2 or INT4 have symmetric bins and zero expected RTNE error in normalized magnitude space.

This distinction propagates upward. The same work decomposes quantized GEMM operands into aligned and orthogonal components,

$$\widehat{A} = \alpha_A A + R_A,\qquad \widehat{B} = \alpha_B B + R_B,$$

so that the signal term in a GEMM is attenuated by approximately $\alpha_A \alpha_B$. Across depth, the effective signal factor becomes

$\prod_{k=1}^{K} (1-\delta_k),$

which the paper interprets as multiplicative accumulation of shrinkage across layers rather than zero-mean noise that averages out (Zhao et al., 18 Jun 2026).

Other work approaches the same issue through format adaptivity rather than a single preferred grid. “MixFP4” extends NVFP4 by selecting between E2M1 and E1M2 per block while reusing NVFP4’s scale hierarchy and encoding the format choice with zero additional metadata by repurposing the sign bit of the FP8 E4M3 block scale (Zou et al., 29 May 2026). Its rationale is statistical heterogeneity at block level: E2M1 is favorable for outlier-heavy blocks, whereas E1M2 behaves more like uniform INT4 and is favorable for flatter blocks. The hardware cost reported for this adaptive micro-format is modest tensor-core overhead, specifically 3.1% area and 1.5% power (Zou et al., 29 May 2026).

At the activation-quantization level, “LLM-FP4” emphasized that floating-point performance depends heavily on exponent allocation and clipping range, and argued that transformer activations exhibit high inter-channel variance and low intra-channel variance (Liu et al., 2023). That paper’s solution—per-channel activation quantization reparameterized as exponential biases of weights—already foreshadowed the later Full-Stack FP4 emphasis on module-specific scaling rules rather than a single universal FP4 rule.

3. Training stack: linear layers, gradients, and optimizers

FP4 training papers converge on the point that naive direct quantization of transformer linear layers is insufficient. “Optimizing LLM Training Using FP4 Quantization” addressed this with a differentiable quantization estimator for weights and an outlier clamping and compensation strategy for activations, combined with vector-wise quantization and mixed precision (Wang et al., 28 Jan 2025). The activation-side motivation was explicit: absmax scaling can collapse most of the activation mass toward zero under FP4, and sparse compensation of the clamped residual avoids the resulting underflow and collapse.

“FP4 All the Way” generalized the scope to fully quantized training, with forward, backward, and update GEMMs all using NVFP4. It reported that NVFP4 with blocks of 16 and E4M3 scales gives the best trade-off among tested scale formats and block sizes, and adopted a split rounding policy: round-to-nearest in the forward pass, stochastic rounding in backward and update passes (Chmiel et al., 25 May 2025). The same work also introduced a theoretical and empirical threshold for effective quantized training, stating that when the gradient-to-noise ratio

$R = \frac{\|g\|}{\sigma_q \sqrt{d}}$

falls to approximately $\sqrt{3}$, quantized training becomes less effective (Chmiel et al., 25 May 2025). Its proposed response was Quantization-Aware Fine-Tuning, in which forward remains FP4 while backward and update switch to BF16 late in training.

“Quartet” shifted attention from isolated FP4 tricks to a precision-aware scaling law. It modeled validation loss as a function of model size $N$, data $D$, and forward/backward precision through parameter efficiency $\text{eff}_N$ and data efficiency $\text{eff}_D$, and used this to argue that native FP4 can be accuracy-optimal relative to compute in realistic Llama-type pretraining regimes (Castro et al., 20 May 2025). Its recipe paired a forward quantization scheme that minimizes forward error with a backward scheme based on stochastic rounding and Hadamard rotation, both implemented with Blackwell-native MXFP4 kernels.

The optimizer stack remained a major gap until “Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention.” That work treated AdamW second moments as a distinct numerical object: non-negative, heavy-tailed values appearing in the denominator of the update, and therefore especially fragile under naive FP4 storage (Ding et al., 5 Jul 2026). Its solution was a transformed-state pipeline for $$\widehat{A} = \alpha_A A + R_A,\qquad \widehat{B} = \alpha_B B + R_B,$$0: square root, tile-wise mean subtraction, Hadamard transform, and then NVFP4 quantization. For Root, it further implemented native NVFP4 Newton–Schulz iterations rather than delegating the orthogonalization matmuls to higher precision (Ding et al., 5 Jul 2026).

A different realization of stack-wide FP4 efficiency was shown on hardware without native FP4 Tensor Cores. “Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs” kept core MoE computations in FP8 but compressed forward activations and expert-parallel communication with MXFP4. At 671B parameters, it reported peak activation memory reduction of 14.8% (11.8 GB) and throughput improvement of 12.5%, from 1157 to 1302 tokens per GPU per second (Zhang et al., 3 Mar 2026). This is a full-stack result in a narrower sense: storage and communication are FP4, arithmetic remains FP8/BF16.

4. Attention and inference-specific stack design

Attention has become the clearest example of why Full-Stack FP4 is not synonymous with homogeneous FP4 everywhere. “ThriftAttention” studies long-context inference and argues that output impact of quantization error is highly non-uniform across query-key blocks. It therefore computes only a small set of important blocks in FP16 and the remainder in FP4, merging both via online softmax (Sharratt, 21 May 2026). Its headline empirical result is that computing only 5% of query-key blocks in FP16 recovers on average 89.1% of the FP4-to-FP16 performance gap, with the advantage increasing with sequence length (Sharratt, 21 May 2026). The implication is direct: pure FP4 attention is not adequate for long contexts, but selective mixed precision is.

“SharQ” addresses a different inference bottleneck: activation compression for LLM serving. It decomposes each activation tensor into an input-adaptive sparse backbone and a dense residual defined relative to the quantized sparse backbone rather than the unquantized sparse values (Meng et al., 25 Jun 2026). A sparse FP4 GEMM handles the backbone; a dense FP4 GEMM compensates for both sparsification loss and sparse-path quantization error. Across several model families, this recovers 43–63% of the NVFP4-to-FP16 accuracy gap, while delivering 2.2–2.4× latency reduction over FP16 and 1.2–1.4× throughput improvement over FP8 on an RTX 5090 (Meng et al., 25 Jun 2026).

Layer sensitivity analysis reinforces the need for such non-uniform policies. “Diagnosing FP4 inference” compares MXFP4 and NVFP4 on Qwen2.5 and finds that MLP up- and down-projection layers consistently dominate in sensitivity, while gate projections are moderate and attention projections are substantially less sensitive (Cim et al., 5 Mar 2026). It also shows that sensitivity does not universally localize to final blocks; early blocks can be highly sensitive, particularly under MXFP4 (Cim et al., 5 Mar 2026). This directly contradicts the common simplification that “attention is always the fragile part” or that only the last transformer blocks require protection.

5. Empirical scaling, stability, and hardware implications

A major theme of the newer literature is that FP4 quality is no longer evaluated only on toy networks or short runs. “Rethinking Shrinkage Bias” evaluated long-run pretraining on Dense 1.5B, MoE 7.9B, and MoE 124B and introduced UFP4, a uniform 4-bit recipe using RHT on all three training GEMMs while restricting stochastic rounding to $$\widehat{A} = \alpha_A A + R_A,\qquad \widehat{B} = \alpha_B B + R_B,$$1 (Zhao et al., 18 Jun 2026). Its BF16-relative loss degradation over the latest 1000 steps was 0.9673% for Dense 1.5B, 1.8469% for MoE 7.9B, and 1.3863% for MoE 124B, all below strong E2M1-based baselines (Zhao et al., 18 Jun 2026). It also reported that fused RHT+quantization costs only about 1.06–1.07× the latency of standalone quantization, whereas unfused implementations cost about 1.41–1.62× fused latency (Zhao et al., 18 Jun 2026).

“Full-Stack FP4” moved the benchmark from module-level ablation to stack-level validation. Its 3B/64B-token pretraining run reported near-BF16 performance with merely a 1.47% loss gap while jointly quantizing projections, optimizer states and arithmetic, and attention-critical tensors (Ding et al., 5 Jul 2026). The same paper attributes the linear-only loss gap reduction from 1.40% to 0.61% specifically to LoRA-SVD (Ding et al., 5 Jul 2026), implying that structural decomposition, not just improved scaling or rounding, is central to pushing FP4 beyond the earlier noise ceiling.

System implications extend beyond arithmetic throughput. “MixFP4” argues that future tensor cores should not force a single E2M1 payload format when block statistics are heterogeneous (Zou et al., 29 May 2026). “Rethinking Shrinkage Bias” similarly argues that future accelerators should support E1M2/INT4-style uniform grids as first-class training primitives alongside E2M1 (Zhao et al., 18 Jun 2026). On current non-FP4-native hardware, “Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs” demonstrates that software-hardware co-design can still realize FP4-level efficiency through direct FP8-to-FP4 conversion, scaling-aware row/column conversion, and FP4-aware communication paths (Zhang et al., 3 Mar 2026).

6. Debates, misconceptions, and future directions

A persistent misconception is that Full-Stack FP4 means every tensor and every arithmetic path must be 4-bit. The literature does not support that interpretation. Practical systems consistently reserve higher precision for certain operations: softmax and normalization statistics in training-oriented work, $$\widehat{A} = \alpha_A A + R_A,\qquad \widehat{B} = \alpha_B B + R_B,$$2 and $$\widehat{A} = \alpha_A A + R_A,\qquad \widehat{B} = \alpha_B B + R_B,$$3 in attention, or a small fraction of long-context attention blocks in inference (Ding et al., 5 Jul 2026). This suggests that Full-Stack FP4 is better understood as a module-wise precision architecture than as absolute precision uniformity.

A second debate concerns whether current E2M1-centered hardware is sufficient. The answer is unsettled. UFP4 attributes instability in many E2M1 recipes to geometric shrinkage bias and argues for uniform grids (Zhao et al., 18 Jun 2026), whereas MixFP4 treats the issue as one of block-level heterogeneity and resolves it by allowing E2M1 and E1M2 to coexist per block without changing the main GEMM path (Zou et al., 29 May 2026). These are not identical positions: one privileges uniformity as a training primitive, the other privileges adaptive micro-format selection.

A third open question is lifecycle policy. Several training results indicate that early or mid-stage FP4 can be effective, but that late-stage optimization may require selective precision restoration. “FP4 All the Way” uses a late Quantization-Aware Fine-Tuning phase once the gradient-to-noise ratio approaches the $$\widehat{A} = \alpha_A A + R_A,\qquad \widehat{B} = \alpha_B B + R_B,$$4 threshold (Chmiel et al., 25 May 2025). The 2026 Full-Stack FP4 work similarly suggests that a practical industrial workflow may use full 4-bit early, then gradually restore higher precision in later stages while preserving a low-bit deployment path (Ding et al., 5 Jul 2026).

Taken together, the literature establishes Full-Stack FP4 as a systems doctrine for low-bit LLM computation. Its defining claims are no longer confined to weight-only quantization or isolated GEMM kernels. They concern the geometry of FP4 grids, the propagation of rounding bias, the organization of optimizer states, the asymmetry of attention paths, the sensitivity of specific layers and depths, and the hardware contract needed to make all of those decisions practical at scale.