---
title: 'FP4 Quantization: Ultra-Low Precision Neural Ops'
url: https://www.emergentmind.com/topics/fp4-quantization
type: topic
---

# FP4 Quantization: Ultra-Low Precision Neural Ops

Four-bit floating-point quantization (FP4) is an ultra-low-precision numerical representation and quantization methodology that enables highly efficient deployment and training of large neural networks. FP4 achieves significant speed, memory, and energy advantages by replacing standard 16- or 32-bit floating-point arithmetic with 4-bit floating-point numbers, subject to the challenges of coarse resolution and limited dynamic range. Recent research has systematically advanced both the theoretical underpinnings and practical hardware/software systems for FP4 inference and training across language models, diffusion models, and multi-modal architectures.

## 1. FP4 Fundamentals and Quantization Formats

FP4 encompasses a family of 4-bit floating-point representations, each specified by a choice of sign (S), exponent (E), and mantissa (M) bit allocations and bias:

| Format        | (S,E,M) | Bias             | Value Range (normals)          | Typical Block Size | Scale Format           |
|:------------- |:-------:|:----------------:|:------------------------------|:------------------:|:----------------------:|
| E2M1 (NVFP4)  | 1,2,1   | $2^{1}-1=1$      | $\pm\{0,0.5,1,1.5,2,3,4,6\}$  | 16                 | E4M3 (8 bits)          |
| E2M1 (MXFP4)  | 1,2,1   | $1$              | $\pm\{0.5,1,1.5,2,3,4,6\}$    | 32                 | E8M0 (8 bits, pow2)    |
| E3M0          | 1,3,0   | $2^{2}-1=3$      | $\pm\{0.125,0.25,0.5,\dots,4\}$| varies             | varies                 |
| IF4           | –       | (int/float mux)  | $\pm[0,1,…,7]$ or FP4         | 16                 | E4M3 (sign as mux flag)|

NVFP4 (NVIDIA) and MXFP4 (AMD/OCP) variants dominate hardware support. Both block macroscaling (16–32 values per scale) and fine-grained quantization are standard. Blockwise scaling is essential for keeping quantization error bounded under ultra-low bitwidths [2603.08747, 2603.28765].

## 2. Algorithms and Quantization Schemes

Fundamental to FP4 is a two-stage quantization pipeline:

- **Blockwise scaling:** For block $B = \{x_1,\ldots,x_{G}\}$, scale $s$ is set as
  $$
  s = \frac{\max_i |x_i|}{\text{FP4}_{\max}}
  $$
  with subsequent block value quantized as $q_i = \operatorname{clip}(\operatorname{round}(x_i/s), q_{\min}, q_{\max})$, and stored as 4 bits.
- **Per-element packing:** Each 4-bit result encodes a sign, exponent, and mantissa.

Several advanced strategies are developed to overcome inherent limitations:

- **Overflow-Aware Scaling (OAS):** Dynamically shifts scale exponent if the block maximum falls near the lower edge of a scale bin, doubling dynamic range at negligible error cost [2603.08713].
- **Macro Block Scaling (MBS):** Allocates enhanced precision at the macro-block (e.g., $128 \times 8$ sub-blocks) level to mitigate outlier-induced overflow [2603.08713].
- **Adaptive Data Types (IF4):** Mixes blockwise FP4/INT4 encodings depending on per-block MSE, flagged via scale sign bit, further lowering error [2603.28765].
- **Mixed-precision allocation and thresholds:** Hybridization with higher-precision channels for outlier-heavy rows/channels (as in MicroMix, HQ-DiT) [2508.02343, 2405.19751].
- **Block-wise affine and Kronecker transformations:** In BATQuant, block-local, learnable affine maps re-shape input distributions without cross-block outlier spread, outperforming previous global or orthogonal rotations [2603.16590].

The quantization process balances dynamic range, outlier suppression, and hardware efficiency, and is adapted by algorithmic innovations such as differentiable quantization estimators [2501.17116], mixture-of-formats layer-wise selection [2305.12356], and scale-aware AdaRound [2503.15465].

## 3. FP4 for Large Language Model Training and Inference

Modern LLMs exploit FP4 for weights, activations, and gradients (full W4A4G4), enabled by blockwise NVFP4 or MXFP4 quantization:

- **Training:** Differentiable gradient estimators (beyond naive STE), mixed-precision scheduling (e.g., FP16/BF16 for non-matmul ops), vector-wise/channel-wise scaling, and outlier clamping+compensation restore training stability. Loss gap to BF16 is reduced to $<0.09$ at 13B model scale, and zero-shot task accuracy can even improve over baseline [2501.17116, 2505.19115].
- **Inference:** Large-scale systematic analysis shows:
  - MLP up/down projections are consistently most sensitive to FP4 errors; attention projections are robust [2603.08747].
  - Per-layer and per-channel scale selection (LLM-FP4) closes the gap with BF16/FP16 inference, e.g., LLaMA-13B suffers only $-5.8$ accuracy drop (63.1 vs 68.9), dramatically outperforming pure INT4 [2310.16836].
  - Mixture-of-formats quantization (MoFQ) and MR-GPTQ boost recovery, bringing MXFP4 accuracy to $\sim$1–2% of NVFP4 [2509.23202].

FP4 quantization is now the standard for high-throughput kernel implementations in Blackwell/TensorCore GPUs, with up to $4\times$ throughput over FP16 and sub-1% end-to-end quality degradation [2505.11594, 2509.23202].

## 4. Architectural and Hardware Considerations

The FP4 design space includes E2M1, E3M0, and extended-range UE5M3 variants, with the following trade-offs [2509.17791]:

- **NVFP4 (block 16, E4M3 scales):** Optimal runtime accuracy, moderate area overhead (+12–15%), often selected.
- **MXFP4 (block 32, E8M0 scales):** Lower area, initially higher error unless enhanced by OAS/MBS or MR-GPTQ.
- **IF4 (adaptive float/int):** Minimal overhead for logic and storage, with area only ∼0.016 mm² per MAC, 5% slower at system level [2603.28765].
- **Physical MAC area:** $92$–$182\,\mu$m² for INT4, $92$–$111\,\mu$m² for FP4 on TSMC 40 nm at $0.5$ GHz [2310.16836].

Efficient memory layouts, scale factor packing, and direct format conversions (e.g. MXFP4 $\leftrightarrow$ FP8), are necessary for contemporary accelerator architectures, including Hopper and Blackwell-class GPUs [2603.02731, 2508.02343].

## 5. FP4 in Diffusion, Vision, and Multimodal Models

FP4 methods increasingly target diffusion models and vision transformers:

- **Diffusion Models:** FP4-based methods (HQ-DiT, FP4DiT, MSFP) outperform INT4 PTQ in both FID and CLIP metrics, retaining image quality at extreme $W4A4$ or $W4A6$ settings [2405.19751, 2503.15465, 2505.21591]. Hybrid channel-wise composition, token-wise minmax scaling, and per-layer mixup-sign quantization address high activation dynamic range and asymmetry.
- **Multi-Modal LLMs:** BATQuant demonstrates MXFP4 can achieve 96.4% recovery in multimodal evaluations under W4A4KV16, by block-wise transformations closely aligned to hardware granularity [2603.16590].

Token- and channel-wise activation quantization and data-driven format selection are fundamental in these architectures.

## 6. Quantization-Aware Training, Stability, and Error Analysis

Stability in FP4 training is achieved via:

- **Differentiable Quantization Estimators:** DGE, spline-based, or non-uniform (non-STE) estimators improve backward-pass stability [2501.17116, 2509.17791].
- **Stochastic Rounding (SR):** Applied on backward and update passes, limits gradient bias and helps convergence in low-precision regimes [2505.19115].
- **Mean-subtraction ("Averis") and SVD Outlier Filtering:** Rank-one bias in activations compresses block dynamic range and scales; mean-centering each block recovers near-BF16 performance at negligible cost [2603.10444, 2405.19751, 2310.16836].
- **Theoretical Thresholds:** Quantized training ceases to make learning progress when gradient norms drop below $\sqrt{3}$ times the quantization noise, setting a lower bound for effective FP4-only training [2505.19115].

A principled recipe for highly robust FP4 training includes block-/tensor-wise scaling, Hadamard scattering (for outlier reduction), and combined use of FP4/INT4/MX/UE5M3 as needed according to workload and model scale [2509.17791].

## 7. Performance, Accuracy Tradeoffs, and Recommendations

Comprehensive empirical benchmarks show:

| Format/Method | BOPs/Element | Perplexity (LLaMA-4B/WT2) | Downstream Recovery | Speedup vs. FP16 | Hardware Area Overhead | Main Use-case |
|---------------|--------------|--------------------------|---------------------|------------------|-----------------------|--------------|
| NVFP4         | 4.5          | 18.03                    | 94–99%              | 2–4$\times$      | +12–14%               | LLM train/infer|
| MXFP4 (OAS+MBS)| 4.25         | 19.77 (→17.78, IF4/OAS)  | 89–97%              | 2–4$\times$      | baseline              | LLM/MLLM     |
| IF4           | 4.5          | 17.78                    | 96%+                | ~–5% vs NVFP4    | Negligible            | Niche, mix   |
| MR-GPTQ       | 4–4.5        | 75.84 (Llama-8B)         | 93–96%              | $\lesssim$1% overhead| low                 | Post-train Q |
| HQ-DiT        | –            | sFID +0.12 @ W4A4        | $\sim$99%           | 3–5$\times$      | none                  | Diffusion    |

For best-practice deployment:

- Use NVFP4 or IF4 on Blackwell or compatible accelerators. For MXFP4 hardware, augment with OAS+MBS for near-parity.
- For training, apply DGE or advanced estimators, mean-subtraction, blockwise SR, and switch to QAF if gradient noise dominates.
- For vision or diffusion, combine channel-/token-wise scaling, mixup-sign quantization, and universal Hadamard transforms.
- For memory/compute-constrained scenarios or MoE, integrate FP4 storage/comm with row→col fused format conversion [2603.02731].

FP4 has matured to a point where, with appropriate algorithmic, calibration, and hardware support, it offers an unrivaled trade-off between efficiency and accuracy for large-model deployment, with continued research closing remaining gaps for extreme-scale and cross-modal regressions.

Source: https://www.emergentmind.com/topics/fp4-quantization