Four-Bit Floating Point (FP4): Formats & Advances

• FP4 is a low-precision family of floating point formats that uses 4-bit representations with flexible exponent and mantissa splits to reduce memory and compute costs.
• Adaptive scaling techniques such as block floating-point and microscaling enable FP4 to mitigate quantization error while supporting efficient large-scale model inference and training.
• Studies show that varying FP4 variants (e.g., E2M1, E1M2) and hardware co-design strategies can optimize performance, balancing dynamic range against local precision.

Four-bit floating point (FP4) denotes a class of 4-bit floating-point number systems used to reduce memory footprint, bandwidth, and arithmetic cost in large-model inference and training. In current literature, FP4 is not a single canonical datatype but a family of encodings and execution schemes that combine a 4-bit element representation with block-wise or hierarchical scaling, calibration or optimization rules, and hardware-specific kernels. It is the lowest precision in recent LLM quantization studies that still preserves essential floating-point structure such as sign and exponent, but its practical behavior is dominated by quantization error, scale quantization, outlier handling, and strong heterogeneity across layers, blocks, and tasks (Cim et al., 5 Mar 2026, Egiazarian et al., 27 Sep 2025).

1. Numerical meaning and representational variants

In standard low-bit floating-point notation, FP4 uses one sign bit and three remaining bits split between exponent and mantissa. Recent post-training quantization work explicitly treats FP4 as a search space over formats such as $E3M0$, $E2M1$, and $E1M2$, rather than as a single fixed layout (Liu et al., 2023). A generic floating-point value is written as

$$f = (-1)^{d_s} 2^{p-b}\left(1+\frac{d_1}{2}+\frac{d_2}{2^2}+\cdots+\frac{d_m}{2^m}\right),$$

with sign bit $d_s$, exponent value $p$, bias $b$, and $m$-bit mantissa (Chen et al., 19 Mar 2025).

The most common FP4 element format in LLM inference and training papers is $E2M1$: one sign bit, two exponent bits, and one mantissa bit. In NVFP4-style quantization, the magnitude codebook for $E2M1$ is $E2M1$0 (Cook et al., 1 Dec 2025). Other works emphasize that $E2M1$1 can be preferable for flatter or more uniform block statistics, while $E2M1$2 can be useful when dynamic range is prioritized over intra-scale precision (Chen et al., 19 Mar 2025, Zou et al., 29 May 2026).

This numerical flexibility is central to FP4’s behavior. In diffusion-transformer PTQ, FP4 is explicitly described as a family of 4-bit floating-point formats whose exponent/mantissa split trades dynamic range against local precision; when the mantissa takes all remaining bits, the format collapses to the uniform integer case, with $E2M1$3 identified as INT4 (Chen et al., 19 Mar 2025). Comparative LLM work reaches a related conclusion from another direction: FP4 is not universally superior or inferior to INT4, and the preferred 4-bit format can vary across layers (Zhang et al., 2023).

Some studies also modify the representable set itself. A redesigned FP4-$E2M1$4 for weight-only LLM quantization reallocates NaN/Inf code points to ordinary numeric values, extending the usable set to $E2M1$5 and $E2M1$6; that redesign yields about 35% lower quantization error than IEEE-aligned FP4-$E2M1$7 in the reported experiments (Zhang et al., 2023).

2. Block scaling, microscaling, and format families

Raw FP4 elements are usually too limited to be deployed without external scaling. The dominant remedy is block floating-point or microscaling: values are partitioned into small groups, each group shares a scale, and only the normalized residual values are stored in FP4. A general microscaling representation is often described by group size $E2M1$8, element representation $E2M1$9, and scale representation $E1M2$0 (Egiazarian et al., 27 Sep 2025). In related training analyses, a micro-scaled block is written as $E1M2$1, where $E1M2$2 is the shared scale and $E1M2$3 the low-bit element (Hu et al., 22 Sep 2025).

The operational advantage is that scale products can be factored outside local dot products. In block-scaled execution,

$E1M2$4

and for a block dot product,

$E1M2$5

so scale application occurs once per block rather than once per element (Zou et al., 29 May 2026).

Scheme	Block structure	Scale representation
MXFP4	32-element groups, FP4 $E1M2$6 elements	$E1M2$7 or shared 8-bit exponent
NVFP4	16-element groups, FP4 $E1M2$8 elements	$E1M2$9 scales with dynamic scaling
HiF4	64-element block with 32 bits metadata	Three-level scaling hierarchy

This table captures the major families that recur across the literature. MXFP4 is defined as $$f = (-1)^{d_s} 2^{p-b}\left(1+\frac{d_1}{2}+\frac{d_2}{2^2}+\cdots+\frac{d_m}{2^m}\right),$$0, while NVFP4 is $$f = (-1)^{d_s} 2^{p-b}\left(1+\frac{d_1}{2}+\frac{d_2}{2^2}+\cdots+\frac{d_m}{2^m}\right),$$1; the corresponding average storage costs are 4.25 bits per element for MXFP4 and 4.5 bits per element for NVFP4 (Egiazarian et al., 27 Sep 2025). Another inference study describes MXFP4 more concretely as $$f = (-1)^{d_s} 2^{p-b}\left(1+\frac{d_1}{2}+\frac{d_2}{2^2}+\cdots+\frac{d_m}{2^m}\right),$$2 values in 32-element blocks with a shared 8-bit exponent, and NVFP4 as a 16-element block-scaled format with dynamic scaling, 4-bit scales, and a max-calibration algorithm (Cim et al., 5 Mar 2026).

HiFloat4 (HiF4) pushes the same idea further with a 64-element block and a three-level scaling hierarchy. In one formulation, each HiF4 unit packs 64 4-bit elements with 32 bits of shared scaling metadata, averaging 4.5 bits per value; the metadata consists of a global $$f = (-1)^{d_s} 2^{p-b}\left(1+\frac{d_1}{2}+\frac{d_2}{2^2}+\cdots+\frac{d_m}{2^m}\right),$$3-like scale and lower-level micro-exponents that capture intra-group variation (Luo et al., 11 Feb 2026). A related evaluation on Ascend NPUs describes HiF4 as a three-level hierarchical format contrasted with single-level MXFP4 and two-level NVFP4, and identifies hierarchical scaling as the key reason it remains robust in 4-bit regimes where integer quantization collapses (Zhao et al., 13 Feb 2026).

Asymmetric and adaptive variants modify the same block-scaled substrate rather than replacing it. AMXFP4 retains 4-bit floating-point elements but replaces symmetric shared scales with asymmetric shared scales to model the group-wise asymmetry induced by microscaling (Lee et al., 2024). MixFP4 keeps NVFP4’s scale hierarchy but allows each block to choose between $$f = (-1)^{d_s} 2^{p-b}\left(1+\frac{d_1}{2}+\frac{d_2}{2^2}+\cdots+\frac{d_m}{2^m}\right),$$4 and $$f = (-1)^{d_s} 2^{p-b}\left(1+\frac{d_1}{2}+\frac{d_2}{2^2}+\cdots+\frac{d_m}{2^m}\right),$$5 payloads (Zou et al., 29 May 2026). IF4 goes further by choosing between FP4 and INT4 per block while still using the NVFP4-style $$f = (-1)^{d_s} 2^{p-b}\left(1+\frac{d_1}{2}+\frac{d_2}{2^2}+\cdots+\frac{d_m}{2^m}\right),$$6 scale field (Cook et al., 30 Mar 2026).

3. Post-training quantization and inference behavior

A substantial fraction of FP4 research concerns post-training quantization (PTQ) for LLM inference. One early line of work argues that floating-point quantization is more natural than uniform integer quantization for LLMs because activations are heavy-tailed and outlier-prone. In that setting, FP8 activations consistently outperform INT8, and FP4 weights are competitive with or superior to INT4 in W4A8 settings. On LLaMA-7b under W4A8, FP--FP achieves mean perplexity 15.14 versus 16.09 for INT--FP; on LLaMA-13b, FP--FP gets 11.08 versus 11.31 (Wu et al., 2023). The same work constrains weight scales to powers of two to avoid expensive FP4-to-FP8 alignment overhead and integrates Low Rank Compensation (LoRC), which is described as especially beneficial in smaller models (Wu et al., 2023).

LLM-FP4 extends the PTQ perspective by searching jointly over FP4 format and clipping range. It identifies a recurring transformer activation pattern—high inter-channel variance and low intra-channel variance—and introduces per-channel activation quantization whose extra scaling factors are reparameterized as exponent biases of weights, incurring negligible cost (Liu et al., 2023). In the reported LLaMA-13B 4/4/4 setting, this yields an average score of 63.1 on common-sense zero-shot reasoning tasks, only 5.8 below the full-precision model and 12.7 points above the previous state of the art cited in that paper (Liu et al., 2023).

FP4 PTQ has also been extended beyond LLMs. FP4DiT applies floating-point PTQ to diffusion transformers, combining mixed-format FP4 weight quantization, a scale-aware extension of AdaRound, and token-wise online activation quantization (Chen et al., 19 Mar 2025). The method assigns $$f = (-1)^{d_s} 2^{p-b}\left(1+\frac{d_1}{2}+\frac{d_2}{2^2}+\cdots+\frac{d_m}{2^m}\right),$$7 to the first linear layer in each pointwise feed-forward block, uses $$f = (-1)^{d_s} 2^{p-b}\left(1+\frac{d_1}{2}+\frac{d_2}{2^2}+\cdots+\frac{d_m}{2^m}\right),$$8 for PixArt-$$f = (-1)^{d_s} 2^{p-b}\left(1+\frac{d_1}{2}+\frac{d_2}{2^2}+\cdots+\frac{d_m}{2^m}\right),$$9 and Hunyuan or $d_s$0 for PixArt-$d_s$1 in the remaining weights, and quantizes activations online with $d_s$2 for W4A6 or $d_s$3 for W4A8 (Chen et al., 19 Mar 2025). On PixArt-$d_s$4, FP4DiT reports 27.43 HPSv2 and 0.3076 CLIP at W4A8, and 23.55 HPSv2 and 0.3031 CLIP at W4A6, outperforming the integer PTQ baselines listed in the study (Chen et al., 19 Mar 2025).

Calibration-free microscaling inference has developed along a separate track. AMXFP4 identifies a tradeoff in microscaling: decreasing group size suppresses activation outliers but increases group-wise asymmetry, and symmetric MXFP4 does not model that asymmetry well (Lee et al., 2024). Its asymmetric shared-scale design outperforms MXFP4 by 3% on VQA and exceeds rotation-based methods by 1.6% on CSQA in the reported results (Lee et al., 2024).

4. FP4 training and pre-training

Training in FP4 is substantially more difficult than inference because weights, activations, and gradients all become low-precision objects. One FP4 training framework for LLMs uses $d_s$5 FP4, a Differentiable Gradient Estimator (DGE) for weight updates, and Outlier Clamping and Compensation (OCC) for activations, together with mixed precision outside GEMMs and vector-wise quantization (Wang et al., 28 Jan 2025). That work reports scaling to 1.3B, 7B, and 13B LLaMA-2-style models trained from scratch on 100B tokens. Final training losses are 2.55 versus 2.49 for 1.3B, 2.17 versus 2.07 for 7B, and 1.97 versus 1.88 for 13B when compared with BF16; average zero-shot scores are 53.13 versus 53.23, 54.42 versus 53.87, and 54.95 versus 54.44, respectively (Wang et al., 28 Jan 2025).

A later study demonstrates fully quantized training of LLMs using predominantly FP4 for weights, activations, and gradients on datasets up to 200 billion tokens (Chmiel et al., 25 May 2025). Its central format choice is NVFP4: blocks of 16 $d_s$6 values sharing an $d_s$7 scale. The paper adopts round-to-nearest in the forward pass, stochastic rounding for backward and update passes, and derives a threshold according to which quantized training becomes less effective when the gradient norm falls below approximately $d_s$8 times the quantization noise (Chmiel et al., 25 May 2025). It then uses a late-stage quantization-aware finetuning phase in which the forward pass stays in FP4 while the backward pass switches to BF16. The reported large-scale demonstration trains a 7B Llama2-style model on 256 Intel Gaudi2 accelerators for about 200B tokens over roughly 6 days, and after the short QAF phase the downstream results are described as on par with BF16 (Chmiel et al., 25 May 2025).

The design space of FP4 training is itself now an explicit object of study. A unified microscaling framework analyzes the computational overhead of scale gradients, tensor scaling, stochastic rounding, Hadamard transforms, and differentiable approximations to quantization, then evaluates thousands of configurations across regression, classification, diffusion, and language tasks (Hu et al., 22 Sep 2025). The configurations with the best reported performance-to-overhead trade-off combine Hadamard transformations, tensor scaling, and stochastic rounding, while $d_s$9 emerges as a potentially favorable scale format because it balances range and precision with manageable overhead (Hu et al., 22 Sep 2025).

Format choice also interacts strongly with accelerator architecture. On Ascend NPUs, HiFloat4 pre-training compares HiF4 against MXFP4 in dense and MoE models, with linear and expert GEMMs executed entirely in FP4 (Taghian et al., 9 Apr 2026). The reported FP4 compute fractions are 79.58% for OpenPangu-1B, 76.82% for Llama3-8B, and 95.90% for Qwen3-MoE-30B; the corresponding relative loss errors after 50B tokens are 1.19%, 0.85%, and 0.88% for HiF4, compared with 1.79%, 1.44%, and 1.55% for MXFP4 (Taghian et al., 9 Apr 2026). In the ablations, HiF4 mainly benefits from random Hadamard transforms and can degrade under stochastic rounding, whereas MXFP4 requires stochastic rounding, Hadamard transforms, and truncation-free scaling to approach acceptable behavior (Taghian et al., 9 Apr 2026).

A bibliographic clarification is necessary in the FP4 training literature. The arXiv record titled "Towards Efficient Pre-training: Exploring FP4 Precision in LLMs" (Zhou et al., 17 Feb 2025) does not contain an FP4 method, equations, experiments, or results in its body; the FP4-related material appears only in the bibliography.

5. Hardware architectures and systems support

FP4 has increasingly been treated as a hardware target rather than only a quantization abstraction. A dual-precision hybrid floating-point processing element supports FP8 ($p$0, $p$1) and FP4 ($p$2, $p$3) by reusing a single 4-bit multiplier as either one $p$4 unit in FP8 mode or two parallel $p$5 multipliers in FP4 mode (Kumar et al., 6 Apr 2026). The same work describes a 6-stage pipeline with early special-case detection, truncation-based alignment, CSA accumulation, leading-zero anticipation, normalization without explicit rounding, and output ReLU. Implemented in TSMC 28 nm, the PE reports 1.938 GHz, 0.00396 mm$p$6, 2.1338 mW, 7.75 GFLOPS FP4 throughput, and 3632 GFLOPS/W FP4 energy efficiency (Kumar et al., 6 Apr 2026).

At the GPU-kernel level, MR-GPTQ argues that practical FP4 inference depends on co-design between quantization method and runtime support (Egiazarian et al., 27 Sep 2025). It analyzes microscaling FP4 through group size, element representation, and scale representation, then introduces Micro-Rotated-GPTQ with block-wise Hadamard transforms, MSE-optimized grids, and static activation re-ordering (Egiazarian et al., 27 Sep 2025). The accompanying QuTLASS library for NVIDIA Blackwell fuses rotations into the weights and computes activation rotations online with negligible overhead. Reported speedups versus FP16 reach up to 3.6x layer-wise and 2.2x end-to-end on B200, and 6x layer-wise and 4x end-to-end on RTX5090 (Egiazarian et al., 27 Sep 2025).

Adaptive block formats seek higher accuracy without abandoning existing tensor-core execution paths. MixFP4 keeps NVFP4’s block-scaled GEMM structure but allows each block to choose between $p$7 and $p$8, storing the type choice in the sign bit of the FP8 $p$9 block scale and decoding both payloads into a unified internal $b$0 compute representation (Zou et al., 29 May 2026). The reported tensor-core overhead is 3.1% area and 1.5% power (Zou et al., 29 May 2026). HiFloat4 pursues a different hardware point: by combining a 64-element group with a three-level scaling hierarchy, it enables a more fixed-point-like compute path and is summarized as requiring roughly one-third of NVFP4’s incremental area and about 10% lower power when integrated into existing 16-bit/8-bit dot-product units (Luo et al., 11 Feb 2026).

Other hardware-oriented proposals confirm the same pattern. AMXFP4 states that its asymmetric shared-scale MAC engine adds only about 10% hardware cost relative to MXFP4-FP8 while materially improving calibration-free inference quality (Lee et al., 2024). IF4 includes a dedicated MAC design for per-block FP4/INT4 selection and reports 4.7% higher latency than an NVFP4 MAC at the isolated MAC level, while arguing that the end-to-end impact is limited because memory movement dominates realistic accelerators (Cook et al., 30 Mar 2026).

6. Sensitivity analysis, adaptive remedies, and unresolved questions

FP4 error is not uniformly distributed within transformer architectures. A controlled inference study on Qwen2.5 models at 0.5B, 7B, and 14B shows that $b$1 and $b$2 are consistently the most sensitive components to FP4 quantization, $b$3 is moderately sensitive, and attention projections are substantially less sensitive (Cim et al., 5 Mar 2026). The same study rejects the idea that only final blocks matter: early blocks can also be highly sensitive, especially under MXFP4, and MXFP4 is generally more brittle than NVFP4 (Cim et al., 5 Mar 2026).

This non-uniformity helps explain why FP4 is not an automatic replacement for INT4. Comparative LLM quantization work found no consistent winner between INT4 and FP4 in 4-bit weight quantization; some layers favor FP4 and others INT4, which motivated layer-wise Mixture of Formats Quantization (MoFQ) (Zhang et al., 2023). MR-GPTQ sharpens the same point for microscaling formats: state-of-the-art PTQ methods struggle with FP4 because NVFP4’s small group size provably neutralizes traditional outlier mitigation techniques, while MXFP4’s power-of-two scale quantization severely degrades accuracy (Egiazarian et al., 27 Sep 2025). The conclusion in that study is explicit: FP4 is not an automatic upgrade over INT4 (Egiazarian et al., 27 Sep 2025).

Several adaptive remedies target the dominant FP4 error modes. Four Over Six (4/6) modifies NVFP4 scale selection by evaluating $b$4 and $b$5 for each block, on the observation that near-maximal values, rather than only classic outliers, are the main error source in block-scaled FP4 (Cook et al., 1 Dec 2025). The reported effects include preventing divergence in several training cases, improving multiple PTQ pipelines, and keeping overhead below 2% at inference-length sequences up to 16384 and below 15% at training-length sequences up to 131072 (Cook et al., 1 Dec 2025). IF4 generalizes this adaptivity by choosing between FP4 and scaled INT4 per 16-value block while reusing the NVFP4 scale field’s sign bit; in the reported W4A4 downstream evaluation, average accuracy is 72.8 for IF4 versus 72.4 for NVFP4 and 72.3 for 4/6, while average WikiText-2 perplexity across several Qwen3.5 sizes is 8.27 versus 8.37 and 8.31 (Cook et al., 30 Mar 2026).

Another branch of work adapts the activation path instead of only the weight format. SharQ combines FP4 with semi-structured activation sparsity by generating an input-adaptive N:M mask, routing the outlier-dominated sparse backbone through a sparse FP4 GEMM, and defining a dense residual relative to the quantized sparse backbone rather than the unquantized sparse values (Meng et al., 25 Jun 2026). In the reported evaluation it recovers 43–63% of the NVFP4-to-FP16 accuracy gap across language and vision-language tasks, delivers 2.2–2.4x lower latency than FP16, and improves throughput over FP8 by 1.2–1.4x on an RTX 5090 (Meng et al., 25 Jun 2026).

The contemporary literature therefore treats FP4 as a co-design problem rather than a single number format. Element encoding, block size, scale representation, asymmetric or adaptive scaling, rounding mode, outlier mitigation, optimizer behavior, and runtime kernels all materially affect whether FP4 behaves like a practical low-precision substrate or a numerically brittle compression scheme (Hu et al., 22 Sep 2025, Egiazarian et al., 27 Sep 2025). This suggests that future progress in FP4 will continue to depend less on nominal bitwidth alone than on the joint design of quantization method, training or calibration procedure, and accelerator datapath.