---
title: 'Four-Bit Floating Point (FP4): Formats & Advances'
url: https://www.emergentmind.com/topics/four-bit-floating-point-fp4
type: topic
---

# Four-Bit Floating Point (FP4): Formats & Advances

Four-bit floating point (FP4) denotes a class of 4-bit floating-point number systems used to reduce memory footprint, bandwidth, and arithmetic cost in large-model inference and training. In current literature, FP4 is not a single canonical datatype but a family of encodings and execution schemes that combine a 4-bit element representation with block-wise or hierarchical scaling, calibration or optimization rules, and hardware-specific kernels. It is the lowest precision in recent LLM quantization studies that still preserves essential floating-point structure such as sign and exponent, but its practical behavior is dominated by quantization error, scale quantization, outlier handling, and strong heterogeneity across layers, blocks, and tasks [2603.08747][2509.23202].

## 1. Numerical meaning and representational variants

In standard low-bit floating-point notation, FP4 uses one sign bit and three remaining bits split between exponent and mantissa. Recent post-training quantization work explicitly treats FP4 as a search space over formats such as \(E3M0\), \(E2M1\), and \(E1M2\), rather than as a single fixed layout [2310.16836]. A generic floating-point value is written as
$$
f = (-1)^{d_s} 2^{p-b}\left(1+\frac{d_1}{2}+\frac{d_2}{2^2}+\cdots+\frac{d_m}{2^m}\right),
$$
with sign bit \(d_s\), exponent value \(p\), bias \(b\), and \(m\)-bit mantissa [2503.15465].

The most common FP4 element format in LLM inference and training papers is \(E2M1\): one sign bit, two exponent bits, and one mantissa bit. In NVFP4-style quantization, the magnitude codebook for \(E2M1\) is \(\{0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0\}\) [2512.02010]. Other works emphasize that \(E1M2\) can be preferable for flatter or more uniform block statistics, while \(E3M0\) can be useful when dynamic range is prioritized over intra-scale precision [2503.15465][2605.31035].

This numerical flexibility is central to FP4’s behavior. In diffusion-transformer PTQ, FP4 is explicitly described as a family of 4-bit floating-point formats whose exponent/mantissa split trades dynamic range against local precision; when the mantissa takes all remaining bits, the format collapses to the uniform integer case, with \(E0M3\) identified as INT4 [2503.15465]. Comparative LLM work reaches a related conclusion from another direction: FP4 is not universally superior or inferior to INT4, and the preferred 4-bit format can vary across layers [2305.12356].

Some studies also modify the representable set itself. A redesigned FP4-\(E2M1\) for weight-only LLM quantization reallocates NaN/Inf code points to ordinary numeric values, extending the usable set to \(\pm 4\) and \(\pm 6\); that redesign yields about 35% lower quantization error than IEEE-aligned FP4-\(E2M1\) in the reported experiments [2305.12356].

## 2. Block scaling, microscaling, and format families

Raw FP4 elements are usually too limited to be deployed without external scaling. The dominant remedy is block floating-point or microscaling: values are partitioned into small groups, each group shares a scale, and only the normalized residual values are stored in FP4. A general microscaling representation is often described by group size \(G\), element representation \(E\), and scale representation \(S\) [2509.23202]. In related training analyses, a micro-scaled block is written as \(x_i = s \cdot p_i\), where \(s\) is the shared scale and \(p_i\) the low-bit element [2509.17791].

The operational advantage is that scale products can be factored outside local dot products. In block-scaled execution,
$$
\mathbf{A} \approx \mathbf{A}_q \odot \mathbf{s}_A,
$$
and for a block dot product,
$$
\sum_{i=0}^{V-1} (a_i s_A)(b_i s_B) = (s_A s_B)\sum_{i=0}^{V-1} a_i b_i,
$$
so scale application occurs once per block rather than once per element [2605.31035].

| Scheme | Block structure | Scale representation |
|---|---|---|
| MXFP4 | 32-element groups, FP4 \(E2M1\) elements | \(E8M0\) or shared 8-bit exponent |
| NVFP4 | 16-element groups, FP4 \(E2M1\) elements | \(E4M3\) scales with dynamic scaling |
| HiF4 | 64-element block with 32 bits metadata | Three-level scaling hierarchy |

This table captures the major families that recur across the literature. MXFP4 is defined as \((G=32, E=\text{FP4}, S=\text{E8M0})\), while NVFP4 is \((G=16, E=\text{FP4}, S=\text{E4M3})\); the corresponding average storage costs are 4.25 bits per element for MXFP4 and 4.5 bits per element for NVFP4 [2509.23202]. Another inference study describes MXFP4 more concretely as \(E2M1\) values in 32-element blocks with a shared 8-bit exponent, and NVFP4 as a 16-element block-scaled format with dynamic scaling, 4-bit scales, and a max-calibration algorithm [2603.08747].

HiFloat4 (HiF4) pushes the same idea further with a 64-element block and a three-level scaling hierarchy. In one formulation, each HiF4 unit packs 64 4-bit elements with 32 bits of shared scaling metadata, averaging 4.5 bits per value; the metadata consists of a global \(E6M2\)-like scale and lower-level micro-exponents that capture intra-group variation [2602.11287]. A related evaluation on Ascend NPUs describes HiF4 as a three-level hierarchical format contrasted with single-level MXFP4 and two-level NVFP4, and identifies hierarchical scaling as the key reason it remains robust in 4-bit regimes where integer quantization collapses [2602.12635].

Asymmetric and adaptive variants modify the same block-scaled substrate rather than replacing it. AMXFP4 retains 4-bit floating-point elements but replaces symmetric shared scales with asymmetric shared scales to model the group-wise asymmetry induced by microscaling [2411.09909]. MixFP4 keeps NVFP4’s scale hierarchy but allows each block to choose between \(E2M1\) and \(E1M2\) payloads [2605.31035]. IF4 goes further by choosing between FP4 and INT4 per block while still using the NVFP4-style \(E4M3\) scale field [2603.28765].

## 3. Post-training quantization and inference behavior

A substantial fraction of FP4 research concerns post-training quantization (PTQ) for LLM inference. One early line of work argues that floating-point quantization is more natural than uniform integer quantization for LLMs because activations are heavy-tailed and outlier-prone. In that setting, FP8 activations consistently outperform INT8, and FP4 weights are competitive with or superior to INT4 in W4A8 settings. On LLaMA-7b under W4A8, FP--FP achieves mean perplexity 15.14 versus 16.09 for INT--FP; on LLaMA-13b, FP--FP gets 11.08 versus 11.31 [2307.09782]. The same work constrains weight scales to powers of two to avoid expensive FP4-to-FP8 alignment overhead and integrates Low Rank Compensation (LoRC), which is described as especially beneficial in smaller models [2307.09782].

LLM-FP4 extends the PTQ perspective by searching jointly over FP4 format and clipping range. It identifies a recurring transformer activation pattern—high inter-channel variance and low intra-channel variance—and introduces per-channel activation quantization whose extra scaling factors are reparameterized as exponent biases of weights, incurring negligible cost [2310.16836]. In the reported LLaMA-13B 4/4/4 setting, this yields an average score of 63.1 on common-sense zero-shot reasoning tasks, only 5.8 below the full-precision model and 12.7 points above the previous state of the art cited in that paper [2310.16836].

FP4 PTQ has also been extended beyond LLMs. FP4DiT applies floating-point PTQ to diffusion transformers, combining mixed-format FP4 weight quantization, a scale-aware extension of AdaRound, and token-wise online activation quantization [2503.15465]. The method assigns \(E3M0\) to the first linear layer in each pointwise feed-forward block, uses \(E2M1\) for PixArt-\(\alpha\) and Hunyuan or \(E1M2\) for PixArt-\(\Sigma\) in the remaining weights, and quantizes activations online with \(E2M3\) for W4A6 or \(E3M4\) for W4A8 [2503.15465]. On PixArt-\(\alpha\), FP4DiT reports 27.43 HPSv2 and 0.3076 CLIP at W4A8, and 23.55 HPSv2 and 0.3031 CLIP at W4A6, outperforming the integer PTQ baselines listed in the study [2503.15465].

Calibration-free microscaling inference has developed along a separate track. AMXFP4 identifies a tradeoff in microscaling: decreasing group size suppresses activation outliers but increases group-wise asymmetry, and symmetric MXFP4 does not model that asymmetry well [2411.09909]. Its asymmetric shared-scale design outperforms MXFP4 by 3% on VQA and exceeds rotation-based methods by 1.6% on CSQA in the reported results [2411.09909].

## 4. FP4 training and pre-training

Training in FP4 is substantially more difficult than inference because weights, activations, and gradients all become low-precision objects. One FP4 training framework for LLMs uses \(E2M1\) FP4, a Differentiable Gradient Estimator (DGE) for weight updates, and Outlier Clamping and Compensation (OCC) for activations, together with mixed precision outside GEMMs and vector-wise quantization [2501.17116]. That work reports scaling to 1.3B, 7B, and 13B LLaMA-2-style models trained from scratch on 100B tokens. Final training losses are 2.55 versus 2.49 for 1.3B, 2.17 versus 2.07 for 7B, and 1.97 versus 1.88 for 13B when compared with BF16; average zero-shot scores are 53.13 versus 53.23, 54.42 versus 53.87, and 54.95 versus 54.44, respectively [2501.17116].

A later study demonstrates fully quantized training of LLMs using predominantly FP4 for weights, activations, and gradients on datasets up to 200 billion tokens [2505.19115]. Its central format choice is NVFP4: blocks of 16 \(E2M1\) values sharing an \(E4M3\) scale. The paper adopts round-to-nearest in the forward pass, stochastic rounding for backward and update passes, and derives a threshold according to which quantized training becomes less effective when the gradient norm falls below approximately \(\sqrt{3}\) times the quantization noise [2505.19115]. It then uses a late-stage quantization-aware finetuning phase in which the forward pass stays in FP4 while the backward pass switches to BF16. The reported large-scale demonstration trains a 7B Llama2-style model on 256 Intel Gaudi2 accelerators for about 200B tokens over roughly 6 days, and after the short QAF phase the downstream results are described as on par with BF16 [2505.19115].

The design space of FP4 training is itself now an explicit object of study. A unified microscaling framework analyzes the computational overhead of scale gradients, tensor scaling, stochastic rounding, Hadamard transforms, and differentiable approximations to quantization, then evaluates thousands of configurations across regression, classification, diffusion, and language tasks [2509.17791]. The configurations with the best reported performance-to-overhead trade-off combine Hadamard transformations, tensor scaling, and stochastic rounding, while \(UE5M3\) emerges as a potentially favorable scale format because it balances range and precision with manageable overhead [2509.17791].

Format choice also interacts strongly with accelerator architecture. On Ascend NPUs, HiFloat4 pre-training compares HiF4 against MXFP4 in dense and MoE models, with linear and expert GEMMs executed entirely in FP4 [2604.08826]. The reported FP4 compute fractions are 79.58% for OpenPangu-1B, 76.82% for Llama3-8B, and 95.90% for Qwen3-MoE-30B; the corresponding relative loss errors after 50B tokens are 1.19%, 0.85%, and 0.88% for HiF4, compared with 1.79%, 1.44%, and 1.55% for MXFP4 [2604.08826]. In the ablations, HiF4 mainly benefits from random Hadamard transforms and can degrade under stochastic rounding, whereas MXFP4 requires stochastic rounding, Hadamard transforms, and truncation-free scaling to approach acceptable behavior [2604.08826].

A bibliographic clarification is necessary in the FP4 training literature. The arXiv record titled "Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models" [2502.11458] does not contain an FP4 method, equations, experiments, or results in its body; the FP4-related material appears only in the bibliography.

## 5. Hardware architectures and systems support

FP4 has increasingly been treated as a hardware target rather than only a quantization abstraction. A dual-precision hybrid floating-point processing element supports FP8 (\(E4M3\), \(E5M2\)) and FP4 (\(2 \times E2M1\), \(2 \times E1M2\)) by reusing a single 4-bit multiplier as either one \(4 \times 4\) unit in FP8 mode or two parallel \(2 \times 2\) multipliers in FP4 mode [2604.04507]. The same work describes a 6-stage pipeline with early special-case detection, truncation-based alignment, CSA accumulation, leading-zero anticipation, normalization without explicit rounding, and output ReLU. Implemented in TSMC 28 nm, the PE reports 1.938 GHz, 0.00396 mm\(^2\), 2.1338 mW, 7.75 GFLOPS FP4 throughput, and 3632 GFLOPS/W FP4 energy efficiency [2604.04507].

At the GPU-kernel level, MR-GPTQ argues that practical FP4 inference depends on co-design between quantization method and runtime support [2509.23202]. It analyzes microscaling FP4 through group size, element representation, and scale representation, then introduces Micro-Rotated-GPTQ with block-wise Hadamard transforms, MSE-optimized grids, and static activation re-ordering [2509.23202]. The accompanying QuTLASS library for NVIDIA Blackwell fuses rotations into the weights and computes activation rotations online with negligible overhead. Reported speedups versus FP16 reach up to 3.6x layer-wise and 2.2x end-to-end on B200, and 6x layer-wise and 4x end-to-end on RTX5090 [2509.23202].

Adaptive block formats seek higher accuracy without abandoning existing tensor-core execution paths. MixFP4 keeps NVFP4’s block-scaled GEMM structure but allows each block to choose between \(E2M1\) and \(E1M2\), storing the type choice in the sign bit of the FP8 \(E4M3\) block scale and decoding both payloads into a unified internal \(E2M2\) compute representation [2605.31035]. The reported tensor-core overhead is 3.1% area and 1.5% power [2605.31035]. HiFloat4 pursues a different hardware point: by combining a 64-element group with a three-level scaling hierarchy, it enables a more fixed-point-like compute path and is summarized as requiring roughly one-third of NVFP4’s incremental area and about 10% lower power when integrated into existing 16-bit/8-bit dot-product units [2602.11287].

Other hardware-oriented proposals confirm the same pattern. AMXFP4 states that its asymmetric shared-scale MAC engine adds only about 10% hardware cost relative to MXFP4-FP8 while materially improving calibration-free inference quality [2411.09909]. IF4 includes a dedicated MAC design for per-block FP4/INT4 selection and reports 4.7% higher latency than an NVFP4 MAC at the isolated MAC level, while arguing that the end-to-end impact is limited because memory movement dominates realistic accelerators [2603.28765].

## 6. Sensitivity analysis, adaptive remedies, and unresolved questions

FP4 error is not uniformly distributed within transformer architectures. A controlled inference study on Qwen2.5 models at 0.5B, 7B, and 14B shows that \(up\_proj\) and \(down\_proj\) are consistently the most sensitive components to FP4 quantization, \(gate\_proj\) is moderately sensitive, and attention projections are substantially less sensitive [2603.08747]. The same study rejects the idea that only final blocks matter: early blocks can also be highly sensitive, especially under MXFP4, and MXFP4 is generally more brittle than NVFP4 [2603.08747].

This non-uniformity helps explain why FP4 is not an automatic replacement for INT4. Comparative LLM quantization work found no consistent winner between INT4 and FP4 in 4-bit weight quantization; some layers favor FP4 and others INT4, which motivated layer-wise Mixture of Formats Quantization (MoFQ) [2305.12356]. MR-GPTQ sharpens the same point for microscaling formats: state-of-the-art PTQ methods struggle with FP4 because NVFP4’s small group size provably neutralizes traditional outlier mitigation techniques, while MXFP4’s power-of-two scale quantization severely degrades accuracy [2509.23202]. The conclusion in that study is explicit: FP4 is not an automatic upgrade over INT4 [2509.23202].

Several adaptive remedies target the dominant FP4 error modes. Four Over Six (4/6) modifies NVFP4 scale selection by evaluating \(\max(|X|)/6\) and \(\max(|X|)/4\) for each block, on the observation that near-maximal values, rather than only classic outliers, are the main error source in block-scaled FP4 [2512.02010]. The reported effects include preventing divergence in several training cases, improving multiple PTQ pipelines, and keeping overhead below 2% at inference-length sequences up to 16384 and below 15% at training-length sequences up to 131072 [2512.02010]. IF4 generalizes this adaptivity by choosing between FP4 and scaled INT4 per 16-value block while reusing the NVFP4 scale field’s sign bit; in the reported W4A4 downstream evaluation, average accuracy is 72.8 for IF4 versus 72.4 for NVFP4 and 72.3 for 4/6, while average WikiText-2 perplexity across several Qwen3.5 sizes is 8.27 versus 8.37 and 8.31 [2603.28765].

Another branch of work adapts the activation path instead of only the weight format. SharQ combines FP4 with semi-structured activation sparsity by generating an input-adaptive N:M mask, routing the outlier-dominated sparse backbone through a sparse FP4 GEMM, and defining a dense residual relative to the quantized sparse backbone rather than the unquantized sparse values [2606.26587]. In the reported evaluation it recovers 43–63% of the NVFP4-to-FP16 accuracy gap across language and vision-language tasks, delivers 2.2–2.4x lower latency than FP16, and improves throughput over FP8 by 1.2–1.4x on an RTX 5090 [2606.26587].

The contemporary literature therefore treats FP4 as a co-design problem rather than a single number format. Element encoding, block size, scale representation, asymmetric or adaptive scaling, rounding mode, outlier mitigation, optimizer behavior, and runtime kernels all materially affect whether FP4 behaves like a practical low-precision substrate or a numerically brittle compression scheme [2509.17791][2509.23202]. This suggests that future progress in FP4 will continue to depend less on nominal bitwidth alone than on the joint design of quantization method, training or calibration procedure, and accelerator datapath.

Source: https://www.emergentmind.com/topics/four-bit-floating-point-fp4