---
title: MXFP4 Training Methods
url: https://www.emergentmind.com/topics/mxfp4-training
type: topic
---

# MXFP4 Training Methods

Microscaling FP4 (MXFP4) training refers to training neural networks—especially large language and vision models—using a 4-bit floating-point format (E2M1) combined with efficient group-wise scaling to achieve aggressive reductions in computational cost and bandwidth without significant loss in accuracy. The technique is anchored in both hardware support (e.g., NVIDIA Blackwell GPUs, FPGAs) and recent algorithmic advances that address the unique quantization noise and precision challenges of sub-8-bit floating-point computation.

## 1. MXFP4 Format and Quantization Scheme

MXFP4 is a 4-bit floating-point format based on a standard E2M1 layout (1 sign, 2 exponent, 1 mantissa bit), with an explicit block-level scaling factor applied to each group of elements:

- **Element encoding:**  
  - Each value: 1 sign bit, 2 exponent bits (bias=1), 1 mantissa bit  
  - Representable values: {–6, –4, –3, –2, –1.5, –1, –0.5, 0, 0.5, 1, 1.5, 2, 3, 4, 6}

- **Microscaling scheme:**  
  - Contiguous blocks of G = 16 (NVFP4, [2505.19115]) or G = 32 (MXFP4, [2502.20586, 2509.23202]) share a single floating-point scale (usually 8-bit, e.g., E8M0 or E4M3).
  - Quantization: each real tensor block $x_i$ is divided by the block scale $s$ and mapped to the nearest representable FP4 value.
  - Dequantization: $y_i = s \times q_i$, with $q_i$ the E2M1 code.
  - Scale selection: generally $s = \max_i |x_i| / M_{fp}$, where $M_{fp}$ is the largest representable FP4 value ([2505.19115], [2407.01475]).

- **Block-wise scaling** enables the dynamic range of the overall format to be much wider than single E2M1 (from 12× span for base FP4 to $>10^3$ span when group scaling is applied).

## 2. Training Algorithms and Stochastic Rounding

A key challenge in using MXFP4 for training is accurate, unbiased propagation of gradient updates despite aggressive quantization noise.

- **Stochastic rounding (SR):**  
  - For gradients, forward pass, or backward pass, stochastic rounding involves mapping each real value to either of its nearest quantized neighbors with a probability proportional to proximity.
  - For a value $x$ and grid distance $\Delta$:
    $$
    SR(x) = 
    \begin{cases}
    \Delta \lfloor x/\Delta \rfloor, & \text{with probability } (\lceil x/\Delta \rceil - x/\Delta) \\
    \Delta \lceil x/\Delta \rceil, & \text{with probability } (x/\Delta - \lfloor x/\Delta \rfloor)
    \end{cases}
    $$
  - Ensures $E[SR(x)] = x$. Crucial for unbiased SGD/Adam updates ([2502.20586], [2505.14669], [2505.19115]).

- **Variance reduction:**
  - Block-wise quantization can cause high error variance due to outlier-induced scale inflation.  
  - **Random Hadamard Transform (RHT):**
    - Preprocess group blocks with Hadamard mixing: for block $x$, $x' = H_g S x$ (Hadamard + sign) so that outliers are mixed across all elements, lowering $\|\cdot\|_\infty$ exposure and greatly reducing quantization error variance.
    - This transformation is proven to lower variance from $O(b \Delta^4 \|A\|_\infty \|B\|_\infty)$ to $O(\Delta^4 \|A\|_2 \|B\|_2 \log(2b/\epsilon))$ ([2502.20586], [2509.23202], [2505.14669]).
    - Empirically, RHT is essential for near-lossless LLM training in MXFP4.

- **Rounding schedule:**
  - Forward pass: Round-to-nearest (deterministic) for weights and activations
  - Backward and parameter update: Stochastic rounding for gradients ([2505.19115], [2505.14669], [2502.20586]).

## 3. Architectural and Algorithmic Enhancements

The literature identifies both architectural bottlenecks and algorithmic remedies that make MXFP4 training feasible across applications.

| Scheme / Format | Block Size | Scale Format | Rounding fwd/bwd | Extra transform          | Notes                   |
|-----------------|------------|--------------|------------------|-------------------------|-------------------------|
| MXFP4 ([2502.20586], [2505.14669]) | 32         | E8M0/E4M3       | RTN/SR                 | Hadamard (SR + RHT)     | LLM, Vision, QAT        |
| NVFP4 ([2505.19115])                 | 16         | E4M3            | RTN/SR                 | n/a                     | LLM, QAT                |

Block size is chosen according to hardware and application: $k=32$ on Blackwell/NVIDIA for MXFP4 ([2502.20586], [2509.23202]), $k=16$ for NVFP4 ([2505.19115]). FPGA implementations ([2407.01475]) also use $k=32$ (with some improvement seen at $k=16$).

- **Forward/Backward block shape symmetry:** To ensure correct group scaling for different GEMM layouts, modern methods (e.g., TetraJet [2502.20853], Quartet [2505.14669]) double-quantize inputs with both 1x32 and 32x1 block layouts.
- **Oscillation reduction (Vision):** "TetraJet" [2502.20853] introduces Q-EMA (Exponential Moving Average quantizer) and Q-Ramping (adaptive optimizer) to suppress weight flipping near block grid thresholds, closing >50% of the accuracy gap in vision transformers.
- **Unbiased STE:** Consistent application of double quantization plus unbiased rounding across forward and backward ensures SGD/Adam convergence under standard theory ([2502.20853]).

## 4. Empirical Results and Scaling Laws

MXFP4 training, enabled by efficient rounding and block-wise mixing, achieves near-baseline quality across modalities with substantial computational gains.

- **Language models (LLMs):**
  - [2502.20586] demonstrates GPT-style models up to 6.7B parameters trained on hundreds of billions of tokens using BF16 in the forward and MXFP4+SR+RHT in the backward pass: validation perplexity within 0.01–0.02 of full-precision BF16.
  - [2505.14669] ("Quartet") presents end-to-end MXFP4 training that closes the loss gap to FP16/FP8, with scaling law analysis. MXFP4 enables 1.8–2.6× end-to-end training speedup over FP8/BF16.
  - [2505.19115] ("FP4 All the Way") shows fully MXFP4-analogous (NVFP4) training of a 7B LLM on 256 Gaudi2s, with mixed RTN/SR, matches or beats BF16 downstream accuracy (e.g., Lambada, HellaSwag, Winogrande, etc.)

- **Vision:**
  - [2502.20853] achieves ImageNet top-1 accuracy within ≲1–2% of the full-precision baseline for DeiT/Swin transformer models trained for 90 epochs, using Q-EMA/Q-Ramping for oscillation reduction.
  - [2407.01475] demonstrates ResNet-18 on ImageNet matched to within 2.6% of FP32 using QAT with MXFP4 and a Brevitas extension.

- **Quantization-aware training (QAT) vs post-training quantization (PTQ):**
  - QAT recovers the majority of accuracy lost, whereas naive PTQ in pure MXFP4 format results in >10% performance drop ([2407.01475]).

- **Scaling laws:**
  - Quartet's low-precision scaling law ([2505.14669]) quantitatively relates model performance to forward/backward bitwidths, parameter/data count, and empirically fit "parameter efficiency" and "data efficiency." It predicts substantial compute-vs-accuracy improvements at 4b.

## 5. Post-Training Quantization and Model Compression

MXFP4 is effective for both QAT and high-throughput PTQ workflows:

- **PTQ for LLMs:**
  - [2509.23202] introduces MR-GPTQ (Micro-Rotated GPTQ), combining block-wise Hadamard transforms, GPTQ error-compensation, and static act-ordering. It boosts MXFP4 inference to 3.6× FP16 per-layer and 2.2× end-to-end speedups, achieving ~93–99% of FP16 accuracy in large LLMs.
  - Pre-quantization with optimized channel scaling and low-rank branches (GPTQ + Low-Rank) is highly effective—rotation further helps in INT4 but is less critical for MXFP4 due to the E2M1 grid's inherent dynamic range ([2507.17417], [2509.23202]).

| Method                | INT4 PPL | MXFP4 PPL |
|---------------------- |----------|-----------|
| RTN                   | 923.7    | 15.2      |
| +GPTQ                 | 1007.3   | 13.2      |
| +Low-Rank             | 723.5    | 15.6      |
| +GPTQ+Low-Rank        | 578.6    | 12.7      |
| Rot+Scale+GPTQ        | 11.73    | 12.29     |

*Table: Low perplexity (PPL) values for MXFP4 approaches INT4 when using optimized scaling and error-mitigation ([2507.17417]).*

- **FPGA implementations:**
  - Open-source designs ([2407.01475]) support all OCP-MX formats and MXFP4-specific arithmetic, with block size $k=32$ preferred for area/power efficiency.
  - Typical end-to-end flows: PyTorch+QAT → Brevitas+MX quantization → ONNX export → Vivado HLS → hardware deployment, achieving 67.2% ImageNet top-1 for ResNet-18 at much lower LUT/energy cost than INT4 GPU.

## 6. Practical Deployment and Best Practices

Deployment of MXFP4 for both training and inference is now supported across major hardware and software stacks, with the following considerations:

- **Hardware**
  - NVIDIA Blackwell (SM100/SM120) Tensor Cores natively accelerate MXFP4 GEMMs with block-wise scaling [2505.14669], [2502.20586].
  - AMD, Intel Gaudi2 accelerators, and FPGAs (OCP MX IP) also support MXFP4 or near-variants [2407.01475], [2505.19115].

- **Recommended block sizes:**  
  - $k=32$ for MXFP4 on Blackwell/NVIDIA, FPGAs.  
  - $k=16$ for NVFP4 (smaller group granularity).
- **Quantization recipes**  
  - Use per-block RMSE-optimal scaling ($\alpha^*$), with deterministic clipping in forward, unbiased stochastic rounding in backward.
  - Always bundle Hadamard transforms in the kernel for stochastic rounding variance reduction.
  - For oscillation: in vision, supplement with Q-EMA and/or Q-Ramping if weight flipping is evident [2502.20853].
- **Model initialization and finetuning:**  
  - Start from accurate FP32/BF16 weights, then QAT for stability; QAF (Quantization-Aware Finetuning in higher precision) can close remaining accuracy gap as gradient noise becomes a limiting factor [2505.19115].
- **Monitoring quantization effectiveness:**  
  - Use the “$\sqrt{3}\,\sigma$” rule ([2505.19115]): when $\|g\| \approx \sqrt{3}\,\sigma_q$ (gradient norm to quantization noise ratio), quantized updates are no longer useful; switch to higher precision for updates or finetune.

## 7. Limitations and Future Directions

- **Scale quantization and MSE:**  
  - Power-of-two scale quantization, as in E8M0, introduces scale flipping effects (up to ±50% relative error at block extremes); block-wise Hadamard and per-block scale optimization are essential [2509.23202].
  - Current approaches require careful block size selection; smaller blocks improve accuracy but increase overhead.

- **Oscillation phenomena:**  
  - Long-term stability of MXFP4-trained weights may suffer from flipping near quantization thresholds; advanced mitigations (Q-EMA, Q-Ramping) are required for vision but less studied in LLMs.

- **Numerical range and outliers:**  
  - While E2M1 provides nonuniform resolution and dynamic range, outliers can still degrade accuracy—rotation and block-wise transformations can alleviate but not fully resolve for extreme cases.

- **Hardware support:**  
  - MXFP4 is fully supported only on recent hardware (Blackwell, select FPGAs, Gaudi2 with emulation); portability to legacy devices is pending.

- **Open research avenues:**  
  - Unified architectures for MXFP4 kernels (GEMM+Hadamard+quantize in fused hardware) are under active development [2502.20586].
  - Application to non-GEMM operations (softmax, layer norm) may require new variance-reducing transforms [2502.20586].
  - Further research is ongoing on optimal block sizes, dynamic scaling logic, and extensions to convolutional and graph architectures.

## References

- [2407.01475] Exploring FPGA designs for MX and beyond  
- [2502.20586] Training LLMs with MXFP4  
- [2502.20853] Oscillation-Reduced MXFP4 Training for Vision Transformers  
- [2505.14669] Quartet: Native FP4 Training Can Be Optimal for Large Language Models  
- [2505.19115] FP4 All the Way: Fully Quantized Training of LLMs  
- [2507.17417] A Comprehensive Evaluation on Quantization Techniques for Large Language Models  
- [2509.23202] Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization

Source: https://www.emergentmind.com/topics/mxfp4-training