---
title: 'NVFP4: 4-bit Lower-Precision Format'
url: https://www.emergentmind.com/topics/nvfp4-lower-precision-format
type: topic
---

# NVFP4: 4-bit Lower-Precision Format

NVFP4 Lower-Precision Format

NVFP4 is a hardware-accelerated, block-microscaled 4-bit floating point data format used to enable efficient, highly compressed storage and computation for large-scale deep neural networks, particularly Large Language Models (LLMs). Backed by native support on NVIDIA Blackwell Tensor Cores, NVFP4 combines aggressive element-level quantization (E2M1 floating-point core, 4 bits/element) with local per-block floating-point scaling (E4M3, 8 bits per 16-element block) and an optional global floating-point scale, yielding favorable dynamic range, robust numerical stability, and practical accuracy in both training and inference settings [2505.19115, 2509.25149, 2512.02010, 2510.27527, 2601.07475, 2601.22813, 2601.20088].

## 1. Format Definition and Bit-Level Structure

NVFP4 uses a “microscaled” mini-float architecture designed for dense hardware efficiency and high representational fidelity under stringent memory constraints. Each tensor is partitioned into blocks of 16 elements, each encoded as follows:

- **FP4 Core (E2M1):**
  - 1 sign bit, 2 exponent bits (bias = 1), 1 mantissa bit
  - Representable set: $x \in \{\pm 0,\, \pm 0.5,\, \pm 1,\, \pm 1.5,\, \pm 2,\, \pm 3,\, \pm 4,\, \pm 6\}$
  - Value decoding: $x = (-1)^s \, 2^{e-1} (1 + m\cdot 2^{-1})$ for normalized $e > 0$
- **Block Scale (E4M3 FP8):**
  - Shared for 16 elements
  - 1 sign bit (usually zero), 4-bit exponent (bias = 7), 3-bit mantissa
  - Decodes as $s = 2^{E-7} (1 + M/8)$; supports scale $\approx$ [1/128, 448]
- **Optional global tensor scale:** Full-precision (FP32)
- **Block packing:** Each 16-element block uses 64 bits (4b × 16) + 8 bits (scale) = 72 bits, or 4.5 bits/element.

This structure ensures that every block achieves local dynamic-range adaptation without the coarseness or loss typical of power-of-two only scaling (as in MXFP4) [2505.19115, 2509.25149, 2504.14152, 2509.23202, 2512.02010].

## 2. Quantization and Dequantization Algorithms

### a. Standard NVFP4 Quantization

Given a real-valued tensor $X$:

1. **Global scale (optional):** $S = \mathrm{max}_i|X_i|/(M_{\rm FP4} M_{\rm FP8})$, where $M_{\rm FP4}=6$, $M_{\rm FP8}\approx 448$.
2. **Block partition:** Split $X$ into blocks of 16 elements.
3. **Per-block scale:** For block $b$, $s_b = \mathrm{max}_{i\in b}|X_i|/M_{FP4}$, quantized to E4M3.
4. **Block-wise quantization:** For each $x_i$ in block $b$:
   \[
   r_i = \frac{x_i}{S\cdot s_b}
   \]
   Round $r_i$ to nearest FP4 value (or use stochastic rounding for gradients), store as 4-bit code $q_i$.
5. **Dequantization:** At GEMM time, recover $x_i \approx q_i \cdot s_b \cdot S$.

The per-block scaling limits the impact of local outliers and avoids the quantization coarseness of larger groupings, with empirical results favoring group size 16 [2505.19115, 2509.25149].

### b. Adaptive Quantization and Error Mitigation

- **"Four Over Six" (4/6) Block Scaling:** Rather than always scaling to FP4's maximal representable value (6), blocks also consider scaling to 4, selecting the regime with minimal per-block mean squared error (MSE). This adaptive scheme reduces quantization error for near-maximal values, crucial for LLMs where value distributions can be highly nonuniform [2512.02010, 2601.22813].
- **Double-block scaling:** Used in TetraJet-v2, combines a large “outer” block scale with conventional inner microscale for further dynamic range refinement [2510.27527].
- **Hadamard/Rotation-based error spreading:** Random Hadamard transforms (RHT) decorrelate outliers for enhanced quantization uniformity in blocks, particularly for gradient tensors [2509.25149, 2510.27527, 2601.22813].

## 3. Rounding Schemes and Noise Analysis

NVFP4 deployments employ a hybrid of deterministic and unbiased quantization:

- **Forward pass (inference and training):** Deterministic round-to-nearest (RtN) for weights and activations. This minimizes quantization variance and is stable under repeated dot-product accumulation.
- **Backward and update GEMMs (training):** Stochastic rounding (SR) applied to gradients and update-activation tensors. SR ensures each quantized value is an unbiased estimator of the source, crucial for unbiased SGD [2505.19115, 2509.25149, 2510.27527].

Quantization noise in SR is modeled as zero-mean, with variance $\sigma_q^2 \approx \Delta^2/12$ (per-block, due to block scale). Theoretical analysis demonstrates effective training in NVFP4 is possible as long as gradient norms remain above $\sqrt{3}\,\sigma_q/\sqrt{d}$; when this threshold is crossed, switching to BF16 or FP8 is recommended [2505.19115].

Recent advances (MS-EDEN, Quartet II [2601.22813]) achieve unbiased quantization with roughly half the mean squared error of elementwise SR, by applying group-scale corrections post-RHT and re-quantizing scales using stochastic rounding.

## 4. Empirical Performance and Model Training

**Training Stability and Accuracy:**

- NVFP4 enables full model training with GEMMs in 4-bit precision up to multi-billion parameter scales and extreme token counts (200B tokens and beyond) [2505.19115, 2509.25149].
- Initial loss gaps to BF16 are small (≈0.05–1.5% loss units); extended quantization-aware fine-tuning or combination with Four Over Six scaling closes the gap.
- Downstream task performance (MMLU, code, reading tasks) is nearly identical to BF16 or FP8, with empirical accuracy within 1–2% for most tasks [2505.19115, 2512.02010, 2601.07475].

**Efficiency:**

- NVFP4 provides 2–6× arithmetic throughput versus BF16/FP16 and up to 50% memory reduction for weights and optimizer states. Measured speedups span 2–4.2× in full-model benchmarks [2509.25149, 2601.22813, 2601.07475].
- Design tradeoffs: blocks smaller than 16 elements yield limited further accuracy gain; larger blocks or coarser (e.g., power-of-two) scales (MXFP4) sharply degrade accuracy for heavy-tailed or block-outlier distributions [2504.14152, 2509.23202].

**Advanced Techniques:**
- Selective retention of high-precision layers (e.g., early/final transform blocks in BF16) can further stabilize very deep or hybrid models with minimal capacity cost [2509.25149, 2512.02010].
- NVFP4 is compatible with both fine-grained mixed-precision (FGMP) inference schemes, allowing critical channels or blocks to be promoted to FP8 as needed [2504.14152].

## 5. Post-Training Quantization, Distillation, and Residual Compensation

**Post-Training Quantizers and Compensation:**

- NVFP4 supports classic PTQ (e.g., GPTQ, AWQ, SmoothQuant), but unique error profiles (notably, increased rounding error for near-maximal values) motivate new algorithms:
    - 4/6 adaptive block scaling [2512.02010, 2601.22813] directly mitigates large-value error, improving perplexity and downstream accuracy in both pre-training and PTQ.
    - ARCQuant [2601.07475] introduces Augmented Residual Channels: outlier dimensions are detected and encoded in additional residual “channels” (also NVFP4), enabling error-compensated GEMM computation with minimal latency increase.
    - MR-GPTQ [2509.23202] fuses small-block Hadamard transformations and blocked grid-search for scale optimization, reducing quantization-induced MSE.

**Distillation for Accuracy Recovery:**

- Quantization-aware distillation (QAD) [2601.20088] recovers or surpasses strong BF16 baselines by training the quantized model to match a reference full-precision model’s output distribution (via KL divergence loss), robustly bridging the small observed gaps left by even aggressive PTQ (including on multi-stage SFT+RL pipelines and vision-language models).

## 6. Hardware and Implementation Considerations

NVFP4 is intimately designed around the requirements and capabilities of modern NVIDIA architectures (Blackwell GPUs and beyond):

- Blackwell Tensor Cores natively implement 4-bit × 4-bit GEMMs using NVFP4 with 16-element blocks and full FP8 scaling, fused with scale computation, layout expansion, and stochastic rounding in hardware [2509.25149, 2505.19115, 2601.22813].
- Integration into software stacks (Transformer Engine, QuTLASS, etc.) provides seamless PyTorch training and inference support, fused quantize/dequantize kernels, and optional flags for advanced rounding or residual techniques [2509.25149, 2509.23202, 2601.07475].
- Hardware datapath extensions (e.g., VMAC-based PE arrays for FGMP) further minimize area and power for fine-grained mixed-precision routing [2504.14152].

Empirical kernel benchmarks demonstrate that NVFP4-centric pipelines are memory-bound rather than compute-bound and yield end-to-end speedups of 2–4× compared to FP16/BF16 baselines.

## 7. Comparative Analysis, Limitations, and Future Directions

### a. Comparative Summary

| Format   | Elem Bits | Block Scale | Block Size | Util. Range | Avg Accuracy vs FP16 (%) | Speedup (SM) |
|----------|-----------|-------------|------------|-------------|-------------------------|--------------|
| MXFP4    | 4         | E8M0        | 32         | ~[–3,3]     | 90–95                   | 4–7×         |
| NVFP4    | 4         | E4M3        | 16         | ~[–6,6]     | 96–99                   | 3–6×         |
| INT4+FP16| 4         | FP16        | 32         | [-8,7]      | 92–95                   | 2–3×         |

*: Empirical accuracy varies by task/model size; speedup is relative to BF16/FP16 [2509.23202, 2509.25149, 2601.07475].

### b. Limitations and Open Challenges

- NVFP4 is not universally lossless: extremely long token contexts or models with unbounded dynamic range may experience modest accuracy drops (<2%), particularly without 4/6 scaling or residual compensation [2512.02010, 2601.07475, 2601.22813].
- The tradeoff between block size (hardware efficiency vs. representational error) is saturated at 16 elements; smaller blocks add storage overhead, larger degrade small-value quantization.
- Extension to all network layers (e.g., attention, embedding, output heads) remains constrained by stability and hardware uniformity restrictions [2509.25149].
- While MR-GPTQ and QAD close much of the quantization gap, some model architectures (notably RL-fine-tuned LLMs) may be brittle to conventional QAT/QAF methods, with QAD currently recognized as most robust [2601.20088].

### c. Research Directions

- Elimination or minimization of residual higher-precision layers
- Generalization of 2D scaling and Hadamard techniques to attention and communication subgraphs
- Exploration of sub-4-bit formats (e.g., integer-only 2-bit), contingent on task and distribution
- Systematic, automated block size and scale tuning
- Integration of advanced unbiased quantizers (e.g., MS-EDEN) [2601.22813]
- Scaling of ARCQuant-type residual schemes to extremely large dimensions

**In summary, NVFP4 stands as a rigorously defined, hardware-optimized, and empirically validated solution for sub-8-bit floating-point quantization, offering near-full-precision performance across LLM training, PTQ, and deployment at unprecedented computational and energy efficiency.**

Source: https://www.emergentmind.com/topics/nvfp4-lower-precision-format