---
title: Blockwise Quantization & Microscaling
url: https://www.emergentmind.com/topics/blockwise-quantization-and-microscaling
type: topic
---

# Blockwise Quantization & Microscaling

Blockwise Quantization and Microscaling

Blockwise quantization and microscaling define a family of data representations and algorithms for compressing neural network weights, activations, and gradients through partitioning tensors into blocks and sharing a scale per block. Each element within a block is quantized, typically using low-bit-width integer or floating-point representations, while the block’s dynamic range is captured by an associated shared scale. Microscaling (MX), a central concept, denotes configurations where block sizes are small—often 16–32 elements—enabling fine-grained adaptation to local distributional statistics. These approaches have emerged as a critical technology in large-scale AI model deployment and training, driven by hardware advances and the need for memory and bandwidth-efficient inference and training on large language models (LLMs) [2310.10537][2601.19026][2405.07135][2409.05902][2601.19213][2506.20752][2411.09909][2509.23202][2512.00956][2502.05376].

## 1. Formal Definition and Data Formats

Fundamentally, blockwise quantization partitions a tensor \( X \in \mathbb{R}^{N_1 \times N_2 \times \ldots} \) into non-overlapping blocks of size \( k \). Within each block \( b \), the maximum absolute value is used to compute a shared scale:

\[
s_b = \frac{\max_{i=1}^k |x_i|}{M_{\max}}
\]

where \( M_{\max} \) is the largest magnitude in the chosen element format. The per-block quantization maps each \( x_i \) to a quantized code \( q_i \):

\[
q_i = \mathrm{clip}\Bigl(\mathrm{round}(x_i/s_b),\,q_{\min},\,q_{\max}\Bigr)
\]

and dequantization reconstructs \( \hat{x}_i = s_b q_i \).

MX formats typically integrate:

- **Scale storage:** 8-bit shared (e.g., E8M0, E4M3, E5M2, or power-of-two exponents).
- **Block size:** commonly 16 or 32, aligned with hardware (e.g., NVIDIA Blackwell).
- **Element representation:** Signed INT4/6/8, FP4 (E2M1), FP6 (E2M3 / E3M2), FP8 (E4M3 / E5M2), with trade-offs between dynamic range and quantization error.

Several derivative formats augment base MX schemes, including asymmetric scaling (AMXFP4 [2411.09909]), hybrid codebooks with block clustering (LO-BCQ [2502.05376]), and metadata-augmented encoding (M²XFP [2601.19213]).

## 2. Algorithmic Workflows for Inference and Training

The practical use of blockwise quantization/microscaling involves multi-stage pipelines:

**Conversion and Inference Flows** [2310.10537][2405.07135]:

- **Direct-cast inference:** All weights/activations are quantized on-the-fly with per-block scales. GEMM kernels operate directly on (scale, code) tuples, accumulating outputs in FP16/32. Non-dot product ops execute in higher precision.
- **Error-diffusion PTQ:** Scale and quantization error are calibrated on a small dataset, propagating residuals across blocks.
- **Quant-aware finetuning:** float2mx quantization is inserted into forward passes during finetuning; backward is done in FP32.
- **Training flow:** FP32 master weights are held; all GEMMs quantize weights and activations; dot-products accumulate in FP32; gradients are quantized before the next operation; optimizer and learning rate remain unchanged.

**Extensions:**

- **OPAL's Outlier-Preserved Quantization:** Top-n outliers in each block are stored in higher precision [2409.05902].
- **Mixed-precision layers:** More sensitive layers may employ higher bit-widths, and robust layers may use lower [2409.05902].
- **Rotation and transform preconditioning:** Use blockwise Hadamard or WUSH transforms prior to quantization to redistribute outlier effects and minimize quantum error [2512.00956][2509.23202].
- **Block clustering with codebooks:** Assign blocks to clusters with custom codebooks (BCQ, LO-BCQ), updating assignments and codebooks to minimize total quantization error [2502.05376].

**Pseudocode for float2mx (basic microscaling quantization) [2310.10537]:**

```python
def float2mx(V, element_format):
    emin_elem = exponent_min(element_format)
    emax_elem = exponent_max(element_format)
    e_shared = floor(log2(max(abs(V)))) - emax_elem
    s_b = 2**e_shared
    q = []
    for v in V:
        y = v / s_b
        q.append(quantize_to_element_format(y))
    return s_b, q
```

## 3. Accuracy, Compression, and Error Characteristics

Empirical results show distinct trade-offs among bit-width, block size, and quantization format. Representative results [2310.10537][2405.07135][2509.23202][2411.09909][2601.19213][2512.00956][2502.05376]:

- **MXINT8 and MXFP6** achieve sub-0.5% top-1 accuracy drop on ImageNet and negligible BLEU/WER/AUC drops on translation, speech, and recommendation.
- **Generative Inference:** MXINT8 and MXFP6 match FP32 on GPT3-175B and LLaMA-7B within statistical error; MXFP6's drop in accuracy is < 0.01 absolute.
- **Sub-8-bit training:** Mixed MXFP4/MXFP6 or pure MXFP6 achieve <0.5% loss increase on generative LMs up to 1.5B parameters.
- **FP4 (MXFP4, NVFP4):** With naive power-of-two scales, quantization incurs significant error: e.g., MXFP4-PoT PPL ≈10.1 vs. FP16 baseline ≈6.0; MR-GPTQ or asymmetric scaling closes the PPL gap to ≈0.5 [2411.09909][2509.23202].
- **Block size:** Smaller blocks typically yield lower error due to tighter scaling, up to a threshold—at very small block sizes, scale quantization granularity or distributional effects can increase error (see Section 4).
- **Outlier preservation:** Directly storing a handful of outliers in BF16 within each block can reduce overhead to <1% accuracy and moderate the (otherwise severe) impact of extreme elements [2409.05902].

A summary table:

| Format           | Block Size | Reported Accuracy Loss | Notable Method                     | Source         |
|------------------|------------|----------------------|-------------------------------------|---------------|
| MXINT8/MXFP6     | 32         | <0.5% top-1          | Direct-cast, PTQ, QAT               | [2310.10537]  |
| MXFP4-PoT        | 32         | ~5%–50% task dep.     | Naive, no calibration               | [2310.10537]  |
| AMXFP4-FP8       | 32         | <0.5 PPL, +3% task    | Asymmetric, FP8 scale               | [2411.09909]  |
| MR-GPTQ (FP4)    | 16/32      | ~1–2%                | Rotated, GPTQ optimized             | [2509.23202]  |
| M²XFP            | 32         | ~1.6% (LLaMA 7B/8B)   | Metadata-augmented                  | [2601.19213]  |
| LO-BCQ           | 8/32/64    | <0.2 PPL              | Block-clustered codebooks           | [2502.05376]  |

## 4. Failure Modes, Anomalies, and Theoretical Limits

### 4.1 Scale Quantization Anomalies

Empirical and theoretical analysis reveal that *decreasing* block size below a model- and distribution-specific threshold can *increase* error when scale quantization is coarse [2601.19026]. For example, block sizes below 16 for FP8 E4M3 scales induce a “perplexity inversion” where PPL rises instead of falling with smaller block size. The cause is the interplay between the spread of tensor distributions (specifically, low-variance/narrow tensors) and the available dynamic range of quantized scales. When the block is too "narrow," quantizing the maximum to a low-precision scale can result in either the entire block being mapped to zero (if the true max falls below the smallest representable scale) or max error dominating the overall MSE.

Theoretical modeling attributes this to three contributors:

- Non-maximum element error (amplified by scale quantization granularity)
- Error from quantizing the block maximum itself
- All-zero block error (entire block mapped to quantized zero under some conditions)

### 4.2 Asymmetry and Clamping

Microscaling suppresses outliers but induces block-level *asymmetry*: when mean values in small blocks drift from zero, symmetric quantizer grids waste range coverage, resulting in increased rounding error [2411.09909]. Solutions include:

- *Asymmetric shared scaling*: Use separate scales for positive and negative subblocks (AMXFP4), substantially lowering empirical MSE and improving accuracy.
- *Rotation or transform-based preconditioning*: Blockwise Hadamard or optimal WUSH transforms can equalize distribution and improve quantization robustness [2512.00956][2509.23202].

## 5. Advanced Techniques: Rotation, Metadata, Outlier Handling

### 5.1 Micro-Rotated Quantization and Transforms

MR-GPTQ realizes significant FP4/NVFP4 accuracy boosts by blockwise Hadamard transforms and format-specific scale optimization [2509.23202]. WUSH derives the provably optimal blockwise linear transform for round-to-nearest, absmax quantizers, further minimizing loss [2512.00956].

### 5.2 Metadata-Augmented Formats

M²XFP introduces minimal block or subgroup metadata to locally refine quantized values. Subgroup-level mantissa (Sg-EM) metadata is used for weights, and top-1 element correction (Elem-EM) for activations, reducing average accuracy loss by >70% compared to MXFP4 at ~0.25 bits/element overhead [2601.19213].

### 5.3 Outlier Preservation

OPAL’s architecture reserves higher-precision representation for a small number of outliers per block (top-4 in k=128 blocks), with the majority quantized to 3–5 bits [2409.05902]. This yields <1 PPL increase and area/power savings of 2.4–3.1x.

### 5.4 Clustered Codebooks (LO-BCQ)

LO-BCQ iteratively assigns blocks to clusters, updating custom codebooks with per-cluster scaling for each, down to 0.2 PPL loss on LLMs in the W4A4 regime [2502.05376].

## 6. Hardware and Software Integration

Efficient deployment of blockwise quantization/microscaling depends on tight hardware/software co-design:

- **Tensor-Core Extension:** Hardware must support blockwise scale fetches and mixed exponent/mantissa alignment. MX block formats align with OCP and NVIDIA Blackwell architectures [2310.10537][2506.20752].
- **Metadata/Hybrid Handling:** M²XFP’s extra metadata processing and OPAL’s outlier buffer additions add minimal (≤10%) area and operate off the critical path [2601.19213][2409.05902].
- **Vectorized Kernels:** All leading libraries (MX-Lib, QuTLASS) provide fused block quantization, scale application, and matrix multiply kernels for NVIDIA/AMD GPUs [2405.07135][2509.23202].
- **Energy and Throughput:** Typical observed gains include 1.5–2x throughput vs. FP16/32 (INT8/FP6/FP4), with energy reduction up to 2.2x and area savings up to 3.1x [2409.05902][2601.19213].

## 7. Training Stability, Mitigation Strategies, and Practical Guidelines

Full-duration sub-8-bit training in MX formats exhibits a propensity for stochastic instability in gradient updates and irrecoverable divergences, particularly with increasing model and compute scale [2506.20752]. This is traced to multiplicative bias from quantized gradients in LayerNorm and activations. Stability is restored by:

- Using higher-precision activations/LayerNorm and only quantizing weights in the backward.
- Employing forward-only quantization or switching to higher-precision mid-training as soon as error metrics (e.g., estimated operator-norm of gradient noise) breach thresholds.

Practical guidelines include:

- Use block size 32 for hardware alignment (NVIDIA Blackwell).
- For 8-bit, MXINT8 or MXFP8 with direct PTQ is sufficient.
- For 4–6 bit, combine SmoothQuant, GPTQ, and/or MR-GPTQ, especially for FP4 formats.
- Avoid shrinking block size below 16 without appropriate scale representation (use FP8 UE5M3, not just E4M3, for FP4) [2601.19026].
- For critical workload stability in training, maintain BF16 or FP32 for activations and LayerNorm where possible. Monitor gradient error and adapt quantization schedules accordingly.

---
**References:**  
[2310.10537], [2601.19026], [2405.07135], [2409.05902], [2601.19213], [2506.20752], [2411.09909], [2509.23202], [2512.00956], [2502.05376].

Source: https://www.emergentmind.com/topics/blockwise-quantization-and-microscaling