---
title: Mixture-of-Formats Quantization (MoFQ)
url: https://www.emergentmind.com/topics/mixture-of-formats-quantization-mofq
type: topic
---

# Mixture-of-Formats Quantization (MoFQ)

Mixture-of-Formats Quantization (MoFQ) refers to a family of quantization strategies in which multiple low-precision number formats (commonly low-bit integer and floating-point) are selectively assigned within a neural network, either at the per-layer, per-block, per-channel, or per-operand level. Distinct from uniform quantization, where one numeric type and bit-width is globally fixed, MoFQ exploits the complementary error profiles and hardware execution properties of various numeric formats to optimize model quality, system throughput, and memory utilization within a single model deployment or training instance. Originating in post-training quantization for large language models (LLMs), MoFQ has rapidly evolved into a general principle now instantiated in both inference and training, weight-only and activation-inclusive regimes, and across heterogeneous accelerators.

## 1. Foundational Principles and Motivation

MoFQ is motivated by the empirical observation that no single low-precision numeric format uniformly minimizes quantization error across all network components. For large transformer models, 8-bit weight quantization often favors INT8, while 4-bit quantization advantages of integers vanish; activation tensors with heavy-tailed distributions are better approximated by floating-point representations such as FP8 due to their non-uniform granularity and greater dynamic range [2305.12356].

The central premise is that network error sensitivity and the shape of tensor distributions (e.g., weight kurtosis, activation sparsity, or temporal frequency of experts in MoE) vary between layers and blocks. As a result, dynamically selecting the optimal format per unit—subject to fixed bit-width and hardware compatibility—consistently yields lower error under the same resource constraints as any single-format scheme. MoFQ generalizes this by providing a framework in which format selection is guided by quantization error statistics (such as per-tensor MSE, relative error, or impact on layer output) and, in advanced extensions, by cost–benefit tradeoff metrics tied to system latency or expert activation profiles [2305.12356, 2505.05799, 2512.22804, 2604.00529, 2511.06838].

## 2. Numeric Formats and Format Assignment Criteria

MoFQ leverages a set of candidate formats, most notably:

- **Integer (INTk):** $k$-bit signed integers, typically with a per-layer or per-group scale and possibly a zero point for symmetric/asymmetric mappings.
- **Floating-point (FPk, MXFP):** $k$-bit representations with explicit sign, exponent, and mantissa partitioning (e.g., FP8-E4M3, FP4-E2M1), sometimes employing format reallocation to repurpose NaN/Inf encodings as additional numbers (e.g., BitMoD in P3-LLM; reallocated FP4) [2305.12356, 2511.06838].
- **Other low-precision formats:** NVFP4, E5M2, BF16, asymmetric INT4 for KV-cache, FP8-S0E4M4 for attention scores [2512.22804, 2511.06838].

The assignment of formats occurs via empirical evaluation over each tensor unit (layer, block, channel), selecting the format that minimizes a target error metric:
- Weight MSE ($\|W-\hat{W}\|^2$)
- Mean relative error (per-tensor or per-block)
- Output error (noise–signal power ratio)
- Downstream performance loss (task-specific metrics, e.g., log-loss or accuracy drop)

Table: Typical Format Selection Policies

| Tensor Unit    | Candidate Formats   | Criterion Used                |
|----------------|--------------------|-------------------------------|
| Layer (LLM)    | INT4, FP4          | Layer MSE, output error [2305.12356] |
| MoE Block      | {2,...,8} bits INT/FP | $S_i \cdot 2^{-b_i}$, budgeted [2505.05799] |
| Sub-tensor     | FP8, BF16          | Mean relative error < thresh. [2512.22804] |
| Operand (PIM)  | INT4, FP4, FP8     | ΔPPL/Err$_r$, memory cost [2511.06838]   |

Format selection can use greedy algorithms, integer linear programming, or hard-coded thresholds, depending on accuracy, latency, and hardware overhead objectives.

## 3. Implementation: Algorithms and Hardware Compatibility

The MoFQ algorithm consists of repeated evaluations per target unit:
- For each unit (layer/block/channel), quantize with each candidate format.
- Evaluate the quantization error metric (runtime MSE, per-block relative error, etc.).
- Assign the format with the minimal acceptable error, optionally subject to cost or latency budgets.

For weight-activation (WA) and weight-only (W-only) quantization, calibration data may be required to set activation scales or estimate output errors. In MF-QAT, the process becomes fully differentiable, using multi-format quantized forward passes and straight-through estimators in the backward pass [2604.00529].

MoFQ is designed for hardware-compatibility:
- Many modern accelerators (e.g., NVIDIA H100, Intel, AMD) provide native support for both integer and floating-point low-bit formats, such that INT8 and FP8 have near-identical MAC area and performance; at 4 bits, the gap further vanishes [2305.12356].
- In specialized hardware (e.g., P3-LLM), operand-specific MAC units are co-designed to exploit operand format diversity (e.g., 6-bit fixed-point units, multi-format GroupGEMM kernels) [2511.06838, 2505.05799].

The "Slice-and-Scale" procedure in MF-QAT facilitates on-the-fly conversion from a single stored anchor format to any lower-precision format, incurring only minor accuracy loss, and can be implemented with simple shifts/adds for integers or truncated re-encoding for floating point [2604.00529].

## 4. Empirical Analysis and Performance Results

MoFQ consistently demonstrates superior trade-offs between model quality, compression, and throughput compared to single-format baselines:

- **W-only 4-bit MoFQ on LLaMA-13B (WikiText-2 PPL):** FP16: 5.09; INT4(GPTQ): 5.40; FP4: 5.35; MoFQ4: 5.33 (97.1% layers in FP4). Quantization time: MoFQ4 is $15.8\times$ faster than GPTQ [2305.12356].
- **WA 8-bit (W8A8) on LLaMA:** FP8 alone achieves much lower PPL than INT8 (e.g., 6.59 vs 368.21); MoFQ8 closes the gap to FP16 within 1–2% [2305.12356].
- **MxMoE (MoE models):** MxMoE at 3.25 bits delivers $0.3{-}0.6$ lower PPL than GPTQ, up to $3.4\times$ speedup on RTX-4090 vs FP16 Group-GEMM at equivalent or lower PPL, and reduces latency by 29.4% versus uniform quantization [2505.05799].
- **MoR (mixed-precision training):** Over 98% of blocks in large transformer models quantized to FP8 while keeping loss and downstream metrics within $1\%$ of BF16 [2512.22804].
- **MF-QAT (multi-format QAT):** Single checkpoint covers INT2/4/6/8 or FP4/6/8; multi-format QAT is always within $1\%$ of the best single-format model at every bit-width. Slice-and-Scale conversion yields $<0.2$ PPL loss vs direct quantization [2604.00529].
- **P3-LLM (accelerator):** Four-format (FP4/INT4/FP8/FP8-special) quantization achieves $\Delta$PPL $<$0.3 vs FP16, $4$–$8\times$ throughput and $2$–$6\times$ energy efficiency boosts over competing LLM accelerators [2511.06838].

## 5. Extensions: Mixed-Precision Optimization and Kernel Co-Design

Beyond simple per-layer format selection, MoFQ has been generalized:
- **Sensitivity-driven bit allocation:** Assign more bits and higher-fidelity formats to blocks/layers with higher error sensitivity (as measured via calibration or analytic proxies) and/or higher activation probabilities (MoE) [2505.05799].
- **Kernel co-design:** Automated generation of hand-tuned microkernels per format and fused multi-format GroupGEMM execution, using uniform resource scheduling and tile-wise makespan minimization to maximize hardware occupancy and throughput [2505.05799].
- **Elastic precision inference:** Store one "anchor" model (e.g., MXINT8), then deploy at any lower bit-width or format (INT4, FP4, etc.) by on-the-fly conversion, obviating the need for retraining or multiple checkpoints. Experiments show negligible accuracy loss and direct hardware acceleration compatibility [2604.00529].

## 6. Broader Design Guidelines and Applicability

Findings from recent empirical studies establish several general design principles for MoFQ:

- **Allocate numeric format and bit-width using sensitivity-to-cost ratio:** $S_i/C_i$, where $S_i$ is quantization error impact and $C_i$ the resource cost [2505.05799].
- **Incorporate activation frequency (especially in MoE):** Downscale precision for rarely-invoked experts or blocks, weighted by measured activation probability [2505.05799].
- **Maintain hardware alignment:** Select formats and block partitioning to match the on-chip vector width and supported MAC data paths; ensure efficient packing and execution with microkernel fusion [2511.06838, 2505.05799].
- **Robustness via dynamic decision-making:** Frameworks like MoR enable runtime quantization decisions based on actual tensor statistics, delivering near FP8 throughput with BF16-level accuracy [2512.22804].

MoFQ strategies are now adopted in diverse contexts spanning standard LLMs, MoE architectures, hybrid NPU-PIM accelerators, and quantization-aware training pipelines for elastic deployment. Continued extensions are anticipated—such as meta-learned quantization policies, support for ultra-low bit formats (e.g., NVFP4), and further hardware–software co-optimization [2512.22804, 2604.00529, 2511.06838].

## 7. Comparative Summary Table

| MoFQ Instantiation | Format Selection Granularity | Formats Used                                   | Empirical Benefit                                     | Reference        |
|---------------------|-----------------------------|------------------------------------------------|-------------------------------------------------------|------------------|
| LLM/PTQ             | Per-layer                   | INT4/FP4, INT8/FP8                             | SOTA 4b/8b quality, $15{\times}$+ speedup over GPTQ   | [2305.12356]     |
| MoE (MxMoE)         | Per-block                   | {2,...,8} INT/FP                               | $0.3-2.4$ PPL lower, $3.4{\times}$ speedup             | [2505.05799]     |
| MoR (Mixed Training)| Per-tensor, per-block       | FP8, BF16, E5M2                                | 98%+ blocks in FP8, $<1\%$ loss vs BF16                | [2512.22804]     |
| MF-QAT              | Block (elastic)             | MXINT/8/6/4/2, MXFP/8/6/4                      | One checkpoint, loss $<0.2$ PPL vs direct             | [2604.00529]     |
| P3-LLM              | Operand (W/A/KV/P)          | FP4, INT4, FP8-E4M3, FP8-S0E4M4                | $4-8{\times}$ throughput, $<0.3$ $\Delta$PPL           | [2511.06838]     |

MoFQ is now established as a foundational principle in the quantization and deployment of large-scale neural networks, enabling hardware-friendly, fine-grained adaptation of numeric representations for maximal efficiency and quality.

Source: https://www.emergentmind.com/topics/mixture-of-formats-quantization-mofq