Mixture-of-Formats Quantization (MoFQ)

• MoFQ is a quantization strategy that assigns different low-precision formats (e.g., INT and FP) to network components based on their error profiles.
• It optimizes model quality, throughput, and memory utilization by leveraging complementary numeric formats and hardware-specific execution features.
• Empirical results show MoFQ reduces quantization error and latency in large models such as LLMs and MoE architectures compared to uniform quantization.

Mixture-of-Formats Quantization (MoFQ) refers to a family of quantization strategies in which multiple low-precision number formats (commonly low-bit integer and floating-point) are selectively assigned within a neural network, either at the per-layer, per-block, per-channel, or per-operand level. Distinct from uniform quantization, where one numeric type and bit-width is globally fixed, MoFQ exploits the complementary error profiles and hardware execution properties of various numeric formats to optimize model quality, system throughput, and memory utilization within a single model deployment or training instance. Originating in post-training quantization for LLMs, MoFQ has rapidly evolved into a general principle now instantiated in both inference and training, weight-only and activation-inclusive regimes, and across heterogeneous accelerators.

1. Foundational Principles and Motivation

MoFQ is motivated by the empirical observation that no single low-precision numeric format uniformly minimizes quantization error across all network components. For large transformer models, 8-bit weight quantization often favors INT8, while 4-bit quantization advantages of integers vanish; activation tensors with heavy-tailed distributions are better approximated by floating-point representations such as FP8 due to their non-uniform granularity and greater dynamic range (Zhang et al., 2023).

The central premise is that network error sensitivity and the shape of tensor distributions (e.g., weight kurtosis, activation sparsity, or temporal frequency of experts in MoE) vary between layers and blocks. As a result, dynamically selecting the optimal format per unit—subject to fixed bit-width and hardware compatibility—consistently yields lower error under the same resource constraints as any single-format scheme. MoFQ generalizes this by providing a framework in which format selection is guided by quantization error statistics (such as per-tensor MSE, relative error, or impact on layer output) and, in advanced extensions, by cost–benefit tradeoff metrics tied to system latency or expert activation profiles (Zhang et al., 2023, Duanmu et al., 9 May 2025, Su et al., 28 Dec 2025, Xu et al., 1 Apr 2026, Chen et al., 10 Nov 2025).

2. Numeric Formats and Format Assignment Criteria

MoFQ leverages a set of candidate formats, most notably:

• Integer (INTk): $k$-bit signed integers, typically with a per-layer or per-group scale and possibly a zero point for symmetric/asymmetric mappings.
• Floating-point (FPk, MXFP): $k$-bit representations with explicit sign, exponent, and mantissa partitioning (e.g., FP8-E4M3, FP4-E2M1), sometimes employing format reallocation to repurpose NaN/Inf encodings as additional numbers (e.g., BitMoD in P3-LLM; reallocated FP4) (Zhang et al., 2023, Chen et al., 10 Nov 2025).
• Other low-precision formats: NVFP4, E5M2, BF16, asymmetric INT4 for KV-cache, FP8-S0E4M4 for attention scores (Su et al., 28 Dec 2025, Chen et al., 10 Nov 2025).

The assignment of formats occurs via empirical evaluation over each tensor unit (layer, block, channel), selecting the format that minimizes a target error metric:

• Weight MSE ($\|W-\hat{W}\|^2$)
• Mean relative error (per-tensor or per-block)
• Output error (noise–signal power ratio)
• Downstream performance loss (task-specific metrics, e.g., log-loss or accuracy drop)

Table: Typical Format Selection Policies

Tensor Unit	Candidate Formats	Criterion Used
Layer (LLM)	INT4, FP4	Layer MSE, output error (Zhang et al., 2023)
MoE Block	{2,...,8} bits INT/FP	$S_i \cdot 2^{-b_i}$, budgeted (Duanmu et al., 9 May 2025)
Sub-tensor	FP8, BF16	Mean relative error < thresh. (Su et al., 28 Dec 2025)
Operand (PIM)	INT4, FP4, FP8	ΔPPL/Err$_r$, memory cost (Chen et al., 10 Nov 2025)

Format selection can use greedy algorithms, integer linear programming, or hard-coded thresholds, depending on accuracy, latency, and hardware overhead objectives.

3. Implementation: Algorithms and Hardware Compatibility

The MoFQ algorithm consists of repeated evaluations per target unit:

• For each unit (layer/block/channel), quantize with each candidate format.
• Evaluate the quantization error metric (runtime MSE, per-block relative error, etc.).
• Assign the format with the minimal acceptable error, optionally subject to cost or latency budgets.

For weight-activation (WA) and weight-only (W-only) quantization, calibration data may be required to set activation scales or estimate output errors. In MF-QAT, the process becomes fully differentiable, using multi-format quantized forward passes and straight-through estimators in the backward pass (Xu et al., 1 Apr 2026).

MoFQ is designed for hardware-compatibility:

• Many modern accelerators (e.g., NVIDIA H100, Intel, AMD) provide native support for both integer and floating-point low-bit formats, such that INT8 and FP8 have near-identical MAC area and performance; at 4 bits, the gap further vanishes (Zhang et al., 2023).
• In specialized hardware (e.g., P3-LLM), operand-specific MAC units are co-designed to exploit operand format diversity (e.g., 6-bit fixed-point units, multi-format GroupGEMM kernels) (Chen et al., 10 Nov 2025, Duanmu et al., 9 May 2025).

The "Slice-and-Scale" procedure in MF-QAT facilitates on-the-fly conversion from a single stored anchor format to any lower-precision format, incurring only minor accuracy loss, and can be implemented with simple shifts/adds for integers or truncated re-encoding for floating point (Xu et al., 1 Apr 2026).

4. Empirical Analysis and Performance Results

MoFQ consistently demonstrates superior trade-offs between model quality, compression, and throughput compared to single-format baselines:

• W-only 4-bit MoFQ on LLaMA-13B (WikiText-2 PPL): FP16: 5.09; INT4(GPTQ): 5.40; FP4: 5.35; MoFQ4: 5.33 (97.1% layers in FP4). Quantization time: MoFQ4 is $15.8\times$ faster than GPTQ (Zhang et al., 2023).
• WA 8-bit (W8A8) on LLaMA: FP8 alone achieves much lower PPL than INT8 (e.g., 6.59 vs 368.21); MoFQ8 closes the gap to FP16 within 1–2% (Zhang et al., 2023).
• MxMoE (MoE models): MxMoE at 3.25 bits delivers $0.3{-}0.6$ lower PPL than GPTQ, up to $3.4\times$ speedup on RTX-4090 vs FP16 Group-GEMM at equivalent or lower PPL, and reduces latency by 29.4% versus uniform quantization (Duanmu et al., 9 May 2025).
• MoR (mixed-precision training): Over 98% of blocks in large transformer models quantized to FP8 while keeping loss and downstream metrics within $1\%$ of BF16 (Su et al., 28 Dec 2025).
• MF-QAT (multi-format QAT): Single checkpoint covers INT2/4/6/8 or FP4/6/8; multi-format QAT is always within $1\%$ of the best single-format model at every bit-width. Slice-and-Scale conversion yields $k$0 PPL loss vs direct quantization (Xu et al., 1 Apr 2026).
• P3-LLM (accelerator): Four-format (FP4/INT4/FP8/FP8-special) quantization achieves $k$1PPL $k$20.3 vs FP16, $k$3–$k$4 throughput and $k$5–$k$6 energy efficiency boosts over competing LLM accelerators (Chen et al., 10 Nov 2025).

5. Extensions: Mixed-Precision Optimization and Kernel Co-Design

Beyond simple per-layer format selection, MoFQ has been generalized:

• Sensitivity-driven bit allocation: Assign more bits and higher-fidelity formats to blocks/layers with higher error sensitivity (as measured via calibration or analytic proxies) and/or higher activation probabilities (MoE) (Duanmu et al., 9 May 2025).
• Kernel co-design: Automated generation of hand-tuned microkernels per format and fused multi-format GroupGEMM execution, using uniform resource scheduling and tile-wise makespan minimization to maximize hardware occupancy and throughput (Duanmu et al., 9 May 2025).
• Elastic precision inference: Store one "anchor" model (e.g., MXINT8), then deploy at any lower bit-width or format (INT4, FP4, etc.) by on-the-fly conversion, obviating the need for retraining or multiple checkpoints. Experiments show negligible accuracy loss and direct hardware acceleration compatibility (Xu et al., 1 Apr 2026).

6. Broader Design Guidelines and Applicability

Findings from recent empirical studies establish several general design principles for MoFQ:

• Allocate numeric format and bit-width using sensitivity-to-cost ratio: $k$7, where $k$8 is quantization error impact and $k$9 the resource cost (Duanmu et al., 9 May 2025).
• Incorporate activation frequency (especially in MoE): Downscale precision for rarely-invoked experts or blocks, weighted by measured activation probability (Duanmu et al., 9 May 2025).
• Maintain hardware alignment: Select formats and block partitioning to match the on-chip vector width and supported MAC data paths; ensure efficient packing and execution with microkernel fusion (Chen et al., 10 Nov 2025, Duanmu et al., 9 May 2025).
• Robustness via dynamic decision-making: Frameworks like MoR enable runtime quantization decisions based on actual tensor statistics, delivering near FP8 throughput with BF16-level accuracy (Su et al., 28 Dec 2025).

MoFQ strategies are now adopted in diverse contexts spanning standard LLMs, MoE architectures, hybrid NPU-PIM accelerators, and quantization-aware training pipelines for elastic deployment. Continued extensions are anticipated—such as meta-learned quantization policies, support for ultra-low bit formats (e.g., NVFP4), and further hardware–software co-optimization (Su et al., 28 Dec 2025, Xu et al., 1 Apr 2026, Chen et al., 10 Nov 2025).

7. Comparative Summary Table

MoFQ Instantiation	Format Selection Granularity	Formats Used	Empirical Benefit	Reference
LLM/PTQ	Per-layer	INT4/FP4, INT8/FP8	SOTA 4b/8b quality, $\|W-\hat{W}\|^2$0+ speedup over GPTQ	(Zhang et al., 2023)
MoE (MxMoE)	Per-block	{2,...,8} INT/FP	$\|W-\hat{W}\|^2$1 PPL lower, $\|W-\hat{W}\|^2$2 speedup	(Duanmu et al., 9 May 2025)
MoR (Mixed Training)	Per-tensor, per-block	FP8, BF16, E5M2	98%+ blocks in FP8, $\|W-\hat{W}\|^2$3 loss vs BF16	(Su et al., 28 Dec 2025)
MF-QAT	Block (elastic)	MXINT/8/6/4/2, MXFP/8/6/4	One checkpoint, loss $\|W-\hat{W}\|^2$4 PPL vs direct	(Xu et al., 1 Apr 2026)
P3-LLM	Operand (W/A/KV/P)	FP4, INT4, FP8-E4M3, FP8-S0E4M4	$\|W-\hat{W}\|^2$5 throughput, $\|W-\hat{W}\|^2$6 $\|W-\hat{W}\|^2$7PPL	(Chen et al., 10 Nov 2025)

MoFQ is now established as a foundational principle in the quantization and deployment of large-scale neural networks, enabling hardware-friendly, fine-grained adaptation of numeric representations for maximal efficiency and quality.