Mix-Quant: Heterogeneous Quantization Strategies

• Mix-Quant is a framework of heterogeneous quantization strategies that assigns varied precision across layers, channels, and inference phases to balance accuracy and efficiency.
• It incorporates methods such as layer-wise bit allocation, activation-centric search, and system-level precision adjustments that are applied in CNNs, diffusion models, and large language models.
• Practical implementations range from linear programming for per-layer bits to phase-aware quantization, achieving notable speedups and reduced resource consumption without significant performance drops.

Mix-Quant denotes a family of quantization strategies that replace a single uniform numerical format with heterogeneous precision assignments or heterogeneous quantization treatments across layers, channels, output features, or inference phases. In the literature, the term spans classical mixed-precision CNN quantization, activation-oriented search, diffusion-specific quantization pipelines, and system-level LLM inference schemes. Representative formulations include class-separability-driven layer-wise bit allocation in CNNs, quantization-error-based per-layer PTQ, global mixed precision across LLM output features, and phase-aware quantization that runs prefilling in NVFP4 while preserving decoding in BF16 (Wang et al., 2022, Kloberdanz et al., 2023, Zheng et al., 2024, Lu et al., 19 May 2026).

1. Scope and terminology

One recurrent misconception is that Mix-Quant denotes a single method. The published usage is broader. In some papers it means per-layer mixed-precision bit-width allocation for weights or activations; in others it means mixing quantization formats, separating quantized and high-precision inference phases, or transforming a mixed-precision requirement into a uniform low-precision computation. The shared principle is selective precision assignment under an accuracy-efficiency constraint, but the selection variable differs substantially across domains.

Usage of Mix-Quant	Representative mechanism	Representative papers
Layer-wise mixed precision	Per-layer weight or activation bits under BOP/model-size constraints	(Wang et al., 2022, Kloberdanz et al., 2023, Kim et al., 2023)
Diffusion-oriented mixed precision	Bit allocation guided by image fidelity and text alignment, or mixed signed/unsigned FP	(Zhao et al., 2024, Zhao et al., 27 May 2025)
LLM system-level mixed precision	Phase-aware, output-feature-aware, or MX-format channel-aware quantization	(Zheng et al., 2024, Lu et al., 19 May 2026, Liu et al., 4 Aug 2025)
Structural or hardware reinterpretation	Mixed-to-uniform decomposition, block-rotation-aware permutations, or mixed schemes on FPGA	(Lee et al., 6 Apr 2026, Sanjeet et al., 29 Jan 2026, Chang et al., 2020)

This diversity matters because the optimization target is not always “bit-width per layer.” Some formulations optimize class separability, some optimize quantization error, some optimize salience to global loss, and some optimize hardware efficiency by restructuring arithmetic. A precise reading therefore requires identifying the granularity of the mix—layer, channel, output feature, timestep, inference phase, or arithmetic scheme.

2. Layer-wise mixed precision in CNN quantization

In CNN quantization, Mix-Quant initially referred to assigning different weight or activation bit-widths to different layers instead of using a single global precision. CSMPQ formalizes this setting as: given a floating-point model and a target resource constraint, choose per-layer bit-widths $b_i$ to maximize accuracy. Its central idea is that layers with more class-separable feature maps deserve more bits. To estimate this without iterative search, CSMPQ adapts TF-IDF from natural language processing to globally pooled feature maps, computes a layer score $\alpha_l$, maps it to $\theta_i = e^{\beta \alpha_i}$, and solves a linear program of the form

$$\max_b \sum_{i=1}^{L} b_i \theta_i \quad \text{s.t.} \quad \sum_{i=1}^{L} M(b_i) \le \mathcal{T}.$$

The search uses only one forward pass over $n=32$ sampled images, solves the linear program with SciPy, and then plugs the resulting bit configuration into HAWQ-V3-based QAT or BRECQ-based PTQ (Wang et al., 2022). On ImageNet-1k, CSMPQ reports 73.03\% Top-1 on ResNet-18 with 59 G BOPs for QAT, and 71.30\% Top-1 on MobileNetV2 with 1.5 Mb for PTQ (Wang et al., 2022).

A distinct but related formulation appears in MixQuant as a post-training weight-bit search algorithm. Instead of class separability, it uses per-layer weight quantization error

$$\mathrm{QE}_l(b) = \frac{1}{|W_l|}\left\|W_l - \tilde{W}_l^{(b)}\right\|_2^2$$

and selects the layer bit-width $\text{optBit}_l$ subject to

$$\mathrm{QE}_l(\text{optBit}_l) \le \mathrm{QE}_l(8)\cdot \mathrm{QEM}.$$

The candidate set is $B=\{8,7,6,5,4,3,2\}$, the search is linear in the number of layers and candidate bit-widths, and the output can be passed to BRECQ or vanilla asymmetric quantization as a preprocessing stage (Kloberdanz et al., 2023). The reported search overhead is 0.1–0.5 s for one QEM and 0.9–5.5 s for 10 QEM values across ResNet and ResNeXt models (Kloberdanz et al., 2023).

These two formulations illustrate an early bifurcation in Mix-Quant research. CSMPQ uses forward-activation semantics and a global resource constraint, whereas MixQuant uses local weight reconstruction error and a QE Multiplier rather than an explicit total model-size or latency budget. Both nonetheless encode the same premise: layer sensitivity is heterogeneous and can be exploited without exhaustive search.

3. Activation-centric search and diffusion-model variants

Activation quantization introduced a separate line of Mix-Quant work because activation distributions fluctuate with both weight updates and bit selection. MetaMix addresses what it terms activation instability due to bit selection. It fixes weight precision globally at 4-bit in the main experiments, searches per-layer activation bits from $\{8,4,3\}$ for MobileNet v2/v3 and $\alpha_l$0 for ResNet-18, and decomposes the process into a short bit-selection phase and a longer weight-training phase (Kim et al., 2023). The bit-selection phase alternates bit-meta training, which learns mixed-precision-aware weights and activation step sizes across all candidate bit-widths, and bit-search training, which freezes the meta-state and optimizes architecture parameters $\alpha_l$1 under a BOP regularizer. This design is intended to decouple weight learning from bit assignment and reduce BatchNorm instability. On ImageNet, MetaMix reports 72.60\% on MobileNet v2 at effective 3.98/4 and 74.24\% on MobileNet v3 Large at 3.83/4, while also reporting 13.4 GPU-hours search on ResNet-18 versus 40 for DNAS and 312 for SPOS (Kim et al., 2023).

In diffusion models, Mix-Quant acquired a more task-specific meaning because few-step text-to-image systems must preserve both visual quality and text alignment. MixDQ targets SDXL-turbo and related few-step models with three coordinated components: BOS-aware text embedding quantization, metric-decoupled sensitivity analysis, and integer-programming-based bit-width allocation (Zhao et al., 2024). BOS-aware quantization isolates the BOS token outlier in CLIP embeddings by precomputing FP outputs for cross-attention to_k and to_v on BOS while quantizing other tokens. Metric decoupling separates content-related layers, scored by SSIM, from quality-related layers, scored by SQNR, and then solves a discrete allocation problem over $\alpha_l$2 under a resource budget. On SDXL-turbo 1-step, the paper reports W8A8 without performance loss, W4A8 with negligible visual degradation, 3–4x reduction in model size and memory cost, and 1.45x latency speedup (Zhao et al., 2024).

A complementary diffusion formulation appears in the 4-bit FP literature. The mixup-sign floating-point quantization framework identifies a specific failure mode of signed FP quantization in layers after SiLU, where activations are highly asymmetric and the negative side occupies a small interval near $\alpha_l$3. To address this, it uses signed FP for weights and a mixed candidate set for activations: signed FP for normal activation layers and either signed FP or unsigned FP with a real-domain zero point for anomalous activation layers, selecting the better option by MSE (Zhao et al., 27 May 2025). Fine-tuning is then performed with Timestep-Aware LoRA, which uses a timestep-conditioned router over a LoRA hub, and Denoising-Factor Loss Alignment, which weights the noise-prediction loss by the denoising factor derived from the diffusion update. The method states that it is the first to achieve superior performance in 4-bit FP quantization for diffusion models over existing PTQ fine-tuning methods in 4-bit INT quantization, and reports CIFAR-10 DDIM FID 6.02 at 4/4 bits with $\alpha_l$4 versus 38.40 for EfficientDM (Zhao et al., 27 May 2025).

4. LLM-oriented Mix-Quant and system co-design

In LLMs, Mix-Quant increasingly refers to quantization schemes that are inseparable from the serving stack. MixLLM introduces global mixed precision between output features. For each output channel $\alpha_l$5, it estimates salience by a first- and second-order Taylor approximation of model-loss change under channel quantization, using a Fisher approximation to the Hessian. The resulting score is

$\alpha_l$6

where $\alpha_l$7 is the original channel, $\alpha_l$8 is its quantized version, and $\alpha_l$9 is the per-sample gradient (Zheng et al., 2024). All output channels across all layers are globally sorted by this salience, and the top fraction is assigned 8-bit while the rest remain 4-bit. This is coupled with 8-bit activations, two-step dequantization, fast integer-to-float conversion, and a software pipeline that overlaps memory access, dequantization, and MatMul. The paper reports that with only 10\% more bits, the PPL increasement is reduced from about 0.5 in SOTA to within 0.2 for Llama 3.1 70B, and MMLU-Pro improves by 0.93 on average over the SOTA of three popular models (Zheng et al., 2024).

A more phase-structured meaning of Mix-Quant appears in agentic LLM inference. The phase-aware Mix-Quant framework for agents runs prefilling in NVFP4 W4A4 and keeps decoding in BF16 (Lu et al., 19 May 2026). The argument is that agentic workloads are input-heavy, prefilling dominates compute, and prefilling is more tolerant to quantization than autoregressive decoding. The paper reports that full NVFP4 harms agentic performance, whereas prefilling-only quantization largely preserves quality; on a 128K-token context, the top 4,096 tokens account for 95.8\% of attention mass on average, and the system achieves up to a 3x speedup during prefilling (Lu et al., 19 May 2026). This formulation is phase-aware rather than layer-aware: the precision split is between context encoding and token generation.

MicroMix takes the Blackwell-specific MX route. It uses MXFP4, MXFP6, and MXFP8 channels within a single linear layer, derives quantization thresholds $\theta_i = e^{\beta \alpha_i}$0 so that MXFP4 or MXFP6 errors do not exceed the INT8 upper bound, and implements a custom kernel that supports arbitrary combinations of MXFP4, MXFP6, and MXFP8 channels while producing BF16 outputs (Liu et al., 4 Aug 2025). The channel mix is calibrated per layer, and weights are prepacked accordingly. The paper reports at least 20\% faster execution than TensorRT-FP8 on both RTX 5070Ti laptop and RTX 5090, together with improved prefill latency and memory efficiency across Llama and Qwen models (Liu et al., 4 Aug 2025).

These LLM variants show that, in contemporary usage, Mix-Quant often means algorithm–system co-design: the unit of mixing may be output features or inference phases, and the quantization policy is only meaningful together with a compatible kernel and memory layout.

5. Outlier handling, rotations, and mixed-to-uniform reinterpretations

Several papers broaden Mix-Quant beyond explicit bit allocation. MixDiT applies mixed Microscaling formats to Diffusion Transformer activations, assigning MX9 to magnitude-based outlier channels or heads and MX6 to the remainder, then reordering them so inliers and outliers do not share MX groups (Kim et al., 11 Apr 2025). Its accelerator supports MX6×MX6, MX6×MX9, and MX9×MX9 GEMMs, and the reported result is 2.10–5.32 times speedup over RTX 3090 with no loss in FID (Kim et al., 11 Apr 2025). Here the mix is neither layer-wise nor phase-wise; it is within-tensor, along channels or heads.

MUXQ proposes a different reinterpretation. It starts from the observation that activation outliers make some channels behave as if they require higher precision, but instead of preserving a mixed-precision execution path, it decomposes the activation matrix into a Body and an Aux branch so that the full output can be reconstructed as

$\theta_i = e^{\beta \alpha_i}$1

This “mixed-to-uniform” construction allows both branches to remain uniformly quantized in low-precision INT while mitigating outlier effects (Lee et al., 6 Apr 2026). The method uses the same outlier criterion as LLM.int8(), namely whether a channel contains an element with $\theta_i = e^{\beta \alpha_i}$2, but keeps the runtime path pure INT instead of mixing INT8 and FP16 (Lee et al., 6 Apr 2026).

MixQuant in block-rotation PTQ addresses outliers from a geometric perspective. It shows that for block Hadamard rotations the post-rotation $\theta_i = e^{\beta \alpha_i}$3 norm is controlled by how the $\theta_i = e^{\beta \alpha_i}$4 mass is distributed across blocks, and that post-rotation outliers are deterministically minimized when the pre-rotation $\theta_i = e^{\beta \alpha_i}$5 norm mass is evenly distributed across blocks (Sanjeet et al., 29 Jan 2026). Guided by this analysis, it introduces the greedy MassDiff permutation algorithm, which equalizes expected blockwise $\theta_i = e^{\beta \alpha_i}$6 norms before rotation and then merges the resulting permutations into transformer weights via permutation-equivariant regions. On Llama3 1B quantized to INT4 with block size 16, this recovers up to 90\% of the full-vector rotation perplexity, compared with 46\% without permutations (Sanjeet et al., 29 Jan 2026).

An earlier hardware-centric precursor is the FPGA mixed scheme quantization framework, which mixes sum-of-power-of-2 quantization and fixed-point quantization across rows of a weight matrix (Chang et al., 2020). SP2 is assigned to Gaussian-like rows and mapped to LUT-based shift-add arithmetic, while fixed-point is assigned to Uniform-like rows and mapped to DSP multiplication. The paper reports 2.1–4.1x throughput improvement versus single-scheme designs, indicating that “mix” can refer to heterogeneous quantization schemes as well as heterogeneous bit-widths (Chang et al., 2020).

6. Common principles, misconceptions, and open constraints

Across these formulations, Mix-Quant is best understood as a family of selective-precision methods rather than a fixed algorithm. A first misconception is that the field is uniformly about per-layer bit-width search. The evidence instead spans per-layer class-separability allocation, per-layer QE search, activation-bit selection on a meta-state, output-feature salience, phase-aware prefilling/decoding, channel/head MX assignment, mixed arithmetic schemes, and algebraic mixed-to-uniform decompositions (Wang et al., 2022, Kloberdanz et al., 2023, Kim et al., 2023, Zheng et al., 2024, Lee et al., 6 Apr 2026).

A second misconception is that mixed precision is always determined by local heuristics. Several recent methods are explicitly global. MixLLM globally ranks output features across the whole model, MixQuant for block rotations globally calibrates permutations from calibration data, and phase-aware Mix-Quant decides precision by inference stage rather than by layer (Zheng et al., 2024, Sanjeet et al., 29 Jan 2026, Lu et al., 19 May 2026). This suggests that the most informative “importance” signal is increasingly model-level or workflow-level rather than strictly layer-local.

The main constraints are equally recurrent. Some methods depend on labels or representative calibration data: CSMPQ uses class separability estimated from sampled labeled images, and its effectiveness depends on sample representativeness (Wang et al., 2022). Some methods use local objectives without an explicit global memory or latency budget: MixQuant’s QE Multiplier controls aggressiveness indirectly rather than through a formal total-bit constraint (Kloberdanz et al., 2023). Some are tightly coupled to hardware generations: phase-aware NVFP4 inference and MicroMix’s MX kernels rely on Blackwell-class support for NVFP4 or MXFP Tensor Core execution (Lu et al., 19 May 2026, Liu et al., 4 Aug 2025). In agentic LLMs, decoding remains fragile because token errors accumulate autoregressively, so aggressive decode quantization still causes performance drops (Lu et al., 19 May 2026). In diffusion models, asymmetric activations and timestep-dependent denoising sensitivity force quantization schemes to model nonuniform temporal difficulty rather than just static tensor distributions (Zhao et al., 27 May 2025).

The most stable encyclopedic conclusion is therefore narrow but robust: Mix-Quant is the research program of replacing uniform quantization with targeted numerical heterogeneity, where the target may be a layer, channel, feature, timestep, arithmetic path, or inference phase. Its technical evolution has moved from layer sensitivity and resource budgets toward global salience estimation, distribution-shape matching, workflow-aware precision separation, and hardware-native execution.