Interaction-aware Mixed-Precision Quantization

• The paper introduces IMPQ, a framework that treats bit allocation as an interaction-aware optimization problem using cooperative game theory and Shapley values.
• It employs SPQE, a progressive quantization method leveraging Monte Carlo sampling to estimate both layer sensitivities and inter-layer interactions.
• The approach achieves significant perplexity reductions—up to 80%—across various LLMs under tight memory budgets compared to conventional methods.

Searching arXiv for IMPQ and closely related mixed-precision quantization work to ground the article. Interaction-aware Mixed-Precision Quantization (IMPQ) is a mixed-precision quantization framework for LLMs that treats bit allocation as an explicitly interaction-aware optimization problem rather than as a collection of isolated layerwise decisions. In the formulation reported for IMPQ, the central premise is that once average precision drops below 4 bits, quantization errors cease to behave independently across layers: quantizing one layer can amplify or suppress the degradation induced by quantizing another. IMPQ therefore models Transformer layers as players in a cooperative game, estimates both layer sensitivities and inter-layer interactions through Shapley-based Progressive Quantization Estimation (SPQE), and then solves a binary quadratic optimization problem that assigns either 2-bit or 4-bit precision under a strict memory budget (Zhao et al., 18 Sep 2025).

1. Conceptual basis and scope

IMPQ is motivated by a failure mode of conventional mixed-precision quantization in the ultra-low-bit regime. Methods based on isolated layerwise criteria—such as gradient sensitivity, activation norms, Z-score statistics, or cosine modification measures—implicitly assume that each layer’s quantization impact is mostly independent. The IMPQ formulation rejects that assumption for sub-4-bit operation and instead treats quantization as interaction-dominated: the marginal effect of quantizing layer $i$ depends on which other layers have already been quantized (Zhao et al., 18 Sep 2025).

The framework is defined for LLM post-training quantization. Its reported experiments cover Llama-3, Gemma-2, and Qwen-3 models, and it is evaluated across three independent PTQ backends: Quanto, HQQ, and GPTQ. The main evaluation metric is Perplexity on WikiText-2, while calibration and bit-allocation optimization use the C4 training split (Zhao et al., 18 Sep 2025).

This suggests that IMPQ belongs to a broader transition in mixed-precision research away from purely local sensitivity proxies. Earlier work typically framed layer importance through learned scale factors, Hessian surrogates, SQNR rankings, or differentiable bit learning (Tang et al., 2022, Pandey et al., 2023, Schaefer et al., 2022). IMPQ’s distinctive claim is that these layerwise views become systematically incomplete when average precision is pushed far enough downward (Zhao et al., 18 Sep 2025).

2. Cooperative-game formulation of layer interactions

IMPQ formulates Transformer layers as players in a cooperative game. Let $T=\{1,2,\dots,L\}$ be the set of layers, and let $S\subseteq T$ denote the layers kept at high precision, while layers in $T\setminus S$ are quantized to low precision. The payoff is defined as the average per-token negative log-likelihood: $$v(S) = \mathbb{E}_{(x,t)\sim\mathcal D} \Big[-\log p(x_{t+1}\mid x_{\le t};S)\Big].$$ Within this formulation, a layer’s importance is not its standalone sensitivity under a single configuration, but its average marginal contribution across quantization contexts (Zhao et al., 18 Sep 2025).

The corresponding Shapley value for layer $i$ is

$$\phi_i = \sum_{S \subseteq N \setminus \{i\}} \frac{|S|!(n-|S|-1)!}{n!} \Big(v(S\cup\{i\})-v(S)\Big).$$

This quantity averages the effect of adding layer $i$ over all coalitions not containing it. In the IMPQ interpretation, a large Shapley value indicates that moving layer $i$ from high precision to low precision tends to produce a large loss increase across many surrounding configurations (Zhao et al., 18 Sep 2025).

The significance of this construction is methodological. Mixed-precision allocation is no longer reduced to ranking layers by a single local metric; it is reframed as estimating marginal degradation in a combinatorial space of jointly quantized models. A plausible implication is that the framework is particularly well suited to settings where error propagation is nonlinear and strongly architecture-dependent.

3. SPQE: Shapley-based Progressive Quantization Estimation

Direct computation of exact Shapley values is infeasible for LLMs, so IMPQ introduces SPQE, or Shapley-based Progressive Quantization Estimation, as an approximation procedure based on Monte Carlo sampling over random permutations (Zhao et al., 18 Sep 2025).

A key design choice is progressive quantization rather than pruning. SPQE starts from a uniformly high-precision model and progressively lowers the precision of layers one by one along a sampled permutation. If $\pi=(\pi_1,\pi_2,\dots,\pi_L)$ is a random permutation and $T=\{1,2,\dots,L\}$0 is the set of layers still kept at high precision after the first $T=\{1,2,\dots,L\}$1 quantizations, then

$T=\{1,2,\dots,L\}$2

When layer $T=\{1,2,\dots,L\}$3 is quantized, its marginal contribution is

$T=\{1,2,\dots,L\}$4

Across $T=\{1,2,\dots,L\}$5 sampled permutations, the Shapley estimate for layer $T=\{1,2,\dots,L\}$6 is

$T=\{1,2,\dots,L\}$7

The practical rationale is explicit: progressive quantization produces smoother, more stable changes in negative log-likelihood than abrupt pruning, thereby reducing estimator noise (Zhao et al., 18 Sep 2025). The method is therefore designed to measure each layer in many different contexts of previously quantized layers, which is precisely where interaction structure enters the estimate.

SPQE also supplies the raw material for pairwise interaction modeling. The empirical marginal contributions are assembled into a covariance-like matrix

$T=\{1,2,\dots,L\}$8

where $T=\{1,2,\dots,L\}$9. To reduce noise, the method applies diagonal shrinkage: $S\subseteq T$0 with $S\subseteq T$1. It then defines an adjusted first-order sensitivity term by subtracting interaction contributions: $S\subseteq T$2 In this decomposition, $S\subseteq T$3 captures total marginal importance, $S\subseteq T$4 serves as an interaction proxy between layers $S\subseteq T$5 and $S\subseteq T$6, and $S\subseteq T$7 is the adjusted standalone sensitivity after accounting for interactions (Zhao et al., 18 Sep 2025).

4. IMPQ optimization problem and deployment pipeline

IMPQ uses a binary precision choice. Each layer is assigned either 2-bit or 4-bit precision. Let $S\subseteq T$8 be the decision variable for layer $S\subseteq T$9, where $T\setminus S$0 means the layer stays at low precision and $T\setminus S$1 means the layer is promoted to high precision. In the reported setup, low precision is 2-bit and high precision is 4-bit (Zhao et al., 18 Sep 2025).

The quantization-induced loss increase is approximated by the binary quadratic objective

$T\setminus S$2

The optimization problem is

$T\setminus S$3

where $T\setminus S$4 is the byte cost of promoting layer $T\setminus S$5 from low precision to high precision, and $T\setminus S$6 is the memory budget (Zhao et al., 18 Sep 2025).

To solve this efficiently, IMPQ linearizes the quadratic term using auxiliary variables $T\setminus S$7 for products $T\setminus S$8: $T\setminus S$9 This converts the problem into a MILP solved with SCIP (Zhao et al., 18 Sep 2025).

The reported end-to-end workflow is: choose a pretrained LLM and a PTQ backend; collect calibration data from the C4 training split; run SPQE by sampling random layer permutations and progressively quantizing layers from 4-bit to 2-bit while measuring NLL changes; construct the interaction proxy $$v(S) = \mathbb{E}_{(x,t)\sim\mathcal D} \Big[-\log p(x_{t+1}\mid x_{\le t};S)\Big].$$0, shrink it to $$v(S) = \mathbb{E}_{(x,t)\sim\mathcal D} \Big[-\log p(x_{t+1}\mid x_{\le t};S)\Big].$$1, and derive $$v(S) = \mathbb{E}_{(x,t)\sim\mathcal D} \Big[-\log p(x_{t+1}\mid x_{\le t};S)\Big].$$2; solve the binary optimization under the memory budget; and evaluate the resulting mixed-precision model on WikiText-2 (Zhao et al., 18 Sep 2025). In this pipeline, SPQE is the estimator and IMPQ is the allocator.

5. Empirical results and observed behavior

The reported empirical finding is that IMPQ is especially effective when the bit budget is tight. Across GPTQ, Quanto, and HQQ, it consistently outperforms baselines on all tested models, and the gap widens as average precision decreases from 4 bits toward 2 bits (Zhao et al., 18 Sep 2025).

The paper states that IMPQ cuts Perplexity by roughly 20% to 80% relative to the best baseline, with the largest gains at the lowest bit-widths. Under GPTQ, the reported examples include Gemma-2-2B at 2.01–2.5 bits, where IMPQ achieves 233.98 versus Sensitivity at $$v(S) = \mathbb{E}_{(x,t)\sim\mathcal D} \Big[-\log p(x_{t+1}\mid x_{\le t};S)\Big].$$3 and LIM at $$v(S) = \mathbb{E}_{(x,t)\sim\mathcal D} \Big[-\log p(x_{t+1}\mid x_{\le t};S)\Big].$$4, corresponding to more than 79% and 81% reduction. For Gemma-2-9B at 2.01–2.5 bits, IMPQ achieves 48.52 versus 189.55 and 214.03, about 74%–77% reduction. For Qwen3-4B at 2.01–2.5 bits, IMPQ achieves 697.28 versus 1560 and 2380, about 55%–71% reduction. For Llama-3.2-3B at 2.5–3.0 bits, IMPQ achieves 73.11 versus Sensitivity at 343.64, about 79% reduction. Even at 3.5–3.99 bits on Llama-3.2-3B, IMPQ reports 17.08, better than ZD at 24.84 by about 31% (Zhao et al., 18 Sep 2025).

The advantage is also reported to be backend-agnostic. On HQQ, IMPQ improves average perplexity by 32.7% on Llama-3.1-8B across all bit ranges. On Quanto, it improves by 52.3% on Gemma-2-9B across all bit ranges, and at the hardest low-bit setting on Quanto for Gemma-2-9B it reduces perplexity by about 78.5% relative to the best baseline (Zhao et al., 18 Sep 2025).

The qualitative trend is consistent across the study: as average precision drops, baselines degrade sharply whereas IMPQ degrades much more gracefully. The paper interprets this as evidence that inter-layer interactions become increasingly important in the low-bit regime (Zhao et al., 18 Sep 2025). This suggests that IMPQ’s benefit is not merely better ranking of sensitive layers, but better modeling of joint degradation pathways.

6. Relation to prior mixed-precision quantization methods

IMPQ sits within a large mixed-precision literature, but differs from earlier methods in what it treats as the primary signal for bit allocation. Learned-layer-importance methods such as LIMPQ use learned scale factors in quantizers as layer-wise importance indicators and then solve an ILP with a linear additive objective (Tang et al., 2022). Differentiable mixed-precision methods learn heterogeneous precision end-to-end through task loss plus size penalties, with per-channel or per-tensor structure and targeted gradient scaling, but they do not explicitly compute interaction-aware importance scores across tensor pairs (Schaefer et al., 2022). Hardware-friendly approaches such as HMQ use Gumbel-Softmax to search over a finite set of bit-width and threshold pairs under hardware constraints, again without explicit cross-layer interaction terms (Habi et al., 2020). Practical PTQ schemes based on SQNR rank layers or quantizer groups by standalone distortion and then perform greedy precision reduction (Pandey et al., 2023).

By contrast, IMPQ makes the interaction term explicit through the quadratic component $$v(S) = \mathbb{E}_{(x,t)\sim\mathcal D} \Big[-\log p(x_{t+1}\mid x_{\le t};S)\Big].$$5 (Zhao et al., 18 Sep 2025). That is the most direct formal distinction between interaction-aware allocation and additive layer-scoring schemes.

There are, however, important neighboring lines of work. “Mixed-Precision Inference Quantization: Radically Towards Faster inference speed, Lower Storage requirement, and Lower Loss” (Cheng et al., 2022) argues that inference loss is often dominated by noise at layer inputs caused by precision conversion between layers rather than by weight noise alone. Its key first-order approximation is

$$v(S) = \mathbb{E}_{(x,t)\sim\mathcal D} \Big[-\log p(x_{t+1}\mid x_{\le t};S)\Big].$$6

and it simplifies the expected dataset-level effect by arguing that $$v(S) = \mathbb{E}_{(x,t)\sim\mathcal D} \Big[-\log p(x_{t+1}\mid x_{\le t};S)\Big].$$7 in expectation after training. That work is interaction-aware in the sense that it emphasizes layer-to-layer conversion noise $$v(S) = \mathbb{E}_{(x,t)\sim\mathcal D} \Big[-\log p(x_{t+1}\mid x_{\le t};S)\Big].$$8, but its mechanism is different: it chooses rounding directions for inputs rather than solving a Shapley-based global allocation problem (Cheng et al., 2022).

A more recent interaction-aware direction is InfoQ, which measures how quantizing a layer perturbs the network’s global information flow by tracking downstream changes in mutual information and sliced mutual information, then formulates bit-width allocation as an ILP under model-size or BitOps budgets (Akbulut et al., 6 Aug 2025). InfoQ therefore shares the view that local criteria miss cascading downstream effects, but it operationalizes that view through observer-layer information degradation rather than cooperative-game marginal contributions (Akbulut et al., 6 Aug 2025).

Taken together, these comparisons locate IMPQ within a broader methodological shift: from local sensitivity estimation toward criteria that explicitly capture propagation, dependence, or global effect. IMPQ’s specific contribution is to make those dependencies first-class optimization variables in LLM layerwise 2-bit/4-bit allocation (Zhao et al., 18 Sep 2025).

7. Strengths, limitations, and interpretation

The principal strengths attributed to IMPQ are its interaction-aware formulation, its grounding in cooperative game theory via Shapley values, the stability of SPQE relative to pruning-style estimation, its strong empirical performance across models and PTQ backends, and its globally optimized bit assignment under a budget rather than a greedy policy (Zhao et al., 18 Sep 2025). These properties distinguish it from methods that either learn or rank layers independently and then aggregate scores additively.

The reported limitations are equally explicit. SPQE is computationally expensive; the paper reports roughly 18 hours on a single A40 GPU for Llama-3.1-8B. Estimation cost scales with the number of sampled permutations. The precision choice is binary—only 2-bit and 4-bit are considered. The interaction proxy based on covariance and shrinkage is a practical surrogate rather than an exact decomposition. The method also depends on representative calibration data (Zhao et al., 18 Sep 2025).

A common misconception is that “interaction-aware” simply means using a more refined sensitivity metric for each layer. In the IMPQ sense, that is not sufficient. Interaction-awareness means that the quality of assigning a bit-width to one layer depends on assignments made elsewhere, and that those dependencies appear directly in the allocation model rather than only indirectly in training dynamics or calibration heuristics (Zhao et al., 18 Sep 2025). This is why IMPQ requires both SPQE and a quadratic optimization stage: without the former, interactions are not estimated; without the latter, they are not enforced in the final allocation.

A second misconception is that all mixed-precision benefits arise from preserving the most sensitive layers. Prior work shows that layer importance can indeed be estimated by learned indicators, SQNR, or differentiable proxies (Tang et al., 2022, Pandey et al., 2023, Schaefer et al., 2022). IMPQ does not negate those observations; instead, it argues that below 4-bit average precision they are incomplete because marginal damage is context-dependent (Zhao et al., 18 Sep 2025). The framework’s reported empirical pattern—an increasingly large advantage as precision tightens—is presented as evidence for that claim.

In the current literature, IMPQ can therefore be understood as a layerwise mixed-precision method specialized for ultra-low-bit LLM PTQ, whose defining feature is the explicit modeling of inter-layer dependence through Shapley-estimated marginal effects and quadratic budget-constrained optimization (Zhao et al., 18 Sep 2025).