---
title: Interaction-aware Mixed-Precision Quantization
url: https://www.emergentmind.com/topics/interaction-aware-mixed-precision-quantization-impq
type: topic
---

# Interaction-aware Mixed-Precision Quantization

Searching arXiv for IMPQ and closely related mixed-precision quantization work to ground the article.
Interaction-aware Mixed-Precision Quantization (IMPQ) is a mixed-precision quantization framework for large language models (LLMs) that treats bit allocation as an explicitly interaction-aware optimization problem rather than as a collection of isolated layerwise decisions. In the formulation reported for IMPQ, the central premise is that once average precision drops below 4 bits, quantization errors cease to behave independently across layers: quantizing one layer can amplify or suppress the degradation induced by quantizing another. IMPQ therefore models Transformer layers as players in a cooperative game, estimates both layer sensitivities and inter-layer interactions through Shapley-based Progressive Quantization Estimation (SPQE), and then solves a binary quadratic optimization problem that assigns either 2-bit or 4-bit precision under a strict memory budget [2509.15455].

## 1. Conceptual basis and scope

IMPQ is motivated by a failure mode of conventional mixed-precision quantization in the ultra-low-bit regime. Methods based on isolated layerwise criteria—such as gradient sensitivity, activation norms, Z-score statistics, or cosine modification measures—implicitly assume that each layer’s quantization impact is mostly independent. The IMPQ formulation rejects that assumption for sub-4-bit operation and instead treats quantization as interaction-dominated: the marginal effect of quantizing layer \(i\) depends on which other layers have already been quantized [2509.15455].

The framework is defined for LLM post-training quantization. Its reported experiments cover Llama-3, Gemma-2, and Qwen-3 models, and it is evaluated across three independent PTQ backends: Quanto, HQQ, and GPTQ. The main evaluation metric is Perplexity on WikiText-2, while calibration and bit-allocation optimization use the C4 training split [2509.15455].

This suggests that IMPQ belongs to a broader transition in mixed-precision research away from purely local sensitivity proxies. Earlier work typically framed layer importance through learned scale factors, Hessian surrogates, SQNR rankings, or differentiable bit learning [2203.08368], [2302.05397], [2206.07741]. IMPQ’s distinctive claim is that these layerwise views become systematically incomplete when average precision is pushed far enough downward [2509.15455].

## 2. Cooperative-game formulation of layer interactions

IMPQ formulates Transformer layers as players in a cooperative game. Let \(T=\{1,2,\dots,L\}\) be the set of layers, and let \(S\subseteq T\) denote the layers kept at high precision, while layers in \(T\setminus S\) are quantized to low precision. The payoff is defined as the average per-token negative log-likelihood:
\[
v(S) = \mathbb{E}_{(x,t)\sim\mathcal D} \Big[-\log p(x_{t+1}\mid x_{\le t};S)\Big].
\]
Within this formulation, a layer’s importance is not its standalone sensitivity under a single configuration, but its average marginal contribution across quantization contexts [2509.15455].

The corresponding Shapley value for layer \(i\) is
\[
\phi_i = \sum_{S \subseteq N \setminus \{i\}} \frac{|S|!(n-|S|-1)!}{n!} \Big(v(S\cup\{i\})-v(S)\Big).
\]
This quantity averages the effect of adding layer \(i\) over all coalitions not containing it. In the IMPQ interpretation, a large Shapley value indicates that moving layer \(i\) from high precision to low precision tends to produce a large loss increase across many surrounding configurations [2509.15455].

The significance of this construction is methodological. Mixed-precision allocation is no longer reduced to ranking layers by a single local metric; it is reframed as estimating marginal degradation in a combinatorial space of jointly quantized models. A plausible implication is that the framework is particularly well suited to settings where error propagation is nonlinear and strongly architecture-dependent.

## 3. SPQE: Shapley-based Progressive Quantization Estimation

Direct computation of exact Shapley values is infeasible for LLMs, so IMPQ introduces SPQE, or Shapley-based Progressive Quantization Estimation, as an approximation procedure based on Monte Carlo sampling over random permutations [2509.15455].

A key design choice is progressive quantization rather than pruning. SPQE starts from a uniformly high-precision model and progressively lowers the precision of layers one by one along a sampled permutation. If \(\pi=(\pi_1,\pi_2,\dots,\pi_L)\) is a random permutation and \(S_\ell\) is the set of layers still kept at high precision after the first \(\ell-1\) quantizations, then
\[
S_{\ell+1}=\{\pi_{\ell+1},\dots,\pi_L\}.
\]
When layer \(\pi_\ell\) is quantized, its marginal contribution is
\[
\Delta v_{\ell} = v(S_{\ell})-v(S_{\ell}\setminus\{\pi_\ell\}) = v(S_\ell)-v(S_{\ell+1}).
\]
Across \(M\) sampled permutations, the Shapley estimate for layer \(i\) is
\[
\hat{\phi}_i = \frac{1}{M}\sum_{m=1}^{M}\Delta v_i^{(m)}.
\]

The practical rationale is explicit: progressive quantization produces smoother, more stable changes in negative log-likelihood than abrupt pruning, thereby reducing estimator noise [2509.15455]. The method is therefore designed to measure each layer in many different contexts of previously quantized layers, which is precisely where interaction structure enters the estimate.

SPQE also supplies the raw material for pairwise interaction modeling. The empirical marginal contributions are assembled into a covariance-like matrix
\[
\mathbf C = \frac{1}{M} \bigl(\Delta v_{\ell}-\hat{\boldsymbol{\phi}}\bigr)^{\top} \bigl(\Delta v_{\ell}-\hat{\boldsymbol{\phi}}\bigr),
\]
where \(\hat{\boldsymbol{\phi}}=[\hat{\phi}_1,\dots,\hat{\phi}_L]\). To reduce noise, the method applies diagonal shrinkage:
\[
\mathbf K = (1-\alpha)\mathbf C+\alpha\,\operatorname{diag}(\mathbf C),
\]
with \(\alpha\in[0,1]\). It then defines an adjusted first-order sensitivity term by subtracting interaction contributions:
\[
\mathbf a_i = \hat{\phi}_i - \sum_{j\neq i}K_{ij}.
\]
In this decomposition, \(\hat{\phi}_i\) captures total marginal importance, \(K_{ij}\) serves as an interaction proxy between layers \(i\) and \(j\), and \(a_i\) is the adjusted standalone sensitivity after accounting for interactions [2509.15455].

## 4. IMPQ optimization problem and deployment pipeline

IMPQ uses a binary precision choice. Each layer is assigned either 2-bit or 4-bit precision. Let \(q_i\in\{0,1\}\) be the decision variable for layer \(i\), where \(q_i=1\) means the layer stays at low precision and \(q_i=0\) means the layer is promoted to high precision. In the reported setup, low precision is 2-bit and high precision is 4-bit [2509.15455].

The quantization-induced loss increase is approximated by the binary quadratic objective
\[
\Delta L(\mathbf q)=\mathbf a^\top \mathbf q+\mathbf q^\top \mathbf K \mathbf q.
\]
The optimization problem is
\[
\begin{aligned}
\min_{\mathbf q\in\{0,1\}^L} \quad & \mathbf a^\top \mathbf q+\mathbf q^\top \mathbf K \mathbf q \\
\text{s.t.} \quad & \sum_{i=1}^L c_i(1-q_i)\le \mathbf B,
\end{aligned}
\]
where \(c_i\) is the byte cost of promoting layer \(i\) from low precision to high precision, and \(\mathbf B\) is the memory budget [2509.15455].

To solve this efficiently, IMPQ linearizes the quadratic term using auxiliary variables \(y_{ij}\) for products \(q_i q_j\):
\[
y_{ij} \ge q_i+q_j-1,\quad y_{ij}\le q_i,\quad y_{ij}\le q_j,\quad y_{ij}\in\{0,1\}.
\]
This converts the problem into a MILP solved with SCIP [2509.15455].

The reported end-to-end workflow is: choose a pretrained LLM and a PTQ backend; collect calibration data from the C4 training split; run SPQE by sampling random layer permutations and progressively quantizing layers from 4-bit to 2-bit while measuring NLL changes; construct the interaction proxy \(\mathbf C\), shrink it to \(\mathbf K\), and derive \(\mathbf a\); solve the binary optimization under the memory budget; and evaluate the resulting mixed-precision model on WikiText-2 [2509.15455]. In this pipeline, SPQE is the estimator and IMPQ is the allocator.

## 5. Empirical results and observed behavior

The reported empirical finding is that IMPQ is especially effective when the bit budget is tight. Across GPTQ, Quanto, and HQQ, it consistently outperforms baselines on all tested models, and the gap widens as average precision decreases from 4 bits toward 2 bits [2509.15455].

The paper states that IMPQ cuts Perplexity by roughly 20% to 80% relative to the best baseline, with the largest gains at the lowest bit-widths. Under GPTQ, the reported examples include Gemma-2-2B at 2.01–2.5 bits, where IMPQ achieves 233.98 versus Sensitivity at \(1.12\times10^3\) and LIM at \(1.25\times10^3\), corresponding to more than 79% and 81% reduction. For Gemma-2-9B at 2.01–2.5 bits, IMPQ achieves 48.52 versus 189.55 and 214.03, about 74%–77% reduction. For Qwen3-4B at 2.01–2.5 bits, IMPQ achieves 697.28 versus 1560 and 2380, about 55%–71% reduction. For Llama-3.2-3B at 2.5–3.0 bits, IMPQ achieves 73.11 versus Sensitivity at 343.64, about 79% reduction. Even at 3.5–3.99 bits on Llama-3.2-3B, IMPQ reports 17.08, better than ZD at 24.84 by about 31% [2509.15455].

The advantage is also reported to be backend-agnostic. On HQQ, IMPQ improves average perplexity by 32.7% on Llama-3.1-8B across all bit ranges. On Quanto, it improves by 52.3% on Gemma-2-9B across all bit ranges, and at the hardest low-bit setting on Quanto for Gemma-2-9B it reduces perplexity by about 78.5% relative to the best baseline [2509.15455].

The qualitative trend is consistent across the study: as average precision drops, baselines degrade sharply whereas IMPQ degrades much more gracefully. The paper interprets this as evidence that inter-layer interactions become increasingly important in the low-bit regime [2509.15455]. This suggests that IMPQ’s benefit is not merely better ranking of sensitive layers, but better modeling of joint degradation pathways.

## 6. Relation to prior mixed-precision quantization methods

IMPQ sits within a large mixed-precision literature, but differs from earlier methods in what it treats as the primary signal for bit allocation. Learned-layer-importance methods such as LIMPQ use learned scale factors in quantizers as layer-wise importance indicators and then solve an ILP with a linear additive objective [2203.08368]. Differentiable mixed-precision methods learn heterogeneous precision end-to-end through task loss plus size penalties, with per-channel or per-tensor structure and targeted gradient scaling, but they do not explicitly compute interaction-aware importance scores across tensor pairs [2206.07741]. Hardware-friendly approaches such as HMQ use Gumbel-Softmax to search over a finite set of bit-width and threshold pairs under hardware constraints, again without explicit cross-layer interaction terms [2007.09952]. Practical PTQ schemes based on SQNR rank layers or quantizer groups by standalone distortion and then perform greedy precision reduction [2302.05397].

By contrast, IMPQ makes the interaction term explicit through the quadratic component \(\mathbf q^\top \mathbf K \mathbf q\) [2509.15455]. That is the most direct formal distinction between interaction-aware allocation and additive layer-scoring schemes.

There are, however, important neighboring lines of work. “Mixed-Precision Inference Quantization: Radically Towards Faster inference speed, Lower Storage requirement, and Lower Loss” [2207.10083] argues that inference loss is often dominated by noise at layer inputs caused by precision conversion between layers rather than by weight noise alone. Its key first-order approximation is
\[
\bar{\ell}(w,x_i,y_i) - \ell(w,x_i,y_i) = \sum_{i=1}^n \frac{\partial \ell}{\partial h_{i+1}} \cdot \epsilon_i + \frac{\partial \ell}{\partial w_i} \cdot \delta_i,
\]
and it simplifies the expected dataset-level effect by arguing that \(\frac{\partial \ell}{\partial w_i}\approx 0\) in expectation after training. That work is interaction-aware in the sense that it emphasizes layer-to-layer conversion noise \(\epsilon_i\), but its mechanism is different: it chooses rounding directions for inputs rather than solving a Shapley-based global allocation problem [2207.10083].

A more recent interaction-aware direction is InfoQ, which measures how quantizing a layer perturbs the network’s global information flow by tracking downstream changes in mutual information and sliced mutual information, then formulates bit-width allocation as an ILP under model-size or BitOps budgets [2508.04753]. InfoQ therefore shares the view that local criteria miss cascading downstream effects, but it operationalizes that view through observer-layer information degradation rather than cooperative-game marginal contributions [2508.04753].

Taken together, these comparisons locate IMPQ within a broader methodological shift: from local sensitivity estimation toward criteria that explicitly capture propagation, dependence, or global effect. IMPQ’s specific contribution is to make those dependencies first-class optimization variables in LLM layerwise 2-bit/4-bit allocation [2509.15455].

## 7. Strengths, limitations, and interpretation

The principal strengths attributed to IMPQ are its interaction-aware formulation, its grounding in cooperative game theory via Shapley values, the stability of SPQE relative to pruning-style estimation, its strong empirical performance across models and PTQ backends, and its globally optimized bit assignment under a budget rather than a greedy policy [2509.15455]. These properties distinguish it from methods that either learn or rank layers independently and then aggregate scores additively.

The reported limitations are equally explicit. SPQE is computationally expensive; the paper reports roughly 18 hours on a single A40 GPU for Llama-3.1-8B. Estimation cost scales with the number of sampled permutations. The precision choice is binary—only 2-bit and 4-bit are considered. The interaction proxy based on covariance and shrinkage is a practical surrogate rather than an exact decomposition. The method also depends on representative calibration data [2509.15455].

A common misconception is that “interaction-aware” simply means using a more refined sensitivity metric for each layer. In the IMPQ sense, that is not sufficient. Interaction-awareness means that the quality of assigning a bit-width to one layer depends on assignments made elsewhere, and that those dependencies appear directly in the allocation model rather than only indirectly in training dynamics or calibration heuristics [2509.15455]. This is why IMPQ requires both SPQE and a quadratic optimization stage: without the former, interactions are not estimated; without the latter, they are not enforced in the final allocation.

A second misconception is that all mixed-precision benefits arise from preserving the most sensitive layers. Prior work shows that layer importance can indeed be estimated by learned indicators, SQNR, or differentiable proxies [2203.08368], [2302.05397], [2206.07741]. IMPQ does not negate those observations; instead, it argues that below 4-bit average precision they are incomplete because marginal damage is context-dependent [2509.15455]. The framework’s reported empirical pattern—an increasingly large advantage as precision tightens—is presented as evidence for that claim.

In the current literature, IMPQ can therefore be understood as a layerwise mixed-precision method specialized for ultra-low-bit LLM PTQ, whose defining feature is the explicit modeling of inter-layer dependence through Shapley-estimated marginal effects and quadratic budget-constrained optimization [2509.15455].

Source: https://www.emergentmind.com/topics/interaction-aware-mixed-precision-quantization-impq