---
title: 'MergeQuant: Static 4-bit Quantization for LLMs'
url: https://www.emergentmind.com/topics/mergequant
type: topic
---

# MergeQuant: Static 4-bit Quantization for LLMs

Searching arXiv for MergeQuant and closely related quantization/model-merging literature.
MergeQuant is an accurate and efficient per-channel static quantization framework for large language models that targets the specific inefficiency of repeated dynamic quantization and dequantization during autoregressive generation. It integrates per-channel quantization steps with the corresponding scalings and linear mappings through Quantization Step Migration (QSM), and complements that mechanism with dimensional reconstruction and adaptive clipping to address the non-uniformity of channel ranges under 4-bit static quantization. In the static quantization setting of W4A4, it reduces the accuracy gap on zero-shot tasks compared to the FP16 baseline to 1.3 points on Llama-2-70B, while on Llama-2-7B it achieves up to 1.77x speedup in decoding and up to 2.06x speedup end-to-end [2503.07654].

## 1. Problem formulation and motivation

MergeQuant is situated in the broader effort to compress and accelerate inference of large language models through quantization, with particular emphasis on the distinction between dynamic and static calibration. Existing methods focus on exploring per-token dynamic calibration to ensure both inference acceleration and model accuracy under 4-bit quantization, but in autoregressive generation inference of long sequences the overhead of repeated dynamic quantization and dequantization becomes considerably expensive [2503.07654].

The method is motivated by the observation that naïve static quantization fails at 4 bits when calibration is not sufficiently sensitive to channel structure. The detailed account identifies two failure modes. First, activation outliers are concentrated in only a subset of channels, so per-tensor or per-token static quantization lets those outliers distort the scale for the entire tensor, drastically reducing effective precision for the majority of channels. Second, the required scale for each channel varies greatly; a single scale cannot capture this diversity, leading to substantial quantization errors. The paper’s evidence, as summarized in the extracted details, is that only per-channel calibration preserves accuracy for 4-bit quantization, whereas per-tensor and per-token calibration lose substantial performance even with rotation-based smoothing [2503.07654].

This suggests that MergeQuant’s central contribution is not merely a lower-bit representation, but a static calibration scheme designed to retain the accuracy properties usually associated with more expensive dynamic quantization. A plausible implication is that the method addresses both algorithmic and systems bottlenecks: quantization error induced by scale mismatch, and runtime overhead induced by online quantize–dequantize operations.

## 2. Channel-wise static quantization and Quantization Step Migration

The framework uses per-channel quantization for activations with strong outlier structure, computes scaling factors offline for each channel, and merges these quantization operations into existing model layers. Its most distinctive mechanism is Quantization Step Migration, which “migrates” quantization and dequantization into adjacent operations so that there are no explicit quantization or dequantization steps during inference [2503.07654].

The basic quantization relation is given as
\[
X_{\text{Int}} = \text{round}\left( \frac{X_{\text{FP}}}{s} \right),
\]
where \(s\) is the quantization scale factor, taken per-channel in MergeQuant, and for \(b=4\),
\[
s = \frac{\max(|X_{\text{FP}}|)}{2^{b-1} - 1}.
\]

Within QSM, quantization migration is described for RMSNorm. For channel \(k\), the RMSNorm output is
\[
\tilde{X}_k = \frac{X_k}{\text{RMS}(X)} \cdot \gamma_k,
\]
and MergeQuant merges quantization into this step as
\[
\tilde{X}_{k,\text{Int}} = \frac{X_k}{s_k},
\]
where \(s_k\) is the pre-computed per-channel scale from calibration statistics. The same migration principle is said to apply to LayerNorm by dividing scale and offset by \(s_k\) ahead of quantization [2503.07654].

Dequantization migration is formulated for linear mappings. In standard per-channel quantization for a linear layer,
\[
Y_{ij} = \sum_{k=1}^n X_{\text{Int},ik} \cdot W_{\text{Int},kj} \cdot s_k s_{W_j}.
\]
Because the scales are entangled inside the sum, efficient integer matrix multiplication is impeded. QSM rewrites the computation as
\[
Y_{ij} = \sum_{k=1}^n X_{\text{Int},ik} \cdot \left(\frac{W_{kj}}{s_k}\right),
\]
so that \(\frac{1}{s_k}\) is folded into the weight matrix. The extracted description states that this transformation enables the use of efficient integer kernels and can be precomputed so that inference reduces to integer GEMM plus a single scale per output channel [2503.07654].

Factually, the method’s name derives from this folding operation: it “merges” quantization steps with neighboring computations. In that sense, MergeQuant is a static quantization method whose merge operation is algebraic rather than a model-merging procedure over checkpoints.

## 3. Dimensional reconstruction and adaptive clipping

A core difficulty of per-channel static quantization is the non-uniformity of scale factors across channels. MergeQuant addresses this with dimensional reconstruction and adaptive clipping, both introduced to redistribute channel variations to subsequent modules and balance the parameter distribution under QSM [2503.07654].

Dimensional reconstruction temporarily splits strong scaling factors to restrict their numerical range while preserving hardware-friendly tensor shapes. The extracted procedure identifies channels whose scaling factors \(s_k\) exceed a threshold \(T\), and splits them repeatedly until the pieces are below that threshold. The threshold is defined as
\[
T = \mu(s) + \alpha \cdot \sigma(s),
\]
with \(\alpha\) a hyperparameter and \(\mu\), \(\sigma\) the mean and variance of the scaling factors. After splitting, Hessian sensitivity is used to prune unimportant neighbor channels so that the activation dimension remains compatible with hardware constraints such as multiples of 8 or 16 [2503.07654].

Adaptive clipping is then applied per channel during calibration. Its role is to search for the optimal clipping factor for each channel after dimensional reconstruction, specifically to reduce the impact of large value ranges and outliers. The loss used in this calibration search is
\[
L_i(s) = \| X_i(s) - \tilde{X}_i(s) \|_1 + \| W_X - \tilde{W}_X \|_1,
\]
where \(X_i(s)\) and \(W_X\) denote the original migrated activations and weights, and \(\tilde{X}_i(s)\), \(\tilde{W}_X\) their quantized counterparts. The clipping value per channel is selected to minimize this loss [2503.07654].

These components are not ancillary. The extracted account explicitly ties them to “the significant differences between the different channel ranges,” and presents them as the mechanism for handling the scale heterogeneity that would otherwise undermine static W4A4 quantization [2503.07654].

## 4. Optional quantization compensation and practical calibration workflow

Beyond QSM, dimensional reconstruction, and adaptive clipping, MergeQuant includes an optional quantization compensation stage. This stage learns low-rank parameters to compensate for residual quantization-induced error after migration and calibration. The extracted formulation introduces low-rank matrices \(A \in \mathbb{R}^{m \times r}\) and \(B \in \mathbb{R}^{r \times n}\) and minimizes
\[
\min_{A,B} \| Y_{\text{true}} - Y_{\text{quant}} \|^2,
\]
with
\[
W_{\text{quant}} = \text{Q}(W + AB).
\]
The summary characterizes this as a LoRA-style post-training compensation step [2503.07654].

The operational workflow implied by the extracted description is a static offline pipeline. First, per-channel scales are estimated from a calibration set. Second, QSM integrates those scales into normalization and linear layers so that quantization and dequantization do not appear as separate runtime operators. Third, dimensional reconstruction reshapes problematic channel-scale distributions. Fourth, adaptive clipping searches per-channel clipping ratios. Fifth, optional low-rank compensation further reduces the remaining discrepancy between true and quantized outputs [2503.07654].

A practical implication is that MergeQuant treats calibration not as a lightweight postscript but as the central design locus of 4-bit static quantization. The extracted timing numbers reinforce that view: full quantization of Llama-3-70B with MergeQuant is reported as 44.5 minutes for calibration and 9.3 hours for full quantization [2503.07654].

## 5. Empirical performance and systems characteristics

The paper’s headline empirical claim is that MergeQuant makes W4A4 static quantization competitive in both accuracy and efficiency. On Llama-2-70B, the accuracy gap on zero-shot tasks relative to FP16 is reduced to 1.3 points. On Llama-2-13B, the reported gap is 2.4 points. For Llama-2-7B, perplexity is described as close to FP16, with an example of 6.09 versus 5.47 on WikiText-2 [2503.07654].

The systems gains are equally prominent. On Llama-2-7B using an RTX 3090, MergeQuant achieves up to 1.77x speedup in decoding, up to 2.06x end-to-end speedup, and up to 2.72x prefill speedup for large batch sizes compared to FP16. Memory is reduced by 3.58x relative to FP16 for W4A4 [2503.07654].

The extracted comparison table for Llama-2-7B gives a concrete illustration. FP16 has PIQA accuracy 79.05, decoding speed 1.00, end-to-end speed 1.00, and memory 13.87 GB. QuaRot, listed as dynamic, reaches PIQA 76.33, decoding 1.50, end-to-end 1.25, and memory 4.16 GB. MergeQuant, listed as static, reaches PIQA 76.71, decoding 1.77, end-to-end 2.06, and memory 3.87 GB [2503.07654].

The ablation account is especially important for interpreting these numbers. Naïve static per-tensor quantization is described as much worse, with the example that PIQA drops from above 80 in FP16 to 62 under static per-tensor quantization. MergeQuant, with QSM, adaptive clipping, and optional LoRA compensation, reportedly brings more than a 10-point improvement over such static per-tensor approaches and closes much of the FP16 gap [2503.07654].

Taken together, the reported results position MergeQuant as a static alternative to dynamic 4-bit inference that retains much of the latter’s accuracy while removing its repeated online calibration overhead.

## 6. Scope, interpretation, and relation to adjacent uses of the term

In the literature summarized here, “MergeQuant” has a specific meaning in the context of large language model quantization: accurate 4-bit static quantization by channel-wise calibration and migration of quantization steps into neighboring operators [2503.07654]. This is distinct from a separate line of work on quantization-aware model merging for multi-target domain adaptation.

That distinction matters because another paper uses the phrase “merge-friendly quantization (MergeQuant)” to describe practical guidelines around HDRQ, a post-training quantization framework for model merging under multi-target domain adaptation. In that setting, the problem is not static W4A4 calibration for LLM inference, but quantization that preserves alignment for subsequent weight-space merging across target-adapted models. The key mechanisms there are Hessian regularization via noise-based quantization, distance regularization toward source weights, and merge-aware noise sampling for rounding ambiguity [2505.23651].

A common misconception is therefore to treat all “MergeQuant” references as referring to the same method family. The available evidence indicates two distinct usages. One is the named LLM quantization framework based on QSM, dimensional reconstruction, and adaptive clipping [2503.07654]. The other is a descriptive label for merge-friendly post-training quantization in multi-target domain adaptation, centered on HDRQ rather than the QSM-based static calibration framework [2505.23651].

This suggests that the semantic overlap lies in the idea of making quantization compatible with a downstream objective that would otherwise be disrupted by naïve low-precision arithmetic. In the LLM case, the downstream objective is efficient autoregressive inference under static W4A4. In the domain-adaptation case, it is smooth checkpoint merging after quantization. The technical realizations, however, are different.

## 7. Significance within low-bit LLM deployment

MergeQuant addresses a concrete deployment problem: static 4-bit quantization is attractive because it eliminates online quantization overhead, but without channel-wise calibration it suffers severe accuracy degradation. The method’s contribution is to combine per-channel static scaling, QSM, dimensional reconstruction, adaptive clipping, and optional low-rank compensation into a framework that preserves accuracy sufficiently well to make W4A4 practical for large models [2503.07654].

The extracted summary emphasizes several deployment-facing consequences. The method runs fully statically, requires no runtime quantization or dequantization, aligns with INT4 GEMM kernels, scales to models such as Llama-2-70B, Llama-2-13B, Llama-2-7B, and Llama-3-8B/70B, and enables “meaningful compression,” including the claim that it allows deployment of Llama-2-70B in 40GB with nearly full accuracy [2503.07654].

For technically oriented readers, the larger importance of MergeQuant lies in the specific design principle it exemplifies: quantization error and runtime overhead must be treated jointly. The method does not merely calibrate channel scales more carefully; it restructures the computation graph so that those scales become part of the surrounding linear algebra. A plausible implication is that its strongest contribution is architectural in the systems sense: it converts a numerically motivated calibration choice into an efficient execution pattern compatible with static low-bit inference.

Source: https://www.emergentmind.com/topics/mergequant