AutoRound: Adaptive Rounding in PTQ

• AutoRound is an adaptive quantization method that learns data-driven rounding decisions to minimize task loss rather than per-weight error.
• It leverages a calibration protocol with layerwise optimization using soft rounding relaxations and continuous variables to improve accuracy.
• Variants like AdaRound and VQRound adapt the concept for different scales, with VQRound scaling rounding to billion-parameter LLMs via vector-quantized codebooks.

Searching arXiv for papers directly about AutoRound and closely related adaptive rounding methods in post-training quantization. Search query: "AutoRound post-training quantization LLM arXiv adaptive rounding" AutoRound, in later usage, is often associated with the same idea as adaptive, optimization-based rounding for post-training quantization (PTQ): learn rounding-up versus rounding-down decisions from calibration data by optimizing a reconstruction or loss proxy rather than using fixed nearest rounding. In that sense, it is most naturally situated in the methodological lineage of AdaRound, which replaces scalar round-to-nearest by a layerwise optimization over binary rounding choices, and in later LLM-oriented variants such as VQRound, which revisit adaptive rounding from an efficiency perspective for billion-parameter models (Nagel et al., 2020, Zhou et al., 2 Feb 2026).

1. Terminological scope and research lineage

The core conceptual shift associated with AutoRound is the claim that, for PTQ, rounding each floating-point weight to its nearest fixed-point value is not generally optimal. The motivating observation is that nearest rounding minimizes immediate scalar error for each weight, but does not align with the actual PTQ objective of preserving model performance, because quantization errors interact through the network and through the task loss (Nagel et al., 2020).

In this usage, AutoRound is best understood as a family resemblance rather than a single immutable algorithmic specification. The canonical formulation in the supplied literature is AdaRound, which uses a small amount of unlabelled calibration data, does not require fine-tuning of the network, and turns weight rounding into a data-driven optimization problem (Nagel et al., 2020). VQRound preserves the same high-level aim—improve over round-to-nearest by learning rounding decisions rather than fixing them greedily—but changes the parameterization so that adaptive rounding becomes feasible for LLMs (Zhou et al., 2 Feb 2026).

Method	Parameterization	Reported emphasis
AdaRound	Element-wise soft relaxation of floor/ceil decisions	PTQ without fine-tuning
VQRound	Vector-quantized codebook for the rounding matrix	Scalable adaptive rounding for LLMs

A plausible implication is that the term “AutoRound” is most precise when used to denote adaptive, calibration-driven rounding mechanisms in PTQ, especially those that exploit cross-element error cancellation rather than per-weight local proximity alone.

2. Optimization problem underlying adaptive rounding

The standard PTQ rule discussed in the AdaRound formulation is

$$\widehat{w} = s \cdot \mathrm{clip}\!\left(\left\lfloor \frac{w}{s}\right\rceil, n, p\right),$$

where $s$ is the quantization scale and $n,p$ are the clipping thresholds. AdaRound argues that this “round-to-nearest” rule is not generally optimal because the relevant objective is task-loss preservation, not per-weight distortion minimization (Nagel et al., 2020).

The analysis begins from a second-order approximation of the task loss:

$$\mathcal{L}(x,y,w+\Delta w)-\mathcal{L}(x,y,w) \approx \Delta w^T g_w + \frac{1}{2} \Delta w^T H_w \Delta w,$$

with

$$g_w=\nabla_w \mathcal{L}(x,y,w), \qquad H_w=\nabla^2_w \mathcal{L}(x,y,w).$$

At convergence, the gradient term is taken to be small, so the loss increase is dominated by the quadratic form. Because each weight can only be rounded to one of two values,

$$\widehat{w}_i^{(\ell)} \in \left\{ w_i^{(\ell),\mathrm{floor}},\, w_i^{(\ell),\mathrm{ceil}} \right\},$$

the resulting optimization is a Quadratic Unconstrained Binary Optimization (QUBO) problem, which is NP-hard (Nagel et al., 2020).

To obtain a tractable surrogate, AdaRound adopts a block-diagonal Hessian approximation across layers and then a local diagonal-Hessian approximation, yielding a layerwise mean-squared reconstruction objective. For fully connected layers, this reduces to

$$\argmin_{\Delta W^{(\ell)}} \left\| \Delta W^{(\ell)} x^{(\ell-1)} \right\|_F^2.$$

For convolutional layers, the appendix is stated to arrive at the same structure, separated by output channel (Nagel et al., 2020).

The discrete optimization is then relaxed by introducing continuous variables $V_{i,j}$ and parameterizing quantized weights as

$$\widehat{W} = s \cdot \mathrm{clip}\!\left(\left\lfloor \frac{W}{s} + h(V)\right\rfloor, n, p\right),$$

where

$$h(V_{i,j}) = \mathrm{clip}\!\left(\sigma(V_{i,j})(\zeta-\gamma)+\gamma,\;0,\;1\right),$$

with $s$0 and $s$1. Binary convergence is encouraged by

$s$2

with $s$3 annealed during optimization. The final local objective is

$s$4

which is the objective optimized by AdaRound (Nagel et al., 2020).

3. Calibration protocol and layerwise procedure

Adaptive rounding in the AdaRound sense is applied sequentially, layer by layer. Each layer is optimized while earlier layers are already quantized, so the reconstruction objective is evaluated under the progressively quantized upstream signal rather than against the original full-precision activations alone (Nagel et al., 2020).

The practical procedure is reported as follows: choose the quantization grid for each layer and fix the scale $s$5 in advance; absorb batch normalization into adjacent weights; quantize layers sequentially; replace each rounding decision with a soft variable $s$6; optimize the reconstruction loss plus regularizer using Adam; anneal $s$7; and finally take the resulting $s$8 values close to 0 or 1 as the final rounding choices (Nagel et al., 2020).

A defining property of the method is that it uses only a small number of unlabelled calibration examples. No labels are required because the objective measures reconstruction error of layer outputs or preactivations rather than supervised task loss. The reported calibration robustness is that even 256 images can get within about 2% of FP32 accuracy on ImageNet, and using images from a related domain can still work well with only minor degradation (Nagel et al., 2020).

For a typical ResNet-18 run, the reported setup is 1024 unlabeled ImageNet images, Adam with default settings, 10k iterations, and batch size 32. The reported runtime is roughly 10 minutes on a single GTX 1080 Ti for ResNet-18. The same source also states that the method does not jointly optimize the step size and rounding, since changing the scale changes the discrete optimization problem (Nagel et al., 2020).

The paper additionally compares the continuous relaxation against direct discrete solvers. In the appendix, a cross-entropy method is said to outperform qbsolv, largely because it can be initialized near nearest rounding and is more effective in this problem, while STE-based direct optimization is reported to be worse than AdaRound, likely due to biased gradients in this restricted binary setting (Nagel et al., 2020).

4. Empirical behavior at low bit-width

The empirical claim most central to the AutoRound lineage is that moving from nearest rounding to a loss-aware rounding strategy produces a large accuracy gain in low-bit PTQ. For 4-bit quantization of only the first layer of ResNet-18, the reported accuracies are 52.29% for Nearest, 68.62% for Hessian-based task-loss optimization, 69.39% for Local MSE loss, and 69.58% for Continuous AdaRound relaxation. For all layers of ResNet-18, the reported numbers are 23.99% for Nearest, 65.83% for Local MSE loss, and 66.56% for Continuous AdaRound relaxation (Nagel et al., 2020).

On ImageNet classification, AdaRound reports near-FP32 accuracy at 4-bit weight precision with 32-bit activations (Nagel et al., 2020):

Model	FP32	AdaRound 4-bit weights / 32-bit activations
ResNet-18	69.68%	68.71%
ResNet-50	76.07%	75.23%
InceptionV3	77.40%	75.76%
MobileNetV2	71.72%	69.78%

The same source states that, without fine-tuning, weights of ResNet18 and ResNet50 can be quantized to 4 bits while staying within an accuracy loss of 1%. It also notes that when activations are quantized to 8 bits, AdaRound remains competitive and typically only slightly degrades accuracy (Nagel et al., 2020).

The method is also reported beyond image classification. On DeeplabV3+ with MobileNetV2 backbone on Pascal VOC, the reported mIoU values are 72.94 for FP32, 6.09 for Nearest 4/8, 14.45 for DFQ 4/8, 70.89 for AdaRound 4/32, and 70.86 for AdaRound 4/8. This suggests that adaptive rounding is not limited to top-1 classification and can transfer to dense prediction tasks (Nagel et al., 2020).

5. Scaling adaptive rounding to LLMs

The main obstacle for applying dense adaptive rounding to LLMs is the cost of learning an element-wise rounding variable for every weight. VQRound explicitly states that dense and element-wise rounding matrices are prohibitively expensive for billion-parameter LLMs, even though adaptive rounding is attractive because it can exploit cross-weight error cancellation (Zhou et al., 2 Feb 2026).

VQRound reparameterizes the latent rounding matrix into a vector-quantized codebook. The latent matrix $s$9 is partitioned into vectors $n,p$0, represented by a codebook

$n,p$1

and each vector is assigned to its nearest codeword. Only the codebook vectors are trainable; the assignment indices are fixed after initialization. The resulting optimization uses $n,p$2 parameters instead of $n,p$3 free parameters for dense adaptive rounding (Zhou et al., 2 Feb 2026).

The theoretical motivation given for this choice is worst-case error control. VQRound argues that low-rank alternatives tend to minimize global average error, while rounding quality is sensitive to element-wise worst-case error. Since the rectified-sigmoid transform $n,p$4 is globally Lipschitz with constant

$n,p$5

the paper derives

$n,p$6

It further introduces the clipping margin

$n,p$7

and states that if

$n,p$8

then clipping is activated and $n,p$9. The stated implication is that suppressing the tails of the element-wise approximation error is critical for avoiding clipping-induced saturation in the rounding matrix (Zhou et al., 2 Feb 2026).

A further reported empirical finding is that initialization is decisive. VQRound therefore uses a Hessian-aware residual initialization: columns are processed sequentially, quantized with RTN, the error

$$\mathcal{L}(x,y,w+\Delta w)-\mathcal{L}(x,y,w) \approx \Delta w^T g_w + \frac{1}{2} \Delta w^T H_w \Delta w,$$0

is computed, later columns are updated by

$$\mathcal{L}(x,y,w+\Delta w)-\mathcal{L}(x,y,w) \approx \Delta w^T g_w + \frac{1}{2} \Delta w^T H_w \Delta w,$$1

and the initial latent rounding signal is

$$\mathcal{L}(x,y,w+\Delta w)-\mathcal{L}(x,y,w) \approx \Delta w^T g_w + \frac{1}{2} \Delta w^T H_w \Delta w,$$2

The paper reports that Hessian-guided initialization improves perplexity over initialization without curvature information (Zhou et al., 2 Feb 2026).

VQRound also introduces an optional end-to-end finetuning mode in which the codebook is optimized jointly across all layers while all other model weights remain frozen. The reported training setup is 128 randomly sampled sequences from C4, sequence length 2048, Adam, learning rate $$\mathcal{L}(x,y,w+\Delta w)-\mathcal{L}(x,y,w) \approx \Delta w^T g_w + \frac{1}{2} \Delta w^T H_w \Delta w,$$3, regularization weight $$\mathcal{L}(x,y,w+\Delta w)-\mathcal{L}(x,y,w) \approx \Delta w^T g_w + \frac{1}{2} \Delta w^T H_w \Delta w,$$4, $$\mathcal{L}(x,y,w+\Delta w)-\mathcal{L}(x,y,w) \approx \Delta w^T g_w + \frac{1}{2} \Delta w^T H_w \Delta w,$$5 decaying from 20 to 2, 5000 training steps, first 10% of steps as distillation-only warm-up, FAISS-accelerated K-means with 100 iterations for initialization, default codebook size $$\mathcal{L}(x,y,w+\Delta w)-\mathcal{L}(x,y,w) \approx \Delta w^T g_w + \frac{1}{2} \Delta w^T H_w \Delta w,$$6, vector dimension $$\mathcal{L}(x,y,w+\Delta w)-\mathcal{L}(x,y,w) \approx \Delta w^T g_w + \frac{1}{2} \Delta w^T H_w \Delta w,$$7, and hardware consisting of a single NVIDIA RTX A6000 (Zhou et al., 2 Feb 2026).

The reported trainable-parameter counts are 4.72M for OPT-1.3B versus 1.21B for AdaRound, 6.29M versus 2.16B for OPT-2.7B, 7.34M versus 6.48B for LLaMA-7B, and 9.18M versus 12.69B for LLaMA-13B. The abstract summarizes this as “as little as 0.2% of the trainable parameters.” Extensive experiments are reported on OPT, LLaMA, LLaMA2, and Qwen3, with the general conclusion that VQRound achieves better convergence than traditional adaptive rounding at the same number of steps while remaining scalable (Zhou et al., 2 Feb 2026).

6. Misconceptions, limitations, and distinct meanings of “rounding”

A common source of confusion is the breadth of the word “rounding.” In the AutoRound/PTQ context, rounding means optimization over quantization decisions under calibration. This is distinct from directed rounding in interval arithmetic, where outward rounding in radix conversion is described as computationally a simpler problem than guaranteeing identity, and from proof-oriented interval tools such as Gappa, which uses interval arithmetic to certify bounds on expressions involving rounded as well as exact operators [0703003] [0701186].

It is likewise distinct from correctly rounded math libraries. RLibm-MultiRound addresses correctness across multiple representations with up to 32-bits and all four standard rounding modes, proposing rounding-invariant outputs and rounding-invariant input bounds to avoid frequent changes to the rounding mode, with reported performance improvements of more than $$\mathcal{L}(x,y,w+\Delta w)-\mathcal{L}(x,y,w) \approx \Delta w^T g_w + \frac{1}{2} \Delta w^T H_w \Delta w,$$8 (Park et al., 10 Apr 2025). That setting concerns floating-point semantics and library generation, not PTQ calibration.

The term is also separate from randomized rounding in combinatorial optimization and online decision-making. In those literatures, rounding refers to converting LP or fractional prescriptions into feasible integral structures or online policies, as in ATSP, dynamic matching, or sequential stochastic optimization (0909.0941, Ma, 2024). AutoRound in quantization instead targets learned up/down decisions that reduce reconstruction error or a task-loss proxy under low-bit numerical constraints.

Within quantization itself, another misconception is to treat adaptive rounding as merely a new loss. The LLM literature explicitly notes that VQRound is not just a new loss; it changes the rounding representation, depends on good initialization, and is mainly evaluated for weight quantization in PTQ settings rather than as a universal quantization framework (Zhou et al., 2 Feb 2026). This suggests that “AutoRound” names a design principle—optimize rounding decisions from calibration data—while concrete instantiations differ substantially in parameterization, initialization, and deployment regime.