Prediction Difference Metrics (PD-Quant)

• Prediction Difference Metrics (PD-Quant) are post-training quantization techniques that optimize parameters by minimizing the KL divergence between full-precision and quantized outputs over a calibration set.
• The method jointly optimizes activation scales and rounding offsets using a hybrid loss that combines global prediction difference and local MSE regularization.
• PD-Quant mitigates overfitting with distribution correction techniques and demonstrates notable accuracy improvements in aggressive quantization scenarios.

Prediction Difference Metrics (PD-Quant) are a class of post-training quantization (PTQ) techniques that employ global measures of prediction shift to optimize quantization parameters in deep neural networks. Unlike conventional PTQ approaches based on local feature or activation distance, PD-Quant determines quantization parameters by explicitly minimizing the difference between the predicted distributions of a full-precision network and its quantized counterpart over a calibration set. This method aims to reduce accuracy degradation in aggressive (e.g., 2-bit) quantization regimes by leveraging a global prediction difference metric, typically instantiated via the Kullback–Leibler (KL) divergence between output softmax distributions (Liu et al., 2022). PD-Quant further incorporates techniques to counteract overfitting with limited calibration data, notably via distribution correction of activations.

1. Formal Definition of the Prediction Difference Metric

Let $f(\cdot)$ denote a full-precision neural network and $f^q(\cdot)$ its quantized version. Given a calibration dataset $\{x_i\}_{i=1}^N$, define:

• $p_i = f(x_i) \in \mathbb{R}^C$: Softmax output of $f$ on input $x_i$
• $q_i = f^q(x_i) \in \mathbb{R}^C$: Softmax output of $f^q$ on $x_i$

The Prediction Difference Metric (PDM) is the average KL divergence between the predictions: $$\mathrm{PDM}(f, f^q) = \frac{1}{N}\sum_{i=1}^N D_{\mathrm{KL}}(p_i \Vert q_i) = \frac{1}{N}\sum_{i=1}^N \sum_{c=1}^C p_i(c) \log \frac{p_i(c)}{q_i(c)}$$

PDM quantifies the shift in predicted distributions resulting from quantization noise, serving as a global criterion for PTQ parameter search.

2. Optimization of Quantization Parameters via PDM

PD-Quant simultaneously optimizes per-layer/block activation scales ($f^q(\cdot)$0) and per-weight rounding offsets ($f^q(\cdot)$1). For block $f^q(\cdot)$2, given quantized output $f^q(\cdot)$3 and original prediction $f^q(\cdot)$4, the optimization problem is:

$f^q(\cdot)$5

• $f^q(\cdot)$6: Global loss (prediction difference)
• $f^q(\cdot)$7: Blockwise output MSE regularizer
• $f^q(\cdot)$8: Regularization coefficient (typ. $f^q(\cdot)$9–$\{x_i\}_{i=1}^N$0)

The quantizer for each value $\{x_i\}_{i=1}^N$1 is: $\{x_i\}_{i=1}^N$2 with scale $\{x_i\}_{i=1}^N$3, zero-point $\{x_i\}_{i=1}^N$4, and round offset $\{x_i\}_{i=1}^N$5.

3. Algorithmic Description and Workflow

PD-Quant is implemented as a blockwise, calibration-driven algorithm. The standard workflow is:

1. Initialize $\{x_i\}_{i=1}^N$6; set all $\{x_i\}_{i=1}^N$7; set all $\{x_i\}_{i=1}^N$8 via min–max.
2. For $\{x_i\}_{i=1}^N$9 (blocks): a. Propagate calibration samples through blocks $p_i = f(x_i) \in \mathbb{R}^C$0 to obtain inputs $p_i = f(x_i) \in \mathbb{R}^C$1. b. (Optional) Apply Distribution Correction (DC) to match batch statistics. c. Jointly optimize $p_i = f(x_i) \in \mathbb{R}^C$2 by minimizing the sum of PDM loss and local regularizer for $p_i = f(x_i) \in \mathbb{R}^C$3 iterations. d. Freeze parameters and proceed to the next block.
3. Output the fully quantized network $p_i = f(x_i) \in \mathbb{R}^C$4.

The pseudocode structure is as follows:

$q_i = f^q(x_i) \in \mathbb{R}^C$6

4. Overfitting Mitigation: Distribution Correction

With limited calibration data, PTQ can overfit to idiosyncratic activation statistics. PD-Quant addresses this by remapping block inputs to match the mean/variance recorded in BatchNorm layers:

Let block $p_i = f(x_i) \in \mathbb{R}^C$5 have $p_i = f(x_i) \in \mathbb{R}^C$6 BatchNorm layers with statistics $p_i = f(x_i) \in \mathbb{R}^C$7. For sample batch statistics $p_i = f(x_i) \in \mathbb{R}^C$8, solve

$p_i = f(x_i) \in \mathbb{R}^C$9

Here, $f$0 tunes the correction's strength ($f$1 corresponds to no correction). Correction is performed using a few steps of gradient descent and is activated when the calibration set is small ($f$2K).

5. Empirical Performance and Benchmarking

PD-Quant demonstrates consistent improvements in accuracy over established PTQ baselines, particularly in low-bit regimes. Selected results include:

Model	Bit	Baseline QDrop	PD-Quant
ResNet-18	2/2	51.42%	53.14%
RegNetX-600MF	2/2	39.01%	40.67%

Wider 4-bit/2-bit (W4A2) or 2-bit/4-bit (W2A4) configurations see $f$3–$f$4 improvements. At 256 calibration images, gains over QDrop remain $f$5–$f$6; the benefit of DC diminishes with calibration set size.

Relative quantization time (ResNet-18, RTX A6000):

• QDrop: $f$7 h
• PD-Quant (no DC): $f$8 h
• PD-Quant ($f$9DC): $x_i$0 h

This is substantially faster than quantization-aware training (QAT; e.g., LSQ, $x_i$1 h).

6. Hyperparameters, Calibration, and Implementation Details

Hyperparameters are found robust across model architectures:

• $x_i$2 (regularization): $x_i$3
• $x_i$4 (DC strength): $x_i$5

Calibration sets of $x_i$6K–$x_i$7K in-domain images yield optimal results; even at $x_i$8 samples, performance exceeds QDrop. Notably, scale optimization for weights via PDM does not provide further benefit, as weight scale search per channel is discrete; MSE or per-channel min–max is preferred for this aspect.

7. Limitations, Methodological Extensions, and Best Practices

PD-Quant incurs additional fine-tuning cost (typically $x_i$9 h per model), contrasting with search-only approaches. Application to Vision Transformers (ViT/DeiT) shows $q_i = f^q(x_i) \in \mathbb{R}^C$0–$q_i = f^q(x_i) \in \mathbb{R}^C$1 improvements at W4A6 and W2A6. Other potential metrics (e.g., Earth Mover’s Distance) may be substituted for KL, and mixed-precision bitwidth selection can be informed by PDM sensitivity.

Best practices include quantizing blocks in order, utilizing $q_i = f^q(x_i) \in \mathbb{R}^C$2–$q_i = f^q(x_i) \in \mathbb{R}^C$3 in-domain calibration samples, tuning $q_i = f^q(x_i) \in \mathbb{R}^C$4 on a single block, always incorporating the local $q_i = f^q(x_i) \in \mathbb{R}^C$5, and applying DC for small calibration sizes.

A plausible implication is that layer-local MSE alone insufficiently captures prediction robustness under quantization, and global output-centric metrics such as PDM provide a superior quantization protocol for aggressive precision reduction (Liu et al., 2022).