Quantization-Aware Computation

• Quantization-Aware Computation is a framework that integrates quantization operators within training to jointly optimize model accuracy and hardware metrics.
• It employs real-device profiling and MILP-based optimization to assign mixed-precision bit-widths, reducing energy, latency, and memory footprints.
• Practical applications like the OHQ framework have demonstrated significant efficiency gains, with up to 30% latency reduction and improved accuracy in architectures such as ResNet-18 and MobileNetV3.

Quantization-Aware Computation

Quantization-Aware Computation (QAC) denotes a class of optimization and training methodologies for deep neural networks (DNNs) in which quantization operators and their hardware consequences are treated as first-class citizens in the computational workflow. In QAC, both the choice of quantization (e.g., bit-width, per-layer granularity, quantizer type) and its impact on model accuracy and concrete hardware deployment metrics (throughput, latency, energy) are jointly modeled, simulated, or measured, often as part of an end-to-end training, fine-tuning, or search pipeline. This approach encompasses mixed-precision assignments, direct hardware-in-the-loop feedback, and accuracy-impact estimation to identify precision configurations that best trade off accuracy and on-chip cost. QAC is exemplified by the On-Chip Hardware-Aware Quantization (OHQ) framework, which realizes a fully on-chip, hybrid metric-driven mixed-precision quantization methodology and provides a practical reference for the QAC paradigm (Huang et al., 2023).

1. Problem Formulation: Joint Optimization for Mixed-Precision Quantization

The QAC paradigm centers on the formulation of a discrete-continuous joint optimization problem targeting a neural network with $L$ layers (or blocks), each indexed by $i=1..\!L$. Each layer is assigned a bit-width $b_i$ from a finite set $B = \{2,4,8,\ldots\}$. For a pre-trained floating-point baseline, QAC explicitly models:

• $\Delta A(\{b_i\})$: The predicted drop in model accuracy (e.g., Top-1 classification error) incurred by quantizing each layer $i$ to $b_i$ bits.
• $C_\mathrm{hard}(\{b_i\})$: The aggregate hardware cost metric (cycles, energy, memory) of executing the quantized model on the deployment target.
• $M_\mathrm{tot}(\{b_i\})$: The overall model size on the memory hierarchy (bits).
• $L_\mathrm{max},\,M_\mathrm{max}$: Tight constraints on runtime and footprint.

The canonical QAC optimization is a mixed-integer linear program (MILP):

$i=1..\!L$0

$i=1..\!L$1

Alternate formulations involve maximizing a sum of per-layer scores $i=1..\!L$2 (normalized sensitivity, cycle, and energy costs) over bit-width allocations $i=1..\!L$3, subject to resource budgets. This formalism enables joint quantizer assignment under strict hardware constraints (Huang et al., 2023).

2. On-Chip Quantization-Aware Pipeline

A distinguishing feature of cutting-edge QAC (as in OHQ) is the use of real hardware profiling—eschewing accuracy or efficiency predictions from desktop simulators. The pipeline consists of:

• On-Chip Synthetic Data Generation: BatchNorm statistics from the full-precision network are matched by solving a minimization over synthetic input batches ($i=1..\!L$4), yielding inputs for accurate, resource-constrained inference profiling.
• FPGA Instrumentation: Network layers are mapped to programmable logic (PL) as IP cores. Each layer’s sub-operations—core compute, input/output transfer, memory writeback—are individually counted for clock cycles and energy using on-chip hardware counters (typically integrated into block RAM).
• Hardware Metric Aggregation: The ARM processing system (PS) reads out all per-layer cycle and energy measurements via AXI, immediately constructing the $i=1..\!L$5 metric for optimization input.
• Integration into Search Loop: Empirically observed metrics feed directly into the MILP or similar search procedure, avoiding errors induced by simulation mismatches or low-fidelity analytic estimates, thus directly matching the deployment substrate (Huang et al., 2023).

3. Accuracy Impact Estimation: Mask-Guided Quantization Estimation

OHQ introduces a hardware-efficient blockwise accuracy impact estimator, Mask-Guided Quantization Estimation (MQE), that dispenses with expensive Hessians or full retraining:

• Begin with all weights quantized to 8 bits ($i=1..\!L$6).
• For each candidate layer $i=1..\!L$7, generate a "masked" variant ($i=1..\!L$8) by zeroing a fixed ($i=1..\!L$9) fraction of weights, simulating the aggressive SNR drop of 4-bit quantization but only for $b_i$0.
• Pass distilled synthetic samples (from the calibration step) through both $b_i$1 and the layer-$b_i$2-masked model, collect softmax outputs $b_i$3 and $b_i$4, then compute $b_i$5 across samples.
• The collection $b_i$6 encapsulates per-layer quantization sensitivity.
• The global accuracy drop $b_i$7 is approximated by a sum of $b_i$8 weighted by quantization depth, i.e., only layers at sub-8-bit precision contribute (Huang et al., 2023).

This technique is lightweight and fully edge-hardware-computable, allowing $b_i$9 empirical estimation of quantization sensitivity per layer via runtime perturbations.

4. Integer-Linear Search and On-Chip Deployment Recipe

The QAC workflow integrates the hardware- and accuracy-aware measurements via a fast bit-width allocation algorithm, typically as a linear integer program:

1. Generate on-chip distilled calibration data.
2. Profile per-layer cycles ($B = \{2,4,8,\ldots\}$0), energy ($B = \{2,4,8,\ldots\}$1) via deployment and runtime logging using the candidate $B = \{2,4,8,\ldots\}$2.
3. For each layer: mask 50% of weights, run samples, calculate $B = \{2,4,8,\ldots\}$3 via blockwise $B = \{2,4,8,\ldots\}$4.
4. Normalize hardware and sensitivity profiles, compute per-layer score $B = \{2,4,8,\ldots\}$5.
5. Maximize $B = \{2,4,8,\ldots\}$6 over $B = \{2,4,8,\ldots\}$7 subject to total model size and latency limits.
6. Deploy the selected configuration, re-quantize/recompile as per search output.

The process is computationally efficient—each full MILP solve with up to $B = \{2,4,8,\ldots\}$8 layers completes in $B = \{2,4,8,\ldots\}$91 second on an ARM edge CPU—enabling practical real-device co-design (Huang et al., 2023).

5. Empirical Results and Comparative Evaluation

On real deployment (Zynq FPGA, ARM Cortex-A9), OHQ demonstrates:

• ResNet-18 (PTQ, ImageNet): Model size 5.5 MB (vs 11.1 MB uniform), latency 63.5 ms (vs 78.3 ms INT8), Top-1 accuracy 70.18% (vs best prior 69.32%)
• MobileNetV3: Size 2.4 MB (vs 3.9 MB INT8), latency 73.4 ms (vs 85.0 ms), Top-1 accuracy 73.01% (vs 72.98% INT8)
• Typical latency reductions of 15–30% compared to INT8, consistently matching or exceeding prior mixed-precision PTQ baselines (ZeroQ, BRECQ, OMPQ, EMQ)

This empirically validates that QAC-based frameworks, by fusing joint bit-width assignment, hardware-in-the-loop profiling, and on-chip sensitivity estimation, produce architectures with measurable gains in efficiency and accuracy, fully determined on the target device (Huang et al., 2023).

6. QAC in the Broader Ecosystem: Extensions and Significance

The QAC framework is not limited to uniform/single quantizer settings; its principles extend naturally to:

• Mixed-Precision Allocation: Per-layer $\Delta A(\{b_i\})$0 selection, layerwise constraints, or even mixed quantizer type assignment under hardware-aware objectives.
• Generalization to Hardware Variants: The on-chip measurement-centric loop allows seamless adaptation to heterogeneous SoCs (CPU, FPGA, ASIC) and overcomes the hardware-software gap endemic to simulation-based quantization.
• Integration with Multi-Objective Search: The QAC problem can be generalized to simultaneously optimize for accuracy, latency, memory footprint, and batch-size, supporting complex trade-offs in deployment scenarios with rigorous resource envelopes.

The fundamental insight is that quantization and deployment can—and should—be solved as an integrated system-level design problem, where algorithm choices are co-tuned to the measured behaviors and constraints of the concrete hardware at hand, embodying a holistic quantization-aware computation paradigm (Huang et al., 2023).