---
title: Quantization-Aware Distillation (QAD)
url: https://www.emergentmind.com/topics/quantization-aware-distillation-qad
type: topic
---

# Quantization-Aware Distillation (QAD)

Quantization-Aware Distillation (QAD) denotes training procedures in which a quantized student is optimized with quantization active in the training loop while simultaneously receiving distillation signals from a higher-precision teacher. In the literature, the term appears both explicitly and as a broader family of methods that combine quantization-aware training (QAT) with knowledge distillation (KD), spanning early “quantized distillation” for CNNs [1802.05668], staged low-bit image classification via self-studying, co-studying, and tutoring [1911.12491], self-supervised QAT framed as joint minimization of KL-Loss and discretization error [2403.11106], data-free and self-distilled low-bit large language models [2305.17888, 2402.10631], and post-training NVFP4 accuracy recovery for LLMs and VLMs [2601.20088]. Across these lines of work, QAD is best understood not as a single loss function but as a design space in which teacher choice, distillation target, quantizer parameterization, and training schedule are all conditioned on the distortions introduced by low precision.

## 1. Historical emergence and conceptual scope

An early canonical formulation appeared in “Model compression via distillation and quantization” [1802.05668], which introduced **quantized distillation**, where a quantized student is trained directly with a distillation loss, and **differentiable quantization**, where quantization points are optimized by stochastic gradient descent. That work already established two enduring QAD themes: first, quantization must be present during optimization rather than treated as a purely post hoc projection; second, the teacher’s softened outputs can compensate for the accuracy loss induced by aggressive weight discretization.

Later work made the coordination between KD and QAT explicit. “QKD: Quantization-aware Knowledge Distillation” [1911.12491] argued that naïvely combining quantization and KD may fail because the regularization effect of KD further diminishes the already reduced representation power of a quantized model. Its three-phase structure—Self-studying, Co-studying, and Tutoring—recast QAD as a curriculum in which the low-precision student is first stabilized, then paired with a teacher that is itself adapted to the student, and only afterward refined by frozen-teacher distillation.

Subsequent work broadened QAD beyond CNN classification. Transformer encoders motivated attention-specific QAD objectives [2211.11014]; multilingual speech recognition introduced joint distillation and 8-bit quantization in sequence-to-sequence ASR [2305.10788]; multi-view 3D detection used geometry-aware feature distillation to stabilize ultra-low precision BEV models [2308.10515]; LLM quantization adopted data-free synthetic distillation [2305.17888], self-distillation with confidence-aware divergences [2402.10631], progressive PTQ-to-QAT schedules for INT2 instruction-tuned models [2506.09104], and NVFP4 recovery pipelines for multi-stage post-trained LLMs and VLMs [2601.20088]. This progression suggests that QAD has evolved from a compression heuristic into a modality-dependent training principle.

## 2. Objective functions and optimization structure

The common QAD structure is a teacher–student optimization in which the student forward pass is quantized and the loss contains either supervised, distillation, or purely teacher-driven terms. A representative student loss from QKD is
$$
\mathcal{L}^{S}_{KD} = \mathcal{L}_{ce}^{S} + \mathcal{T}^2 \,\mathrm{KL}(\mathbf{z}^T \,\|\, \mathbf{z}^S; \mathcal{T}),
$$
while the teacher can also be trained online with
$$
\mathcal{L}^{T}_{KD} = \mathcal{L}_{ce}^{T} + \mathcal{T}^2 \,\mathrm{KL}(\mathbf{z}^S \,\|\, \mathbf{z}^T; \mathcal{T}),
$$
so that the teacher becomes more quantization-friendly rather than remaining fixed [1911.12491]. In older CNN settings, the distillation term was typically the weighted combination of cross-entropy with hard labels and softened teacher targets [1802.05668].

Later QAD methods diversified the divergence itself. “BitDistiller” [2402.10631] replaced conventional forward-KL-only KD with **Confidence-Aware Kullback-Leibler Divergence (CAKLD)**,
$$
D_{\text{CAKLD}}(P_T \Vert P_S) = \gamma \, D_{\mathrm{KL}}(P_S \Vert P_T) + (1 - \gamma)\, D_{\mathrm{KL}}(P_T \Vert P_S),
$$
where the confidence coefficient is computed from teacher confidence on the training set. In contrast, “Unified Progressive Quantization” [2506.09104] used a generalized Jensen–Shannon divergence between the FP16 teacher and the INT2 student, arguing that pure JSD better preserves instruction-following and reasoning behavior than next-token prediction or hybrid JSD+NTP objectives. “Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery” [2601.20088] adopted a pure KL divergence with temperature \(T=1\), using only distillation loss and no explicit cross-entropy on labels.

A different branch removed labels altogether. “Self-Supervised Quantization-Aware Knowledge Distillation” formulates QAT as a co-optimization problem that simultaneously minimizes the KL-Loss between full-precision and low-bit models and the discretization error, without supervision from labels [2403.11106]. The poster version makes this explicit through a generalized backward rule,
$$
\frac{\partial L}{\partial x_c} = \frac{\partial L}{\partial x_q} + \mu \cdot (x_c - x_q),
$$
so that discretization error is embedded directly into the gradient rather than only into the forward quantizer [2309.13220]. Taken together, these formulations show that QAD objectives range from conventional CE+KL composites to pure teacher-matching objectives and explicit quantization-error regularization.

## 3. Distillation targets and teacher design

Although early QAD emphasized output logits, later work established that the most effective target depends on what quantization disrupts. In large Transformer encoders, MSE on attention scores was shown to be insufficient for recovering self-attention information; the proposed alternatives were **attention-map loss**, a KL divergence on attention maps, and **attention-output loss**, an MSE loss on attention outputs \(Y_l\), with all-layer distillation proving superior to selected-layer distillation for QAT [2211.11014]. This directly tied the distillation target to the decomposition of self-attention into attention generation and attention propagation.

Feature-level QAD became especially prominent in vision. “Quantized Feature Distillation for Network Quantization” first trains a quantized representation as the teacher and then quantizes the student using MSE between the student feature and the teacher’s **quantized** feature [2307.10638]. Empirically, the teacher’s feature bit-width mattered: on CIFAR100 with ResNet-18, a 1-bit teacher feature yielded better student performance than a 32-bit feature teacher in several low-bit settings, even though the teacher’s own accuracy was slightly lower. In speech, “DQ-Whisper” combined prediction-layer KL divergence with hidden-layer distillation and used quantization loss to guide the dynamic mapping from teacher layers to student layers [2305.10788].

Several recent methods replaced generic feature matching by structure-aware targets. “Stabilizing Quantization-Aware Training by Implicit-Regularization on Hessian Matrix” used Channel-wise Standardization Distillation (CSD) between full-precision and quantized features, while injecting quantization-scale stochastic feature perturbations into the student [2503.11159]. “QD-BEV” distilled both image features and BEV features and then fused them with camera-extrinsic visibility masks into a view-guided distillation objective [2308.10515]. “Q-VDiT” defined Temporal Maintenance Distillation through KL divergence between teacher and student temporal relation distributions over video frames, explicitly targeting the temporal structure most damaged by low-bit quantization in video diffusion transformers [2505.22167]. “Decoder-Free Distillation for Quantized Image Restoration” restricted KD to the network bottleneck, arguing that decoder-level distillation in precision-sensitive image restoration amplifies spatial quantization errors [2603.09624].

Teacher choice also changed. BitDistiller found that self-distillation from the full-precision version of the same architecture outperformed using a larger teacher for 2-bit LLMs [2402.10631]. QDR used FP32 self-distillation for the same reason, namely to eliminate teacher–student capacity mismatch in image restoration [2603.09624]. By contrast, QKD explicitly trained the teacher during co-studying so that it became both more powerful and more quantization-friendly [1911.12491]. A plausible implication is that QAD is as much about matching representational geometry as about transferring “knowledge” in the conventional sense.

## 4. Quantization regimes, schedules, and data usage

QAD methods differ sharply in their quantizers, training schedules, and assumptions about data access. Uniform QAT with STE remains a basic template; it appears in early CNN work [1802.05668], in Whisper compression [2305.10788], and in several low-bit vision studies. Other works instead rely on learned step sizes, clipping parameters, or asymmetric quantizers. BitDistiller combines sub-4-bit QAT with **asymmetric clipping** and asymmetric NF3 or INT2 quantization, and reports that asymmetric quantization plus clipping is crucial for making 2-bit QAT workable [2402.10631]. LLM-QAT uses **uniform symmetric MinMax quantization** without clipping, per-channel for weights and per-token for activations and KV cache, explicitly because clipping outliers was found to be harmful for LLaMA-style models at low bits [2305.17888]. NVFP4 QAD adopts the NVFP4 numeric format, with 4-bit floating-point values stored in blocks of 16 elements, an FP8 per-block scale, and an FP32 per-tensor scale [2601.20088].

Scheduling is equally important. QKD’s SS–CS–TU decomposition is one curriculum [1911.12491]. QD-BEV instead uses **progressive QAT**, quantizing the Backbone, Neck, Encoder, and Decoder in stages before enabling view-guided distillation [2308.10515]. UPQ makes progression explicit at the precision level—FP16 \(\rightarrow\) INT4 block-wise PTQ \(\rightarrow\) INT2 Distill-QAT—because direct FP16 \(\rightarrow\) INT2 distillation-aware QAT is too destructive for instruction-tuned models [2506.09104]. In another direction, SQAKD aims to be agnostic to the underlying quantization function by unifying forward and backward dynamics across various QAT methods, then optimizing KL-Loss and discretization error without labels [2403.11106].

Data assumptions vary from fully supervised to synthetic and label-free. LLM-QAT is explicitly **data-free**: it generates up to 1024-token sequences from the teacher itself and trains the quantized student on teacher soft targets rather than original pretraining data [2305.17888]. NVFP4 QAD reports robustness to data quality and coverage, including recovery with cold-start SFT data, RL-prompt generations, BOS-generated data, and even random token sequences [2601.20088]. SQAKD removes labels entirely and relies on teacher-driven KL plus discretization-aware gradients [2309.13220]. These results indicate that QAD can decouple quantization recovery from the availability of the original task data, although the degree of robustness is architecture- and task-dependent.

## 5. Domain-specific instantiations and representative results

QAD is no longer confined to image classification; it has become a cross-domain strategy for low-bit compression and post-training recovery.

| Domain | Representative method | Representative result |
|---|---|---|
| Image classification | QFD | 4-bit ResNet-34 on ImageNet reached 74.7 top-1, above the 73.3 full-precision baseline [2307.10638] |
| Speech recognition | DQ-Whisper | Up to 5.18x reduction in model size with marginal performance degradation [2305.10788] |
| LLMs and VLMs | LLM-QAT / NVFP4 QAD | LLaMA-30B W4A8KV4 reached 69.9 average versus 42.5 for RTN and 40.7 for SmoothQuant; NVFP4 QAD recovered Llama Nemotron Super V1 49B to 45.6 on AIME25 versus 46.0 in BF16 [2305.17888, 2601.20088] |
| Multi-view 3D detection | QD-BEV | QD-BEV-Tiny W4A6 achieved 37.2% NDS with 15.8 MB, outperforming BevFormer-Tiny by 1.8% with an 8x model compression [2308.10515] |
| Video generation | Q-VDiT | W3A6 achieved a scene consistency of 23.40 [2505.22167] |
| Image restoration | QDR | INT8 recovered 96.5% of FP32 performance and reached 442 FPS on NVIDIA Jetson Orin [2603.09624] |

Within language models, the same pattern appears under very different design choices. LLM-QAT uses data-free logit distillation and quantizes weights, activations, and KV cache [2305.17888]. BitDistiller targets weight-only sub-4-bit quantization with self-distillation and CAKLD, reaching 29.25 MMLU for 2-bit LLaMA-2-7B versus 23.62 for LLM-QAT [2402.10631]. UPQ targets instruction-tuned INT2 models by combining INT4 PTQ initialization with JSD-based Distill-QAT, substantially improving MMLU and IFEval over direct NTP-QAT and direct Distill-QAT [2506.09104]. In post-trained systems, NVFP4 QAD is positioned not as a replacement for the original supervised fine-tuning or RL pipeline but as a single additional stage that recovers near-BF16 behavior after quantization [2601.20088].

Vision shows equally heterogeneous target choices. QFD uses quantized teacher features for classification and detection [2307.10638]; QD-BEV uses geometry-aware multi-domain feature distillation [2308.10515]; QDR uses bottleneck-only self-distillation for dense restoration [2603.09624]. This breadth supports a general characterization of QAD as a modality-conditioned interface between discretized computation and teacher-guided optimization.

## 6. Empirical behavior, misconceptions, and open problems

A recurring misconception is that any KD term improves QAT. QKD explicitly showed that straightforward KD can underperform quantization-only training because the regularization effect of KD further diminishes the already reduced representation power of a quantized model [1911.12491]. A second misconception is that output-logit distillation is universally sufficient. Transformer encoders required attention-map and attention-output losses to recover self-attention under sub-2-bit quantization [2211.11014]; QFD showed that quantized feature targets can outperform both logit KD and full-precision feature KD in CNN quantization [2307.10638]; QDR found that decoder-level distillation harms quantized image restoration whereas bottleneck-only self-distillation helps [2603.09624].

Teacher quality and teacher size are also less straightforward than in conventional KD. BitDistiller reported that self-distillation with the same architecture outperformed a larger teacher in 2-bit LLM QAT [2402.10631]. QKD, by contrast, improved outcomes by adapting the teacher during co-studying rather than enlarging it [1911.12491]. This suggests that teacher compatibility with the student’s quantized hypothesis class may matter more than teacher capacity alone.

Another misconception is that QAD necessarily requires the original training set. LLM-QAT and NVFP4 QAD showed strong results with generated or partial data [2305.17888, 2601.20088], and SQAKD dispensed with labels altogether [2403.11106]. Yet these successes do not eliminate the remaining limitations. LLM-QAT explicitly states that 4-bit activation quantization remains insufficient [2305.17888]. BitDistiller is weight-only and identifies binary quantization, vector quantization, and deeper theoretical explanations for teacher choice as open directions [2402.10631]. UPQ does not quantize activations and has only been validated up to 8B instruction-tuned models [2506.09104]. QD-BEV improves BEVFormer more dramatically than some other camera-only detectors, indicating architecture dependence [2308.10515]. QDR still relies on a FP32 teacher and nontrivial training-time overhead, even though its runtime model is edge-efficient [2603.09624].

The broader literature therefore points to a restrained conclusion. QAD is not a single recipe but a family of quantization-conditioned distillation strategies whose success depends on aligning four factors: the target of distillation, the way quantization is simulated or parameterized, the schedule by which the student is exposed to low precision, and the compatibility between teacher behavior and the student’s constrained function class. The technical trend is toward more structured and modality-specific objectives—attention distributions, standardized channel features, temporal relation matrices, geometry-aware BEV mappings, bottleneck latents, or pure output KL in post-trained LLMs—rather than toward a universal KD loss.

Source: https://www.emergentmind.com/topics/quantization-aware-distillation-qad