Quantization-Aware Distillation (QAD)
- Quantization-Aware Distillation (QAD) is a training strategy that integrates quantization-aware training with knowledge distillation to optimize low-bit models while mitigating precision loss.
- It employs various distillation targets—ranging from output logits and attention maps to structured features—to recover degraded model representations.
- QAD adapts teacher design, quantizer parameterization, and progressive training schedules to effectively balance compression and performance across multiple domains.
Quantization-Aware Distillation (QAD) denotes training procedures in which a quantized student is optimized with quantization active in the training loop while simultaneously receiving distillation signals from a higher-precision teacher. In the literature, the term appears both explicitly and as a broader family of methods that combine quantization-aware training (QAT) with knowledge distillation (KD), spanning early “quantized distillation” for CNNs (Polino et al., 2018), staged low-bit image classification via self-studying, co-studying, and tutoring (Kim et al., 2019), self-supervised QAT framed as joint minimization of KL-Loss and discretization error (Zhao et al., 2024), data-free and self-distilled low-bit LLMs (Liu et al., 2023, Du et al., 2024), and post-training NVFP4 accuracy recovery for LLMs and VLMs (Xin et al., 27 Jan 2026). Across these lines of work, QAD is best understood not as a single loss function but as a design space in which teacher choice, distillation target, quantizer parameterization, and training schedule are all conditioned on the distortions introduced by low precision.
1. Historical emergence and conceptual scope
An early canonical formulation appeared in “Model compression via distillation and quantization” (Polino et al., 2018), which introduced quantized distillation, where a quantized student is trained directly with a distillation loss, and differentiable quantization, where quantization points are optimized by stochastic gradient descent. That work already established two enduring QAD themes: first, quantization must be present during optimization rather than treated as a purely post hoc projection; second, the teacher’s softened outputs can compensate for the accuracy loss induced by aggressive weight discretization.
Later work made the coordination between KD and QAT explicit. “QKD: Quantization-aware Knowledge Distillation” (Kim et al., 2019) argued that naïvely combining quantization and KD may fail because the regularization effect of KD further diminishes the already reduced representation power of a quantized model. Its three-phase structure—Self-studying, Co-studying, and Tutoring—recast QAD as a curriculum in which the low-precision student is first stabilized, then paired with a teacher that is itself adapted to the student, and only afterward refined by frozen-teacher distillation.
Subsequent work broadened QAD beyond CNN classification. Transformer encoders motivated attention-specific QAD objectives (Kim et al., 2022); multilingual speech recognition introduced joint distillation and 8-bit quantization in sequence-to-sequence ASR (Shao et al., 2023); multi-view 3D detection used geometry-aware feature distillation to stabilize ultra-low precision BEV models (Zhang et al., 2023); LLM quantization adopted data-free synthetic distillation (Liu et al., 2023), self-distillation with confidence-aware divergences (Du et al., 2024), progressive PTQ-to-QAT schedules for INT2 instruction-tuned models (Lee et al., 10 Jun 2025), and NVFP4 recovery pipelines for multi-stage post-trained LLMs and VLMs (Xin et al., 27 Jan 2026). This progression suggests that QAD has evolved from a compression heuristic into a modality-dependent training principle.
2. Objective functions and optimization structure
The common QAD structure is a teacher–student optimization in which the student forward pass is quantized and the loss contains either supervised, distillation, or purely teacher-driven terms. A representative student loss from QKD is
while the teacher can also be trained online with
so that the teacher becomes more quantization-friendly rather than remaining fixed (Kim et al., 2019). In older CNN settings, the distillation term was typically the weighted combination of cross-entropy with hard labels and softened teacher targets (Polino et al., 2018).
Later QAD methods diversified the divergence itself. “BitDistiller” (Du et al., 2024) replaced conventional forward-KL-only KD with Confidence-Aware Kullback-Leibler Divergence (CAKLD),
where the confidence coefficient is computed from teacher confidence on the training set. In contrast, “Unified Progressive Quantization” (Lee et al., 10 Jun 2025) used a generalized Jensen–Shannon divergence between the FP16 teacher and the INT2 student, arguing that pure JSD better preserves instruction-following and reasoning behavior than next-token prediction or hybrid JSD+NTP objectives. “Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery” (Xin et al., 27 Jan 2026) adopted a pure KL divergence with temperature , using only distillation loss and no explicit cross-entropy on labels.
A different branch removed labels altogether. “Self-Supervised Quantization-Aware Knowledge Distillation” formulates QAT as a co-optimization problem that simultaneously minimizes the KL-Loss between full-precision and low-bit models and the discretization error, without supervision from labels (Zhao et al., 2024). The poster version makes this explicit through a generalized backward rule,
so that discretization error is embedded directly into the gradient rather than only into the forward quantizer (Zhao et al., 2023). Taken together, these formulations show that QAD objectives range from conventional CE+KL composites to pure teacher-matching objectives and explicit quantization-error regularization.
3. Distillation targets and teacher design
Although early QAD emphasized output logits, later work established that the most effective target depends on what quantization disrupts. In large Transformer encoders, MSE on attention scores was shown to be insufficient for recovering self-attention information; the proposed alternatives were attention-map loss, a KL divergence on attention maps, and attention-output loss, an MSE loss on attention outputs , with all-layer distillation proving superior to selected-layer distillation for QAT (Kim et al., 2022). This directly tied the distillation target to the decomposition of self-attention into attention generation and attention propagation.
Feature-level QAD became especially prominent in vision. “Quantized Feature Distillation for Network Quantization” first trains a quantized representation as the teacher and then quantizes the student using MSE between the student feature and the teacher’s quantized feature (Zhu et al., 2023). Empirically, the teacher’s feature bit-width mattered: on CIFAR100 with ResNet-18, a 1-bit teacher feature yielded better student performance than a 32-bit feature teacher in several low-bit settings, even though the teacher’s own accuracy was slightly lower. In speech, “DQ-Whisper” combined prediction-layer KL divergence with hidden-layer distillation and used quantization loss to guide the dynamic mapping from teacher layers to student layers (Shao et al., 2023).
Several recent methods replaced generic feature matching by structure-aware targets. “Stabilizing Quantization-Aware Training by Implicit-Regularization on Hessian Matrix” used Channel-wise Standardization Distillation (CSD) between full-precision and quantized features, while injecting quantization-scale stochastic feature perturbations into the student (Pang et al., 14 Mar 2025). “QD-BEV” distilled both image features and BEV features and then fused them with camera-extrinsic visibility masks into a view-guided distillation objective (Zhang et al., 2023). “Q-VDiT” defined Temporal Maintenance Distillation through KL divergence between teacher and student temporal relation distributions over video frames, explicitly targeting the temporal structure most damaged by low-bit quantization in video diffusion transformers (Feng et al., 28 May 2025). “Decoder-Free Distillation for Quantized Image Restoration” restricted KD to the network bottleneck, arguing that decoder-level distillation in precision-sensitive image restoration amplifies spatial quantization errors (Sharif et al., 10 Mar 2026).
Teacher choice also changed. BitDistiller found that self-distillation from the full-precision version of the same architecture outperformed using a larger teacher for 2-bit LLMs (Du et al., 2024). QDR used FP32 self-distillation for the same reason, namely to eliminate teacher–student capacity mismatch in image restoration (Sharif et al., 10 Mar 2026). By contrast, QKD explicitly trained the teacher during co-studying so that it became both more powerful and more quantization-friendly (Kim et al., 2019). A plausible implication is that QAD is as much about matching representational geometry as about transferring “knowledge” in the conventional sense.
4. Quantization regimes, schedules, and data usage
QAD methods differ sharply in their quantizers, training schedules, and assumptions about data access. Uniform QAT with STE remains a basic template; it appears in early CNN work (Polino et al., 2018), in Whisper compression (Shao et al., 2023), and in several low-bit vision studies. Other works instead rely on learned step sizes, clipping parameters, or asymmetric quantizers. BitDistiller combines sub-4-bit QAT with asymmetric clipping and asymmetric NF3 or INT2 quantization, and reports that asymmetric quantization plus clipping is crucial for making 2-bit QAT workable (Du et al., 2024). LLM-QAT uses uniform symmetric MinMax quantization without clipping, per-channel for weights and per-token for activations and KV cache, explicitly because clipping outliers was found to be harmful for LLaMA-style models at low bits (Liu et al., 2023). NVFP4 QAD adopts the NVFP4 numeric format, with 4-bit floating-point values stored in blocks of 16 elements, an FP8 per-block scale, and an FP32 per-tensor scale (Xin et al., 27 Jan 2026).
Scheduling is equally important. QKD’s SS–CS–TU decomposition is one curriculum (Kim et al., 2019). QD-BEV instead uses progressive QAT, quantizing the Backbone, Neck, Encoder, and Decoder in stages before enabling view-guided distillation (Zhang et al., 2023). UPQ makes progression explicit at the precision level—FP16 INT4 block-wise PTQ INT2 Distill-QAT—because direct FP16 INT2 distillation-aware QAT is too destructive for instruction-tuned models (Lee et al., 10 Jun 2025). In another direction, SQAKD aims to be agnostic to the underlying quantization function by unifying forward and backward dynamics across various QAT methods, then optimizing KL-Loss and discretization error without labels (Zhao et al., 2024).
Data assumptions vary from fully supervised to synthetic and label-free. LLM-QAT is explicitly data-free: it generates up to 1024-token sequences from the teacher itself and trains the quantized student on teacher soft targets rather than original pretraining data (Liu et al., 2023). NVFP4 QAD reports robustness to data quality and coverage, including recovery with cold-start SFT data, RL-prompt generations, BOS-generated data, and even random token sequences (Xin et al., 27 Jan 2026). SQAKD removes labels entirely and relies on teacher-driven KL plus discretization-aware gradients (Zhao et al., 2023). These results indicate that QAD can decouple quantization recovery from the availability of the original task data, although the degree of robustness is architecture- and task-dependent.
5. Domain-specific instantiations and representative results
QAD is no longer confined to image classification; it has become a cross-domain strategy for low-bit compression and post-training recovery.
| Domain | Representative method | Representative result |
|---|---|---|
| Image classification | QFD | 4-bit ResNet-34 on ImageNet reached 74.7 top-1, above the 73.3 full-precision baseline (Zhu et al., 2023) |
| Speech recognition | DQ-Whisper | Up to 5.18x reduction in model size with marginal performance degradation (Shao et al., 2023) |
| LLMs and VLMs | LLM-QAT / NVFP4 QAD | LLaMA-30B W4A8KV4 reached 69.9 average versus 42.5 for RTN and 40.7 for SmoothQuant; NVFP4 QAD recovered Llama Nemotron Super V1 49B to 45.6 on AIME25 versus 46.0 in BF16 (Liu et al., 2023, Xin et al., 27 Jan 2026) |
| Multi-view 3D detection | QD-BEV | QD-BEV-Tiny W4A6 achieved 37.2% NDS with 15.8 MB, outperforming BevFormer-Tiny by 1.8% with an 8x model compression (Zhang et al., 2023) |
| Video generation | Q-VDiT | W3A6 achieved a scene consistency of 23.40 (Feng et al., 28 May 2025) |
| Image restoration | QDR | INT8 recovered 96.5% of FP32 performance and reached 442 FPS on NVIDIA Jetson Orin (Sharif et al., 10 Mar 2026) |
Within LLMs, the same pattern appears under very different design choices. LLM-QAT uses data-free logit distillation and quantizes weights, activations, and KV cache (Liu et al., 2023). BitDistiller targets weight-only sub-4-bit quantization with self-distillation and CAKLD, reaching 29.25 MMLU for 2-bit LLaMA-2-7B versus 23.62 for LLM-QAT (Du et al., 2024). UPQ targets instruction-tuned INT2 models by combining INT4 PTQ initialization with JSD-based Distill-QAT, substantially improving MMLU and IFEval over direct NTP-QAT and direct Distill-QAT (Lee et al., 10 Jun 2025). In post-trained systems, NVFP4 QAD is positioned not as a replacement for the original supervised fine-tuning or RL pipeline but as a single additional stage that recovers near-BF16 behavior after quantization (Xin et al., 27 Jan 2026).
Vision shows equally heterogeneous target choices. QFD uses quantized teacher features for classification and detection (Zhu et al., 2023); QD-BEV uses geometry-aware multi-domain feature distillation (Zhang et al., 2023); QDR uses bottleneck-only self-distillation for dense restoration (Sharif et al., 10 Mar 2026). This breadth supports a general characterization of QAD as a modality-conditioned interface between discretized computation and teacher-guided optimization.
6. Empirical behavior, misconceptions, and open problems
A recurring misconception is that any KD term improves QAT. QKD explicitly showed that straightforward KD can underperform quantization-only training because the regularization effect of KD further diminishes the already reduced representation power of a quantized model (Kim et al., 2019). A second misconception is that output-logit distillation is universally sufficient. Transformer encoders required attention-map and attention-output losses to recover self-attention under sub-2-bit quantization (Kim et al., 2022); QFD showed that quantized feature targets can outperform both logit KD and full-precision feature KD in CNN quantization (Zhu et al., 2023); QDR found that decoder-level distillation harms quantized image restoration whereas bottleneck-only self-distillation helps (Sharif et al., 10 Mar 2026).
Teacher quality and teacher size are also less straightforward than in conventional KD. BitDistiller reported that self-distillation with the same architecture outperformed a larger teacher in 2-bit LLM QAT (Du et al., 2024). QKD, by contrast, improved outcomes by adapting the teacher during co-studying rather than enlarging it (Kim et al., 2019). This suggests that teacher compatibility with the student’s quantized hypothesis class may matter more than teacher capacity alone.
Another misconception is that QAD necessarily requires the original training set. LLM-QAT and NVFP4 QAD showed strong results with generated or partial data (Liu et al., 2023, Xin et al., 27 Jan 2026), and SQAKD dispensed with labels altogether (Zhao et al., 2024). Yet these successes do not eliminate the remaining limitations. LLM-QAT explicitly states that 4-bit activation quantization remains insufficient (Liu et al., 2023). BitDistiller is weight-only and identifies binary quantization, vector quantization, and deeper theoretical explanations for teacher choice as open directions (Du et al., 2024). UPQ does not quantize activations and has only been validated up to 8B instruction-tuned models (Lee et al., 10 Jun 2025). QD-BEV improves BEVFormer more dramatically than some other camera-only detectors, indicating architecture dependence (Zhang et al., 2023). QDR still relies on a FP32 teacher and nontrivial training-time overhead, even though its runtime model is edge-efficient (Sharif et al., 10 Mar 2026).
The broader literature therefore points to a restrained conclusion. QAD is not a single recipe but a family of quantization-conditioned distillation strategies whose success depends on aligning four factors: the target of distillation, the way quantization is simulated or parameterized, the schedule by which the student is exposed to low precision, and the compatibility between teacher behavior and the student’s constrained function class. The technical trend is toward more structured and modality-specific objectives—attention distributions, standardized channel features, temporal relation matrices, geometry-aware BEV mappings, bottleneck latents, or pure output KL in post-trained LLMs—rather than toward a universal KD loss.