DiffQ: Differentiable Model Compression

• DiffQ is a differentiable method for compressing neural networks that replaces hard quantization with additive pseudo quantization noise to enable smooth gradient updates.
• It optimizes both real-valued weights and continuous bitwidth parameters, allowing mixed-precision allocation within a unified training objective balancing task loss and model size.
• Empirical results on benchmarks like image classification, language modeling, and audio source separation show competitive compression with minimal accuracy loss.

DiffQ is a differentiable method for neural-network model compression by weight quantization that replaces hard quantization during training with additive pseudo quantization noise, thereby avoiding gradient approximations such as the Straight-Through Estimator (STE). Introduced in “Differentiable Model Compression via Pseudo Quantization Noise” (Défossez et al., 2021), it is designed to optimize both the underlying real-valued weights and the bitwidth allocation used for quantization, including mixed-precision settings in which different weights or groups of weights use different numbers of bits. The method targets end-to-end training under a single tradeoff parameter balancing task loss against compressed model size, and is evaluated on image classification, language modeling, and audio source separation.

1. Concept and problem formulation

DiffQ addresses model compression by weight quantization, especially when low-precision weights are required with minimal accuracy loss and when mixed precision is desirable. In the paper’s formulation, standard uniform quantization maps a real weight $w \in [0,1]$ to one of $2^B$ levels: $$Q(w,B)=\frac{\mathrm{round}\!\left(w(2^B-1)\right)}{2^B-1}.$$ For a weight vector $w\in\mathbb{R}^d$, quantization is preceded by layerwise normalization,

$\hat w=\frac{w-\min(w)}{\max(w)-\min(w)},$

with $\min(w)$ and $\max(w)$ stored separately. The ideal quantization-aware objective is

$\min_{w\in\mathbb{R}^d} L\bigl(f_{Q(w,B)}\bigr),$

but the rounding operator has zero gradient almost everywhere, which makes direct gradient-based optimization difficult (Défossez et al., 2021).

The paper positions DiffQ against the common STE-based approach to quantization-aware training. In that setup, $Q(w,B)$ is used in the forward pass while the quantizer’s Jacobian is replaced by the identity in backpropagation. The paper characterizes this as a usable but biased surrogate-gradient method and gives a one-dimensional least-squares example in which STE causes the learned quantized value to oscillate between adjacent quantization levels rather than converge to the optimal quantized point. This motivates replacing hard quantization during training with a differentiable approximation whose scale depends continuously on the bitwidth.

A central feature of DiffQ is that bitwidth is itself optimized. Rather than fixing $B$ as an integer hyperparameter, the method relaxes bit allocations to continuous variables during training and penalizes model size directly. This makes mixed-precision allocation part of the same optimization problem as task fitting.

2. Pseudo quantization noise and differentiability

The technical core of DiffQ is pseudo quantization noise (PQN). The paper starts from the signal-processing approximation that true quantization noise can be modeled as an independent random variable uniformly distributed in half a quantization bin. If

$2^B$0

then the true quantization noise is

$2^B$1

and the training-time pseudo-quantizer is defined as

$2^B$2

Because the training graph contains additive noise rather than hard rounding, $2^B$3 is differentiable with respect to both the real-valued weight $2^B$4 and the bitwidth parameter $2^B$5. The paper notes in particular that

$2^B$6

and that the dependence on $2^B$7 enters through $2^B$8, so gradients can push the bitwidth upward or downward according to the loss (Défossez et al., 2021).

The paper’s toy analysis compares this with STE on the objective

$2^B$9

Under STE, the expected gradient is

$$Q(w,B)=\frac{\mathrm{round}\!\left(w(2^B-1)\right)}{2^B-1}.$$0

which depends on the quantized value and can induce oscillation between neighboring quantization levels. Replacing $$Q(w,B)=\frac{\mathrm{round}\!\left(w(2^B-1)\right)}{2^B-1}.$$1 by $$Q(w,B)=\frac{\mathrm{round}\!\left(w(2^B-1)\right)}{2^B-1}.$$2 gives

$$Q(w,B)=\frac{\mathrm{round}\!\left(w(2^B-1)\right)}{2^B-1}.$$3

because $$Q(w,B)=\frac{\mathrm{round}\!\left(w(2^B-1)\right)}{2^B-1}.$$4. The paper presents this as an unbiased approximation in the toy setting and as the main analytic argument for PQN.

Although the theoretical exposition uses uniform pseudo noise, the implementation used in the main experiments employs Gaussian noise with unit variance. The paper states that this empirically improved robustness to the final mismatch introduced by hard rounding of both weights and bitwidths at inference time. On ResNet-18/CIFAR-10 at 2.7MB, the paper reports $$Q(w,B)=\frac{\mathrm{round}\!\left(w(2^B-1)\right)}{2^B-1}.$$5 with uniform noise and $$Q(w,B)=\frac{\mathrm{round}\!\left(w(2^B-1)\right)}{2^B-1}.$$6 with Gaussian noise, and all main experiments use Gaussian noise.

3. Compression objective and bitwidth parameterization

DiffQ formulates compression either as constrained optimization of task loss under a model-size budget or as constrained minimization of model size under a target loss. The relaxed training objective is written as

$$Q(w,B)=\frac{\mathrm{round}\!\left(w(2^B-1)\right)}{2^B-1}.$$7

where $$Q(w,B)=\frac{\mathrm{round}\!\left(w(2^B-1)\right)}{2^B-1}.$$8 is the main tradeoff hyperparameter and $$Q(w,B)=\frac{\mathrm{round}\!\left(w(2^B-1)\right)}{2^B-1}.$$9 is a train-time estimate of model size (Défossez et al., 2021).

The paper uses a group-based representation of bit allocation. A tensor of $w\in\mathbb{R}^d$0 weights is partitioned into groups of size $w\in\mathbb{R}^d$1, each group $w\in\mathbb{R}^d$2 being assigned a bitwidth $w\in\mathbb{R}^d$3. This supports per-weight allocation when $w\in\mathbb{R}^d$4, small-group mixed precision for finite $w\in\mathbb{R}^d$5, and effectively layerwise allocation when $w\in\mathbb{R}^d$6. Most experiments use group-wise allocation with default group size $w\in\mathbb{R}^d$7, which the paper describes as the best tradeoff between flexibility and metadata overhead.

The train-time model-size penalty is

$w\in\mathbb{R}^d$8

At evaluation time, the actual representation also includes per-layer min/max scaling values and metadata for encoding the selected bitwidth of each group. The paper gives the true compressed layer size as

$w\in\mathbb{R}^d$9

where

$\hat w=\frac{w-\min(w)}{\max(w)-\min(w)},$0

and $\hat w=\frac{w-\min(w)}{\max(w)-\min(w)},$1 is the final integer bitwidth.

Continuous bitwidths are parameterized through logits $\hat w=\frac{w-\min(w)}{\max(w)-\min(w)},$2 via

$\hat w=\frac{w-\min(w)}{\max(w)-\min(w)},$3

with default settings

$\hat w=\frac{w-\min(w)}{\max(w)-\min(w)},$4

This makes each $\hat w=\frac{w-\min(w)}{\max(w)-\min(w)},$5 differentiable and keeps it within a bounded interval. The same $\hat w=\frac{w-\min(w)}{\max(w)-\min(w)},$6 determines both the noise scale

$\hat w=\frac{w-\min(w)}{\max(w)-\min(w)},$7

and the size penalty $\hat w=\frac{w-\min(w)}{\max(w)-\min(w)},$8, so the learned bitwidth allocation directly reflects the compression–accuracy tradeoff induced by $\hat w=\frac{w-\min(w)}{\max(w)-\min(w)},$9.

The paper also emphasizes that DiffQ optimizes naive bit count rather than entropy-coded size. It notes, for example, that on Wikitext-103 a QAT 8-bit model compresses from 236MB naive to 150MB after ZLib, whereas a DiffQ model changes from 113MB naive to 122MB after ZLib. This suggests that DiffQ encourages fuller use of the available discrete code space rather than low-entropy weight distributions.

4. Training procedure and implementation

The training loop described in the paper is conceptually simple. A standard full-precision model is augmented with trainable bit logits for each weight group. During each forward pass, a group-specific step size

$\min(w)$0

is computed from the current continuous bitwidth, pseudo quantization noise is added to the weights, the task loss is evaluated with the noisy weights, and the model-size penalty is added: $\min(w)$1 Backpropagation then updates both the real-valued weights and the bitwidth parameters, and at inference time the continuous bitwidths are rounded,

$\min(w)$2

followed by standard hard quantization with the resulting integer precisions (Défossez et al., 2021).

The paper states that DiffQ can optimize the number of bits used per individual weight or groups of weights. It also reports architectural patterns in learned allocation: for EfficientNet-B3, earlier layers tend to keep higher bitwidth while deeper layers use lower bitwidth except for the final classifier, and for DeiT the pattern is described as similar but less pronounced and more symmetric across depth.

Implementation details matter for reproducibility. The main experiments use a separate Adam optimizer for the bit logits, typically with learning rate $\min(w)$3, except $\min(w)$4 for language modeling, and $\min(w)$5, $\min(w)$6. Very small tensors are not quantized: below 0.01MB in general and below 0.1MB for DeiT. The paper also notes that tied embeddings and output weights in the LLM require shared bit parameters and shared sampled pseudo noise across reuses; otherwise validation performance degrades significantly.

Code is stated to be available at github.com/facebookresearch/diffq. The method is described as simple to implement because it amounts to injecting noise with bit-dependent scale and adding a model-size penalty, rather than introducing a separate surrogate-gradient mechanism.

5. Empirical performance

DiffQ is evaluated on image classification, language modeling, and audio source separation, and the paper’s overall claim is that it is competitive with or better than STE-based and related baselines in the compression–accuracy tradeoff (Défossez et al., 2021).

A concise summary of several headline results is given below.

Benchmark	DiffQ result	Comparison stated in the paper
DeiT-B / ImageNet	81.5%, 33.02MB	81.8%, 371.4MB uncompressed; 81.6%, 82.9MB for QAT 8-bit
Wikitext-103 / 16-layer Transformer	PPL 18.6, 113MB	PPL 18.1, 942MB uncompressed; PPL 29.9, 118MB for QAT 4-bit
Demucs / MusDB	6.28 SDR, 120MB	6.31 SDR, 1014MB uncompressed; 5.99 SDR, 130MB for QAT 4-bit

The abstract’s most prominent example is ImageNet with DeiT-B, a 12-layer transformer-based model. The paper reports that DiffQ compresses it from 371.4MB to 33.02MB with top-1 accuracy changing from $\min(w)$7 to $\min(w)$8, which the abstract summarizes as more than an $\min(w)$9 compression at lower than 4 bits precision per weight on average, with about $\max(w)$0 accuracy loss. The same table reports QAT 8-bit at $\max(w)$1 and 82.9MB, and QAT 4-bit at $\max(w)$2 and 41.7MB.

On Wikitext-103 with a 16-layer Transformer, the paper reports: uncompressed perplexity $\max(w)$3 at 942MB; QAT 8-bit plus 8-bit activations at $\max(w)$4 and 236MB; QAT 4-bit plus 8-bit activations at $\max(w)$5 and 118MB; LSQ 4-bit plus 8-bit activations at $\max(w)$6 and 118MB; DiffQ at $\max(w)$7 with $\max(w)$8 and 130MB; and DiffQ at $\max(w)$9 with $\min_{w\in\mathbb{R}^d} L\bigl(f_{Q(w,B)}\bigr),$0 and 113MB. The paper interprets this as showing that DiffQ reaches virtually baseline perplexity at about 4.4 bits per weight on average and that QAT “breaks down” at 4 bits in this task.

For audio source separation with Demucs on MusDB, the reported results are 6.31 SDR and 1014MB uncompressed, 6.27 SDR and 162MB for QAT 5-bit, 5.99 SDR and 130MB for QAT 4-bit, and 6.28 SDR and 120MB for DiffQ at $\min_{w\in\mathbb{R}^d} L\bigl(f_{Q(w,B)}\bigr),$1. The paper interprets this as a smaller model than QAT 4-bit with only 0.03dB loss from baseline, compared with 0.3dB loss for QAT 4-bit.

The paper also includes fixed-bit comparisons intended to isolate PQN versus STE rather than mixed-precision allocation. On ResNet-18/CIFAR-10, uncompressed accuracy is $\min_{w\in\mathbb{R}^d} L\bigl(f_{Q(w,B)}\bigr),$2, QAT 2-bit gives $\min_{w\in\mathbb{R}^d} L\bigl(f_{Q(w,B)}\bigr),$3, and DiffQ 2-bit fixed gives $\min_{w\in\mathbb{R}^d} L\bigl(f_{Q(w,B)}\bigr),$4. On ResNet-18/CIFAR-100, QAT 2-bit gives $\min_{w\in\mathbb{R}^d} L\bigl(f_{Q(w,B)}\bigr),$5 while DiffQ 2-bit fixed gives $\min_{w\in\mathbb{R}^d} L\bigl(f_{Q(w,B)}\bigr),$6. On WideResNet/CIFAR-100, QAT 2-bit gives $\min_{w\in\mathbb{R}^d} L\bigl(f_{Q(w,B)}\bigr),$7 while DiffQ 2-bit fixed gives $\min_{w\in\mathbb{R}^d} L\bigl(f_{Q(w,B)}\bigr),$8. The paper presents these as direct evidence for the benefit of pseudo quantization noise itself.

6. Position among related methods and limitations

DiffQ is presented as a differentiable alternative to STE-based quantization-aware training, and the paper repeatedly emphasizes that it avoids surrogate gradients. It is also compared with LSQ, DQ, NICE/UNIQ, Quant-Noise, and Product Quantization in various experiments, with the general conclusion that it is competitive or superior in the compression–accuracy tradeoff on the reported benchmarks (Défossez et al., 2021).

The method also has explicit limitations. First, pseudo quantization noise is an approximation to real quantization noise, and the paper notes that this is less reliable in the extreme low-bit regime, especially around 1 bit. It states that $\min_{w\in\mathbb{R}^d} L\bigl(f_{Q(w,B)}\bigr),$9 is often unstable in practice, although some task-specific experiments alter $Q(w,B)$0. Second, the train-time size penalty models raw bit count rather than entropy-coded size, so it does not directly optimize final compressed entropy. Third, training incurs overhead from the extra bit parameters and the stochastic noise injection. The paper reports batch-time increases such as 115ms to 125ms for language modeling, 120ms to 150ms for ResNet-18 on CIFAR-10, 0.9s to 1.1s for Demucs, and nearly doubling for EfficientNet-B3.

A further limitation is that $Q(w,B)$1 must still be swept to reach a desired size–accuracy operating point, even though it is the only sensitive tradeoff hyperparameter highlighted by the method. The paper also notes the need for careful treatment of shared or tied weights, since bit parameters and sampled noise must be shared consistently across reuses.

Finally, the name “DiffQ” is not unique across the research literature. The data block also includes “DiffQ: Unified Parameter Initialization for Variational Quantum Algorithms via Diffusion Models” (Zhang et al., 22 Sep 2025), which uses the same name for a distinct method in variational quantum algorithms. In the context of model compression and pseudo quantization noise, however, DiffQ refers specifically to the differentiable compression method introduced in 2021 (Défossez et al., 2021).