---
title: QDrop Methodology in Quantization & Unlearning
url: https://www.emergentmind.com/topics/qdrop-methodology
type: topic
---

# QDrop Methodology in Quantization & Unlearning

QDrop Methodology comprises two distinct methodologies prominently referred to as "QDrop" or "QuickDrop" in the literature: (1) QDrop for extremely low-bit post-training quantization (PTQ) in neural networks, and (2) QuickDrop for efficient federated unlearning (FU) by integrated dataset distillation. Both approaches aim at computational efficiency, yet target unrelated domains—model compression for resource-constrained deployments and privacy-compliant model updates, respectively.

## 1. QDrop for Extremely Low-Bit Post-Training Quantization

QDrop [2203.05740] addresses the sub-optimal performance of conventional PTQ schemes under aggressive quantization, particularly in the 2–4 bit regime for activations and weights. Traditional PTQ, applied with reconstruction-based block-wise solutions such as AdaRound or BRECQ, focuses predominantly on weight quantization while treating activation quantization as a post hoc perturbation, which leads to poor generalization and accuracy at ultra-low precisions.

### Theoretical Framework: Activation Quantization, Flatness, and Test Performance

QDrop introduces a theoretical framework where activation quantization is modeled as a multiplicative noise term on the activations:
$$
\tilde A = A \odot (1 + u)
$$
where $A$ denotes the full-precision activation and $u$ is a noise vector reflecting quantization error. The PTQ objective is rewritten to explicitly incorporate this activation noise:
$$
\min_{\hat W} \, \mathbb{E}_{x\sim\mathcal{D}_c} \Big[ L(A(x)\odot(1+u), \hat W) - L(A(x), W)\Big]
$$
This formulation reveals that the loss increase due to quantization decomposes into two terms: (7-1) the effect of weight quantization, and (7-2) the expected loss increase under further activation-induced perturbations of $\hat{W}$. Critically, (7-2) quantifies the "flatness" of the quantized model’s solution under activation noise. Flatness, in this context, corresponds to the model’s robustness to input perturbations and is strongly correlated with generalization capability.

### Random Element-wise Activation Dropping and Loss Surface Flatness

QDrop’s central innovation is to sample $u$ independently per activation coordinate, randomly setting each activation to either full-precision (probability $p$) or its quantized version (probability $1-p$):
$$
u_i = \begin{cases}
0, & \text{with prob. } p; \\
(\lfloor A_i/s\rfloor\cdot s)/A_i - 1, & \text{with prob. } 1-p.
\end{cases}
$$
This randomness ensures that the optimization process during block reconstruction encounters a wide variety of quantization noise patterns, promoting minimization of (7-2) in expectation. As a consequence, the model converges to a flatter loss minimum w.r.t. activation noise, empirically verified by lower Hessian curvature and smoother three-dimensional visualizations of the loss surface.

## 2. QuickDrop for Efficient Federated Unlearning via Dataset Distillation

QuickDrop [2311.15603] addresses the problem of removing the influence of designated data partitions (e.g., specific clients or classes) from a federated model with minimal retraining. It leverages dataset distillation (DD) to create compact, synthetic distilled datasets which enable targeted unlearning at greatly reduced computational cost.

### High-Level Procedure

- **Dataset Distillation**: Each client $i$ compresses its private data $D_i$ into a much smaller distilled set $S_i$ by matching synthetic and real data gradients across learning trajectories:
  $$
  \min_{S} \mathbb{E}_{\theta_0 \sim P_0} \left[ \sum_{t=0}^{T-1} d(\nabla_\theta L^S(\theta_t), \nabla_\theta L^D(\theta_t)) \right]
  $$
- **Integrated Distillation**: The gradient-matching distillation objective is piggy-backed onto normal local FL steps, making the cost of dataset distillation negligible since it re-uses gradients computed for FL updates.
- **Unlearning and Recovery**: Upon a request to unlearn subset $D_f$, clients perform a round of stochastic gradient ascent (SGA) only on the distilled analog $S_f$, followed by 1–2 rounds of standard SGD over the distilled recovery set $S \setminus S_f$ to restore global model accuracy on the non-forgotten data.

The QuickDrop procedure transforms federated unlearning from an $O(|D|)$ operation (where $|D|$ is the sum of all clients’ data points) into an $O(|S|)$ operation, where $|S| \ll |D|$—yielding up to 463.8× empirical speedup compared to naive retraining and 65.1× over previous SGA-based FU approaches.

## 3. Pseudocode and Practical Implementation

### QDrop for PTQ

The QDrop algorithm fits seamlessly into standard block-reconstruction PTQ routines:

1. For each block and input mini-batch:
   - Each activation neuron is assigned full-precision or quantized value using an independent Bernoulli mask at each forward pass.
   - Propagate mixed activations through quantized weights.
2. Compute the reconstruction loss relative to full-precision outputs and backpropagate to update quantization parameters.
3. Proceed blockwise. Key parameters include drop probability $p \in [0.4, 0.6]$ (typ. $p=0.5$), 20,000 updates per block, and default PTQ learning rates.

### QuickDrop for Federated Unlearning

Unified pseudocode (Python-style, with inline mathematical operations):

```python
# Phase 1: FL training with integrated distillation
for round in 0..K-1:
    broadcast global parameters θ_k to all clients
    for client i in parallel:
        for step in 0..T-1:
            # Update model and distilled samples in parallel
            θ_i -= η * ∇_θ L^{D_i}(θ_i)
            S_i -= η_S * ∇_{S_i} d(∇_θ L^{S_i}(θ_i), ∇_θ L^{D_i}(θ_i))
    aggregate θ_{k+1} = weighted average of θ_i

# Phase 2: Unlearning
for client i in parallel:
    θ_i = θ_K + α * ∇_θ L^{S_{f,i}}(θ_K)    # SGA on distilled forgetting set
aggregate θ_U = weighted average of θ_i

# Recovery rounds
for r in 1..R:
    for client i in parallel:
        θ_i = θ_curr - η * ∇_θ L^{S_{r,i}}(θ_curr)
    aggregate θ_curr = weighted average of θ_i
```

Empirical settings use $|S_i|\approx100$ samples per client. Distilled sets are maintained in local storage, and drop/recovery rounds can be efficiently parallelized.

## 4. Empirical Evaluation and Computational Complexity

Empirical results for QDrop/PTQ demonstrate:

- On ImageNet, 2-bit activation quantization accuracy improvement:
  - ResNet-18 W2A2: 46.6% $\to$ 51.1%
  - ResNet-50 W2A2: 47.9% $\to$ 54.7%
  - RegNet-3.2GF W2A2: 39.8% $\to$ 52.4%
- Up to 6 mAP increase for COCO detection and 7–9% accuracy recovery for BERT on GLUE/SQuAD in extremely low-bit settings.

For QuickDrop/FU:

- On CIFAR-10 with 10 clients, QuickDrop unlearning (1 round SGA + 2 recovery rounds) requires 15.61s, compared to 7239.6s (full retraining) and 1046s (prior SGA-Or). 
- The computational savings scale directly with the $|D|/|S|$ ratio and hold for 100-client federations on SVHN (600,000 images).
- QuickDrop supports sequential and parallel unlearning requests without additional asymptotic cost, contingent on the quality of distillation.

## 5. Practical Considerations, Scalability, and Limitations

**PTQ with QDrop**: Calibration requires only 1024 images (ImageNet) or 256 (COCO), no labels needed. The drop mask is sampled afresh for each mini-batch; the method imposes negligible computational overhead on blockwise reconstruction methods such as BRECQ or AdaRound. In NLP models, embeddings and first/last layers are typically quantized to 8 bits for hardware compatibility.

**QuickDrop for FU**: Each client maintains only a distilled set of 100–200 samples, rendering the approach scalable to large $N$. The main limitation lies in the fidelity of the distilled sets; insufficient or poorly trained distilled sets degrade recovery accuracy after unlearning. The method is inherently designed for class-level or client-level unlearning; unlearning arbitrary subsets within a client is not directly supported due to the matching between real samples and their synthetic proxies.

## 6. Impact and Applications

QDrop for PTQ, by optimizing against random quantization noise in the activation space, establishes new state-of-the-art resilience at 2-bit quantization, enabling deployment of highly compact yet accurate deep learning models in vision (classification/detection) and NLP tasks [2203.05740]. QuickDrop yields orders-of-magnitude speedups for federated unlearning workloads, making compliance with user data erasure requests feasible at scale in federated settings without incurring the prohibitive cost of full retraining [2311.15603].

Both methodologies reflect ongoing trends in deep learning: increasing emphasis on efficiency, robustness to noise or data removal, and scalability in distributed environments, each enabled by algorithmic innovations grounded in theoretical model perturbation analysis and dataset distillation.

Source: https://www.emergentmind.com/topics/qdrop-methodology