Cumulative benefits of combining pruning, quantization, and knowledge distillation

Determine whether combining pruning, quantization, and knowledge distillation produces cumulative compression benefits for Vision Transformers beyond those achieved by applying the techniques individually.

Background

The paper reviews pruning, quantization, and knowledge distillation as three major model-compression techniques for Vision Transformers. Although each technique has been studied separately, the authors note that their interactions may not be additive: sequentially applying individually effective methods can produce compounding accuracy losses, and the performance of one method may depend on the output of another.

The unresolved problem is to establish whether combining these techniques yields genuinely cumulative benefits rather than merely combining their separate compression effects. The paper investigates one particular sequential pipeline for agricultural disease detection, but the broader question is presented as an open research question.

References

Each of these methods has been analyzed in isolation, meaning that whether their combination yields cumulative compression benefits remains an open research question.

— Lightweight Vision Transformer Compression for On-Device Plant Disease Detection in Resource-Constrained Agricultural Field Conditions  (2609.05334 - Kumar et al., 4 Sep 2026) in Section 2.2, “Lightweight architectures and model compression”