Data-Free Knowledge Distillation for Deep Neural Networks (1710.07535v2)

Abstract: Recent advances in model compression have provided procedures for compressing large neural networks to a fraction of their original size while retaining most if not all of their accuracy. However, all of these approaches rely on access to the original training set, which might not always be possible if the network to be compressed was trained on a very large dataset, or on a dataset whose release poses privacy or safety concerns as may be the case for biometrics tasks. We present a method for data-free knowledge distillation, which is able to compress deep neural networks trained on large-scale datasets to a fraction of their size leveraging only some extra metadata to be provided with a pretrained model release. We also explore different kinds of metadata that can be used with our method, and discuss tradeoffs involved in using each of them.

• The paper introduces a data-free knowledge distillation method that leverages activation statistics as metadata to create synthetic training data.
• It employs both top-layer and all-layer activation statistics, including spectral methods, to reconstruct neural activations for effective model compression.
• Experimental results demonstrate significant compression with minimal accuracy loss, achieving up to 91.24% accuracy on MNIST fully-connected networks.

Data-Free Knowledge Distillation for Deep Neural Networks

The paper "Data-Free Knowledge Distillation for Deep Neural Networks" authored by Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner presents a novel approach to model compression in scenarios where access to the original training dataset is constrained. Traditional methods for compressing neural networks, such as weight quantization, network pruning, and knowledge distillation, often assume the availability of the original training data. However, this assumption is not always viable, especially in cases where the datasets are large-scale or pose privacy concerns, such as those handling biometric data. This work addresses the unique challenge of compressing deep neural networks without the original training data by leveraging metadata generated during the training phase.

Key Contributions

The main contribution of this paper is the introduction of a data-free knowledge distillation (DFKD) method, which circumvents the need for original training data by using metadata. The method captures and utilizes activation statistics of a pre-trained network during its training phase as a form of metadata. These activation records facilitate the synthesis of a pseudo-dataset that approximates the original training data. This synthetic data then aids in training a compressed student model that approximates the teacher model's performance.

Activation Records and Reconstruction Methods:

• Top Layer Activation Statistics: This approach records statistics of the final layer pre-softmax activations, simplifying the metadata required but potentially leading to under-constrained data reconstruction.
• All Layers Activation Statistics: By considering statistics across all layers, this method provides more constraints, enabling better reconstruction fidelity.
• Spectral Methods: Employing graph Fourier transforms, these methods aim to retain important neural responses in a compressed form, requiring increased computation but yielding higher reconstruction accuracy.

Results and Implications

The approach's efficacy is demonstrated through experiments involving both fully connected and convolutional networks across datasets such as MNIST and CelebA. The method achieves significant compression with minimal accuracy loss, notably with MNIST fully-connected models where spectral methods enabled accuracy of up to 91.24%. This demonstrates the viability of using synthesized datasets derived from activation records for model distillation.

The implications are substantial, offering a promising avenue for model deployment in scenarios where the original training data is inaccessible or when distributing large pre-trained models. By mitigating privacy and data transmission concerns, this method broadens the scope of neural network applications in embedded devices and privacy-sensitive environments.

Future Directions

Future endeavors might include the exploration of more sophisticated activation recording strategies, potentially blending metadata with limited access snippets of the original dataset to further enhance model fidelity. The scalability and efficiency of the spectral method also warrant further optimization to accommodate even larger models or datasets.

Overall, this work expands the frontier of neural network compression by providing a foundation for data-independent knowledge distillation, offering a compelling alternative in the era of increasingly large and private datasets. The integration of metadata-based models could become a critical component in the toolkit for neural network deployment and distribution in data-sensitive applications.