Paraphrasing Complex Network: Network Compression via Factor Transfer (1802.04977v3)

Abstract: Many researchers have sought ways of model compression to reduce the size of a deep neural network (DNN) with minimal performance degradation in order to use DNNs in embedded systems. Among the model compression methods, a method called knowledge transfer is to train a student network with a stronger teacher network. In this paper, we propose a novel knowledge transfer method which uses convolutional operations to paraphrase teacher's knowledge and to translate it for the student. This is done by two convolutional modules, which are called a paraphraser and a translator. The paraphraser is trained in an unsupervised manner to extract the teacher factors which are defined as paraphrased information of the teacher network. The translator located at the student network extracts the student factors and helps to translate the teacher factors by mimicking them. We observed that our student network trained with the proposed factor transfer method outperforms the ones trained with conventional knowledge transfer methods.

• The paper introduces Factor Transfer, a novel compression technique that uses a paraphraser to distill teacher features and a translator to adapt them for the student network.
• The methodology reinterprets traditional knowledge distillation by transforming complex teacher outputs into manageable factors for enhanced student performance.
• Empirical evaluations on CIFAR, ImageNet, and PASCAL VOC demonstrate that Factor Transfer significantly improves accuracy, error rates, and detection mAP.

Paraphrasing Complex Network: Network Compression via Factor Transfer

The paper "Paraphrasing Complex Network: Network Compression via Factor Transfer" introduces a distinctive approach to model compression in deep neural networks (DNNs), focusing specifically on the technique of knowledge transfer. As DNNs have demonstrated substantial capabilities in various computer vision tasks, their deployment in resource-constrained environments, such as embedded systems, necessitates efficient models that balance size and performance. This paper proposes a novel methodology to address this challenge by refining the knowledge transfer process between teacher and student networks.

Methodology Summary

The principal innovation proposed by the authors involves two core components: the paraphraser and the translator. These convolutional modules serve to transform and relay the knowledge from the teacher network to the student network. The paraphraser operates in an unsupervised manner, distilling complex features from the teacher network into more manageable 'teacher factors'. Concurrently, the translator, placed within the student network, adapts these paraphrased insights into 'student factors', thus enhancing the student's ability to emulate the teacher network's performance.

The authors position their method, termed Factor Transfer (FT), against established knowledge distillation techniques such as Knowledge Distillation (KD) and Attention Transfer (AT). Unlike these prior methodologies which directly transpose attention maps or softened outputs, FT involves a reinterpretation and transformation of the teacher's output, accounting for structural differences between the networks.

Empirical Evaluation

The paper evaluates the efficacy of the proposed method across several datasets, including CIFAR-10, CIFAR-100, ImageNet, and PASCAL VOC 2007. The results consistently highlight that FT offers superior performance enhancements compared to both the base student network and conventional knowledge transfer techniques.

• CIFAR-10 and CIFAR-100: FT outperformed AT and KD in various architectural setups, most notably demonstrating significant accuracy enhancements when employing deep networks as teachers. Impressively, in specific configurations, the FT-enhanced student network even outstripped the performance of the teacher model.
• ImageNet: The application of FT led to marked improvements in top-1 and top-5 error rates for ResNet architectures, surpassing KD and AT, thereby affirming the scalability and robustness of the approach for more complex datasets.
• PASCAL VOC 2007: FT was also applied to an object detection task using a Faster-RCNN framework, yielding a noticeable increase in mean average precision (mAP), underscoring the versatility of the method beyond classification tasks.

Theoretical Implications and Future Directions

By focusing on feature-based factor extraction and translation, the authors propose an elegant solution to the inherent discrepancies in teacher-student network structures, addressing a key limitation in straightforward knowledge transfer methods. This approach not only preserves but enhances the critical features needed for generalization and task performance.

The implications of this research are far-reaching. In practical terms, it enables the deployment of high-performing compact networks in real-world scenarios where computational resources are limited. Theoretically, it paves the way for further exploration into the unsupervised extraction of semantic features and their role in the transfer learning paradigm.

Moving forward, potential areas of exploration could include the application of FT in reinforcement learning environments or its integration with neural architecture search methodologies to autonomously discover optimal network structures. Additionally, the authors' insights into the paraphrase and translation mechanism might inform more generalizable strategies for model interpretability and explainability in AI systems.

In conclusion, this paper provides a significant contribution to the field of model compression, offering a robust framework that balances computational efficiency with performance, suitable for a broad range of applications in modern artificial intelligence.