EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning

Abstract: Finding out the computational redundant part of a trained Deep Neural Network (DNN) is the key question that pruning algorithms target on. Many algorithms try to predict model performance of the pruned sub-nets by introducing various evaluation methods. But they are either inaccurate or very complicated for general application. In this work, we present a pruning method called EagleEye, in which a simple yet efficient evaluation component based on adaptive batch normalization is applied to unveil a strong correlation between different pruned DNN structures and their final settled accuracy. This strong correlation allows us to fast spot the pruned candidates with highest potential accuracy without actually fine-tuning them. This module is also general to plug-in and improve some existing pruning algorithms. EagleEye achieves better pruning performance than all of the studied pruning algorithms in our experiments. Concretely, to prune MobileNet V1 and ResNet-50, EagleEye outperforms all compared methods by up to 3.8%. Even in the more challenging experiments of pruning the compact model of MobileNet V1, EagleEye achieves the highest accuracy of 70.9% with an overall 50% operations (FLOPs) pruned. All accuracy results are Top-1 ImageNet classification accuracy. Source code and models are accessible to open-source community https://github.com/anonymous47823493/EagleEye .

• The paper EagleEye introduces a novel evaluation module using adaptive batch normalization to enhance accuracy prediction of pruned neural networks.
• It demonstrates up to a 3.8% top-1 accuracy improvement and achieves 70.9% accuracy on pruned MobileNet V1 with a 50% reduction in FLOPs.
• The approach integrates seamlessly with existing pruning frameworks, enabling efficient deployment of deep neural networks on resource-constrained devices.

Efficient Neural Network Pruning with EagleEye

In the domain of deep learning, the reduction of computational redundancy in Deep Neural Networks (DNNs) is of paramount importance, particularly for the deployment of models on resource-constrained devices. The paper "EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning" presents a novel method that streamlines this process. The focus is on efficient neural network pruning using an innovative evaluation module called "EagleEye," which leverages adaptive batch normalization to significantly improve the correlation between pruned DNN structures and their final accuracy, thereby improving the pruning process.

The central problem addressed in this study is the effective prediction of performance in pruned sub-networks of DNNs, a task vital to the pruning process. Existing algorithms often face challenges in accuracy or complexity, impeding general application. EagleEye introduces a robust yet straightforward evaluation mechanism that circumvents these issues by incorporating adaptive batch normalization. This technique recalibrates batch normalization statistics to align with pruned network structures, eliminating statistical discrepancies that previously skewed accuracy assessments. This adjustment substantially enhances the correlation between preliminary evaluations of pruned networks and their eventual performance after fine-tuning.

For the quantitative validation of EagleEye, the authors conducted extensive experiments, specifically focusing on pruning MobileNet V1 and ResNet-50 architectures. The results were compelling: EagleEye achieved up to a 3.8% top-1 accuracy improvement over existing methods, with the pruned MobileNet V1 reaching a notable accuracy of 70.9% after a 50% reduction in FLOPs. This efficiency is realized with greatly reduced computational costs compared to traditional methods that rely heavily on resource-intensive fine-tuning processes.

The paper offers a comparative analysis of the correlation between results from traditional (vanilla) evaluation methods and those obtained via EagleEye. It utilizes metrics such as Pearson, Spearman, and Kendall correlation coefficients to demonstrate the superiority of EagleEye's predictions regarding the final accuracy of pruned networks. In all scenarios, EagleEye's adaptive evaluation displayed stronger correlations, suggesting more reliable pruning candidate selection.

A salient feature of EagleEye is its adaptability and the ability to integrate with existing pruning frameworks to enhance their performance. For instance, by replacing the evaluation component in AutoML-based methods like AMC, EagleEye improved the accuracy of pruned networks. This evidence supports the argument for EagleEye's general applicability across different network architectures and pruning methodologies.

The practical implications of this research are substantial. In real-world applications, such as deploying neural networks on mobile or edge devices, reducing computational overhead without sacrificing performance is crucial. By providing a method that is both efficient and highly accurate in preserving network performance post-pruning, EagleEye paves the way for more practical deployment of deep learning models across various platforms.

From a theoretical standpoint, the introduction of a correlation-based evaluation approach grounded in adaptive normalization processes introduces new considerations in the field of neural network compression. It offers a nuanced understanding of how internal network adaptations can lead to more precise pruning strategies. Furthermore, the paper's insights have broader implications for the development of more automated neural architecture search and pruning frameworks.

Future work could aim to broaden the applicability of EagleEye to even more varied architectures and explore integration possibilities with other neural network optimization techniques. The potential for combining EagleEye with advanced strategy-generation methods such as evolutionary algorithms could further enhance its utility in complex model pruning tasks.

In conclusion, EagleEye offers a highly effective solution for neural network pruning, advancing both the theoretical framework and practical methodologies employed in neural network compression. Its adoption could significantly impact the deployment of DNNs in low-resource settings, enhancing the feasibility and performance of deep learning applications across diverse environments.