Towards Optimal Structured CNN Pruning via Generative Adversarial Learning (1903.09291v1)

Abstract: Structured pruning of filters or neurons has received increased focus for compressing convolutional neural networks. Most existing methods rely on multi-stage optimizations in a layer-wise manner for iteratively pruning and retraining which may not be optimal and may be computation intensive. Besides, these methods are designed for pruning a specific structure, such as filter or block structures without jointly pruning heterogeneous structures. In this paper, we propose an effective structured pruning approach that jointly prunes filters as well as other structures in an end-to-end manner. To accomplish this, we first introduce a soft mask to scale the output of these structures by defining a new objective function with sparsity regularization to align the output of baseline and network with this mask. We then effectively solve the optimization problem by generative adversarial learning (GAL), which learns a sparse soft mask in a label-free and an end-to-end manner. By forcing more scaling factors in the soft mask to zero, the fast iterative shrinkage-thresholding algorithm (FISTA) can be leveraged to fast and reliably remove the corresponding structures. Extensive experiments demonstrate the effectiveness of GAL on different datasets, including MNIST, CIFAR-10 and ImageNet ILSVRC 2012. For example, on ImageNet ILSVRC 2012, the pruned ResNet-50 achieves 10.88\% Top-5 error and results in a factor of 3.7x speedup. This significantly outperforms state-of-the-art methods.

An Analysis of "Towards Optimal Structured CNN Pruning via Generative Adversarial Learning"

The paper "Towards Optimal Structured CNN Pruning via Generative Adversarial Learning" introduces a novel approach to the structured pruning of convolutional neural networks (CNNs), leveraging generative adversarial learning (GAL) to address existing challenges in CNN compression. The authors propose an end-to-end pruning method that optimizes the removal of redundant structures efficiently and effectively, enhancing performance on resource-constrained devices without sacrificing significant accuracy.

Key Contributions

1. Soft Mask Integration: The authors introduce a soft mask that scales the output of network structures, facilitating the removal of non-essential components. This mask is optimized using sparsity regularization within a newly defined objective function, aligning pruned network outputs with a baseline network.
2. Generative Adversarial Learning Framework: The optimization of the soft mask is achieved through generative adversarial learning, structured as a two-player game. This approach allows for label-free training, where the generator focuses on reducing network complexity while maintaining accuracy, competing against a discriminator that distinguishes between the pruned and baseline network outputs.
3. Efficiency via FISTA: Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) is employed to optimize the soft mask, enhancing the reliability and speed of the pruning process. This integration eliminates the need for external thresholds to remove near-zero parameters, leading to more effective pruning results.
4. Comprehensive Evaluation: The paper provides extensive empirical validation of the proposed method across various datasets—including MNIST, CIFAR-10, and ImageNet ILSVRC 2012—and different CNN architectures such as ResNets, GoogLeNet, and DenseNets. The results demonstrate superior compression rates and computational speed-ups compared to state-of-the-art pruning methods.

Numerical Results and Claims

The method shows significant improvements in computational efficiency. For example, the pruned ResNet-50 on ImageNet achieves a Top-5 error of 10.88% with a computational speedup factor of 3.7x. This performance not only supports the reduction in computational burden but also exceeds the performance of existing structured pruning methods that require iterative retraining.

Implications and Future Directions

The paper's approach presents practical implications for deploying CNNs in real-world applications where computational resources are constrained, such as mobile and embedded devices. The integration of label-free generative adversarial learning for CNN pruning is an ambitious step towards more flexible and efficient neural network optimization.

Looking forward, the methodology introduced in this paper could inspire further exploration into more generalized pruning strategies that could apply across a broader range of neural network architectures and use cases. Future research might also consider integrating this pruning approach with other model compression strategies like quantization or knowledge distillation to achieve even greater efficiency.

Overall, this work contributes significantly to the toolbox of CNN optimization strategies, providing a robust framework for improving network efficiency while maintaining competitive accuracy.