Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure (1904.03837v1)

Abstract: The redundancy is widely recognized in Convolutional Neural Networks (CNNs), which enables to remove unimportant filters from convolutional layers so as to slim the network with acceptable performance drop. Inspired by the linear and combinational properties of convolution, we seek to make some filters increasingly close and eventually identical for network slimming. To this end, we propose Centripetal SGD (C-SGD), a novel optimization method, which can train several filters to collapse into a single point in the parameter hyperspace. When the training is completed, the removal of the identical filters can trim the network with NO performance loss, thus no finetuning is needed. By doing so, we have partly solved an open problem of constrained filter pruning on CNNs with complicated structure, where some layers must be pruned following others. Our experimental results on CIFAR-10 and ImageNet have justified the effectiveness of C-SGD-based filter pruning. Moreover, we have provided empirical evidences for the assumption that the redundancy in deep neural networks helps the convergence of training by showing that a redundant CNN trained using C-SGD outperforms a normally trained counterpart with the equivalent width.

• The paper introduces Centripetal SGD (C-SGD), a novel optimization method designed to train deep CNN filters to converge, enabling pruning without finetuning by reducing filter redundancy.
• C-SGD helps address the constrained filter pruning problem in complex architectures like ResNets by handling interdependencies between layers, which traditional pruning struggles with.
• Experiments show C-SGD facilitates over 60% FLOPs reduction on standard datasets like CIFAR-10 and ImageNet with negligible or no accuracy loss, demonstrating significant practical efficiency gains.

Insights on Centripetal SGD for Pruning Deep Convolutional Networks

The paper introduces Centripetal SGD (C-SGD), a novel optimizer aimed at enhancing the compressing capacity of deep convolutional neural networks (CNNs), particularly those with intricate architectures. The challenge tackled by the authors relates to filter redundancy in CNNs, and how best to prune these to maintain performance while reducing resource demands such as memory footprint and computational load.

Key Contributions

1. Centripetal SGD Optimization Method: C-SGD is designed to minimize redundancy in CNNs by training filters to converge to identical points in the parameter hyperspace. This allows filters to be pruned without any performance loss, thus avoiding the need for finetuning typically required after pruning using zeroing-out techniques. This approach promises significant efficiency improvements in neural network execution, particularly on resource-constrained platforms like embedded systems.
2. Constrained Filter Pruning Problem: The paper partially resolves the constrained filter pruning problem inherent in modern deep networks with complicated structures, such as ResNets with residual blocks. In these architectures, layers are interdependent, particularly between "pacesetter" and "follower" layers, making traditional pruning techniques insufficient.
3. Experimental Validation and Implications: The authors validated C-SGD through extensive experiments on standard datasets like CIFAR-10 and ImageNet, demonstrating remarkable performance in terms of accuracy retention and computational savings. Specifically, C-SGD facilitated pruning that resulted in over 60% reduction in FLOPs with negligible or no decrease in accuracy—a significant claim backed by rigorous numerical evidence.

Theoretical Implications

Theoretically, the convergence encouraged by C-SGD aligns with the hypothesis that redundancy aids convergence in non-convex optimization problems characteristic of deep learning training. This further implies that redundancy patterns could be leveraged not only for pruning but potentially in facilitating faster and more efficient training processes across various neural architectures.

Practical Implications and Future Directions

From a practical standpoint, the C-SGD methodology can transform deep CNN implementation on low-resource platforms, facilitating the deployment of sophisticated applications that were previously hindered by hardware limitations. As neural networks grow deeper and more complex, finding efficient ways to slim models becomes incredibly vital, and C-SGD provides an intelligent mechanism to address this challenge.

Moreover, the realization that pruning techniques might possess additional benefits—like enhancing network representational capacity—opens avenues for further research. Explorations into optimizing C-SGD settings, integration with other compression techniques, and extending its application to other neural architectures could be valuable directions for future work.

In conclusion, this paper offers a compelling advancement in network pruning techniques through the introduction of Centripetal SGD, especially tailored for modern deep convolutional networks. It navigates both theoretical aspects of model redundancy and practical challenges in constrained pruning, making a significant contribution to the domain of neural network compression and optimization.