Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

REPrune: Channel Pruning via Kernel Representative Selection (2402.17862v3)

Published 27 Feb 2024 in cs.CV and cs.AI

Abstract: Channel pruning is widely accepted to accelerate modern convolutional neural networks (CNNs). The resulting pruned model benefits from its immediate deployment on general-purpose software and hardware resources. However, its large pruning granularity, specifically at the unit of a convolution filter, often leads to undesirable accuracy drops due to the inflexibility of deciding how and where to introduce sparsity to the CNNs. In this paper, we propose REPrune, a novel channel pruning technique that emulates kernel pruning, fully exploiting the finer but structured granularity. REPrune identifies similar kernels within each channel using agglomerative clustering. Then, it selects filters that maximize the incorporation of kernel representatives while optimizing the maximum cluster coverage problem. By integrating with a simultaneous training-pruning paradigm, REPrune promotes efficient, progressive pruning throughout training CNNs, avoiding the conventional train-prune-finetune sequence. Experimental results highlight that REPrune performs better in computer vision tasks than existing methods, effectively achieving a balance between acceleration ratio and performance retention.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences.
  2. Automatic channel pruning via clustering and swarm intelligence optimization for CNN. Applied Intelligence.
  3. Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks. ACM SIGARCH Computer Architecture News.
  4. Towards efficient model compression via learned global ranking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  5. Imagenet: A large-scale hierarchical image database. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  6. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  7. Resrep: Lossless cnn pruning via decoupling remembering and forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
  8. Network pruning via transformable architecture search. In Advances in Neural Information Processing Systems.
  9. Cup: Cluster pruning for compressing deep neural networks. 2021 IEEE International Conference on Big Data.
  10. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  11. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations.
  12. Network pruning via performance maximization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  13. Discrete model compression with resource constraint for deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  14. Bootstrap your own latent-a new approach to self-supervised learning. In Advances in Neural Information Processing Systems.
  15. Multi-Dimensional Pruning: A Unified Framework for Model Compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  16. Dmcp: Differentiable markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  17. Gdp: Stabilized neural network pruning via gates with differentiable polarization. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
  18. Deep compression: compressing deep neural network with pruning, trained quantization and Huffman coding. In International Conference on Learning Representations.
  19. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  20. Learning filter pruning criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  21. Soft filter pruning for accelerating deep convolutional neural networks. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence.
  22. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  23. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
  24. Chex: Channel exploration for CNN model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  25. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European Conference on Computer Vision.
  26. Operation-aware soft channel pruning using differentiable masks. In International Conference on Machine Learning.
  27. Filter pruning and re-initialization via latent space clustering. IEEE Access.
  28. Ensemble knowledge guided sub-network search and fine-tuning for filter pruning. In Proceedings of the European Conference on Computer Vision.
  29. Eagleeye: Fast sub-net evaluation for efficient neural network pruning. In Proceedings of the European Conference on Computer Vision.
  30. Dynamic slimmable network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  31. Pruning filters for efficient convnets. In International Conference on Learning Representations.
  32. Compressing convolutional neural networks via factorized convolutional filters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  33. Group sparsity: The hinge between filter pruning and decomposition for network compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  34. Exploiting kernel sparsity and entropy for interpretable CNN compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  35. Differentiable transportation pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
  36. Provable filter pruning for efficient neural networks. In International Conference on Learning Representations.
  37. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  38. Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  39. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision.
  40. Ssd: Single shot multibox detector. In Proceedings of the European Conference on Computer Vision.
  41. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
  42. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
  43. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
  44. Pconv: The missing but desirable sparsity in dnn weight pruning for real-time execution on mobile devices. In Proceedings of the AAAI Conference on Artificial Intelligence.
  45. Milligan, G. W. 1979. Ultrametric hierarchical clustering algorithms. Psychometrika.
  46. Ward’s hierarchical agglomerative clustering method: which algorithms implement ward’s criterion? Journal of Classification.
  47. Dsa: More efficient budgeted pruning via differentiable sparsity allocation. In Proceedings of the European Conference on Computer Vision.
  48. Patdnn: Achieving real-time dnn execution on mobile devices with pattern-based weight pruning. In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems.
  49. Sosp: Efficiently capturing global correlations by second-order structured pruning. In International Conference on Learning Representations.
  50. Balanced column-wise block pruning for maximizing GPU parallelism. In Proceedings of the AAAI Conference on Artificial Intelligence.
  51. Collaborative channel pruning for deep networks. In International Conference on Machine Learning.
  52. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  53. Structural pruning via latency-saliency knapsack. In Advances in Neural Information Processing Systems.
  54. Locally Free Weight Sharing for Network Width Search. In International Conference on Learning Representations.
  55. Chip: Channel independence-based pruning for compact neural networks. In Advances in Neural Information Processing Systems.
  56. Scop: Scientific control for reliable neural network pruning. In Advances in Neural Information Processing Systems.
  57. Neural pruning via growing regularization. In International Conference on Learning Representations.
  58. Ward Jr, J. H. 1963. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association.
  59. Data mining: Practical machine learning tools and techniques with Java implementations. ACM Sigmod Record.
  60. A filter pruning method of CNN models based on feature maps clustering. Applied Sciences.
  61. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. In International Conference on Learning Representations.
  62. Good subnetworks provably exist: Pruning via greedy forward selection. In International Conference on Machine Learning.
  63. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In Advances in Neural Information Processing Systems.
  64. Accelerating convolutional neural networks by group-wise 2D-filter pruning. International Joint Conference on Neural Networks.
  65. Group-based network pruning via nonlinear relationship between convolution filters. Applied Intelligence.
  66. Variational convolutional neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  67. Automatic Attention Pruning: Improving and Automating Model Pruning using Attentions. In International Conference on Artificial Intelligence and Statistics.
  68. Revisit kernel pruning with lottery regulated grouped convolutions. In International Conference on Learning Representations.
  69. Neuron-level structured pruning using polarization regularizer. In Advances in Neural Information Processing Systems.
  70. Discrimination-aware channel pruning for deep neural networks. In Advances in Neural Information Processing Systems.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Mincheol Park (6 papers)
  2. Dongjin Kim (13 papers)
  3. Cheonjun Park (1 paper)
  4. Yuna Park (3 papers)
  5. Gyeong Eun Gong (1 paper)
  6. Won Woo Ro (9 papers)
  7. Suhyun Kim (16 papers)