Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

EvoPruneDeepTL: An Evolutionary Pruning Model for Transfer Learning based Deep Neural Networks (2202.03844v3)

Published 8 Feb 2022 in cs.NE

Abstract: In recent years, Deep Learning models have shown a great performance in complex optimization problems. They generally require large training datasets, which is a limitation in most practical cases. Transfer learning allows importing the first layers of a pre-trained architecture and connecting them to fully-connected layers to adapt them to a new problem. Consequently, the configuration of the these layers becomes crucial for the performance of the model. Unfortunately, the optimization of these models is usually a computationally demanding task. One strategy to optimize Deep Learning models is the pruning scheme. Pruning methods are focused on reducing the complexity of the network, assuming an expected performance penalty of the model once pruned. However, the pruning could potentially be used to improve the performance, using an optimization algorithm to identify and eventually remove unnecessary connections among neurons. This work proposes EvoPruneDeepTL, an evolutionary pruning model for Transfer Learning based Deep Neural Networks which replaces the last fully-connected layers with sparse layers optimized by a genetic algorithm. Depending on its solution encoding strategy, our proposed model can either perform optimized pruning or feature selection over the densely connected part of the neural network. We carry out different experiments with several datasets to assess the benefits of our proposal. Results show the contribution of EvoPruneDeepTL and feature selection to the overall computational efficiency of the network as a result of the optimization process. In particular, the accuracy is improved, reducing at the same time the number of active neurons in the final layers.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (79)
  1. Transfer learning using cnn for handwritten devanagari character recognition. In 1st International Conference on Advances in Information Technology (ICAIT), Chikmagalur, India, 2019.
  2. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13, Article 32. https://doi.org/10.1145/3005348.
  3. Denser: deep evolutionary network structured representation. Genetic Programming and Evolvable Machines, 20, 5–35. https://doi.org/10.1007/s10710-018-9339-y.
  4. Handbook of Evolutionary Computation. (1st ed.). IOP Publishing Ltd.
  5. Chambers, L. D. (2000). The Practical Handbook of Genetic Algorithms: Applications. (2nd ed.). Chapman and Hall/CRC.
  6. [dataset]Sungjoon Choi (2020). Cataract dataset. Retrieved from https://www.kaggle.com/jr2ngb/cataractdataset. Accessed September 10, 2020.
  7. Neural architecture search: A survey. The Journal of Machine Learning Research, 20, 1997–2017. https://dl.acm.org/doi/10.5555/3322706.3361996.
  8. Efficient multi-objective neural architecture search via lamarckian evolution. In 7th International Conference on Learning Representations (ICLR), New Orleans, LA, USA, 2019.
  9. A study on non-random mating and varying population size in genetic algorithms using a royal road function. In Proceedings of the 2001 Congress on Evolutionary Computation, Seoul, Korea, 2001.
  10. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations (ICLR), New Orleans, LA, USA, 2019.
  11. Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimization and Machine Learning. (1st ed.). Addison-Wesley Longman Publishing Co., Inc.
  12. Coral species identification with texture or structure images using a two-level classifier based on convolutional neural networks. Knowledge-Based Systems, 184, Article 104891. https://doi.org/10.1016/j.knosys.2019.104891.
  13. Deep Learning. (1st ed.). MIT Press.
  14. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory, Singapore, 2005.
  15. Towards highly accurate coral texture images classification using deep convolutional neural networks and data augmentation. Expert Systems with Applications, 118, 315 – 328. https://doi.org/10.1016/j.eswa.2018.10.010.
  16. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, 2016.
  17. Learning both weights and connections for efficient neural network. In Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal Canada, 2015.
  18. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016.
  19. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017.
  20. Iba, H. (2018). Evolutionary Approach to Machine Learning and Deep Neural Networks: Neuro-Evolution and Gene Regulatory Networks. (1st ed.). Springer.
  21. An introduction to variable and feature selection. Journal of Machine Learning Research, 3, 1157–1182. https://dl.acm.org/doi/10.5555/944919.944968.
  22. Polyphonic sound event detection using convolutional bidirectional lstm and synthetic data-based transfer learning. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 2019.
  23. A novel deep learning based framework for the detection and classification of breast cancer using transfer learning. Pattern Recognition Letters, 125, 1–6. https://doi.org/10.1016/j.patrec.2019.03.022.
  24. Task-aware performance prediction for efficient architecture search. In 24th European Conference on Artificial Intelligence, Santiago de Compostela, Spain, 2020.
  25. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, California, CA, USA, 2019.
  26. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, Lake Tahoe, NV, USA, 2012.
  27. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86, 2278–2324. https://doi.org/10.1109/5.726791.
  28. Unsupervised feature learning for audio classification using convolutional deep belief networks. In Advances in Neural Information Processing Systems, Vancouver, B.C., Canada. 2009.
  29. Progressive neural architecture search. In European Conference on Computer Vision, Munich, Germany 2018.
  30. Hierarchical representations for efficient architecture search. In 6th International Conference on Learning Representations (ICLR), Vancouver, B.C., Canada. 2018.
  31. Sparse deep transfer learning for convolutional neural network. In 31st AAAI Conference on Artificial Intelligence (AAAI), San Francisco, CA , USA.
  32. A survey of deep neural network architectures and their applications. Neurocomputing, 234, 11–26. https://doi.org/10.1016/j.neucom.2016.12.038.
  33. Rethinking the value of network pruning. In 7th International Conference on Learning Representations (ICLR), New Orleans, LA, USA, 2019.
  34. A survey of related research on compression and acceleration of deep neural networks. Journal of Physics: Conference Series, 1213, Article 052003. https://doi.org/10.1088/1742-6596/1213/5/052003.
  35. Learning sparse convolutional neural network via quantization with low rank regularization. IEEE Access, 7, 51866–51876. https://doi.org/10.1109/ACCESS.2019.2911536.
  36. Neural architecture transfer. IEEE Transactions on Pattern Analysis & Machine Intelligence, 43, 2971–2989. https://doi.org/10.1109/TPAMI.2021.3052758.
  37. NSGA-Net: Neural architecture search using multi-objective genetic algorithm. In GECCO 2019 - Proceedings of the 2019 Genetic and Evolutionary Computation Conference, Prague, Czech Republic, 2019.
  38. ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression. In IEEE International Conference on Computer Vision, Venice, Italy, 2017.
  39. Genetic algorithm pruning of probabilistic neural networks in medical disease estimation. Neural Networks, 24, 831 – 835. https://doi.org/10.1016/j.neunet.2011.06.003.
  40. Evodeep: a new evolutionary approach for automatic deep neural networks parametrisation. Journal of Parallel and Distributed Computing, 117, 180–191. https://doi.org/10.1016/j.jpdc.2017.09.006.
  41. Lights and shadows in evolutionary deep learning: Taxonomy, critical methodological analysis, cases of study, learned lessons, recommendations and challenges. Information Fusion, 67, 161–194. https://doi.org/10.1016/j.inffus.2020.10.014.
  42. Exploiting prunability for person re-identification. EURASIP Journal on Image and Video Processing, 2021, Article 22. https://doi.org/10.1186/s13640-021-00562-6.
  43. Plant identification using deep neural networks via optimization of transfer learning parameters. Neurocomputing, 235, 228 – 235. https://doi.org/10.1016/j.neucom.2017.01.018.
  44. Adaswarm: Augmenting gradient-based optimizers in deep learning with swarm intelligence. IEEE Transactions on Emerging Topics in Computational Intelligence, 6, 329–340. https://doi.org/10.1109/TETCI.2021.3083428.
  45. [dataset]Laurence Moroney (2019). Rock, paper, scissors dataset. Retrieved from http://www.laurencemoroney.com/rock-paper-scissors-dataset/. Accessed September 10, 2020.
  46. Deep learning for multigrade brain tumor classification in smart healthcare systems: A prospective survey. IEEE Transactions on Neural Networks and Learning Systems, 32, 507–522. https://doi.org/10.1109/TNNLS.2020.2995800.
  47. Deep learning for safe autonomous driving: Current challenges and future directions. IEEE Transactions on Intelligent Transportation Systems, 22, 4316–4336. https://doi.org/10.1109/TITS.2020.3032227.
  48. Pulmonary nodule classification with deep residual networks. International journal of computer assisted radiology and surgery, 12, 1799–1808. https://doi.org/10.1007/s11548-017-1605-6.
  49. Audio-visual speech recognition using deep learning. Applied Intelligence, 42, 722–737. https://doi.org/10.1007/s10489-014-0629-7.
  50. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22, 1345–1359. https://doi.org/10.1109/TKDE.2009.191.
  51. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 2019.
  52. Automl-zero: evolving machine learning algorithms from scratch. In International Conference on Machine Learning, 2020.
  53. Large-scale evolution of image classifiers. In International Conference on Machine Learning, Sydney, Australia, 2017.
  54. Feature selection using deep neural networks. In 2015 International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland, 2015.
  55. [dataset]Virtual Russian Museum (2018). Art images: Drawing/painting/sculptures/engravings. Retrieved from https://www.kaggle.com/thedownhill/art-images-drawings-painting-sculpture-engraving, Accessed September 10, 2020.
  56. Deepfruits: A fruit detection system using deep neural networks. Sensors, 16, Article 1222. https://doi.org/10.3390/s16081222.
  57. Edropout: Energy-based dropout and pruning of deep neural networks. IEEE Transactions on Neural Networks and Learning Systems, (pp. 1–14). In Press https://doi.org/10.1109/TNNLS.2021.3069970.
  58. Evolutionary pruning of transfer learned deep convolutional neural network for breast cancer diagnosis in digital breast tomosynthesis. Physics in Medicine and Biology, 63, Article 095005. https://doi.org/10.1088/1361-6560/aabb5b.
  59. Training deep convolutional neural networks for land–cover classification of high-resolution imagery. IEEE Geoscience and Remote Sensing Letters, 14, 549–553. https://doi.org/10.1109/LGRS.2017.2657778.
  60. Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning. IEEE Transactions on Medical Imaging, 35, 1285–1298. https://doi.org/10.1109/TMI.2016.2528162.
  61. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, San Diego, CA, USA, 2015.
  62. Plantdoc: A dataset for visual plant disease detection. In 7th ACM IKDD CoDS and 25th COMAD, Hyderabad, India, 2020.
  63. Data-free parameter pruning for deep neural networks. In Proceedings of the British Machine Vision Conference (BMVC), Swansea, UK, 2015.
  64. Evolving neural networks through augmenting topologies. Evolutionary Computation, 10, 99–127. https://doi.org/10.1162/106365602320169811.
  65. Advancements in image classification using convolutional neural network. In 2018 Fourth International Conference on Research in Computational Intelligence and Communication Networks (ICRCICN), Kolkata, India, 2018.
  66. Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Transactions on Medical Imaging, 35, 1299–1312. https://doi.org/10.1109/TMI.2016.2535302.
  67. A survey on deep transfer learning. In International Conference On Artificial Neural Networks (ICANN), Rhodes, Greece, 2018.
  68. A citrus fruits and leaves dataset for detection and classification of citrus diseases through machine learning. Data in Brief, 26, Article 104340. https://doi.org/10.1016/j.dib.2019.104340.
  69. Soft weight-sharing for neural network compression. In 5th International Conference on Learning Representations (ICLR), Toulon, France, 2017.
  70. Deep convolutional neural networks for diabetic retinopathy detection by image classification. Computers and Electrical Engineering, 72, 274–282. https://doi.org/10.1016/j.compeleceng.2018.07.042.
  71. Network pruning using sparse learning and genetic algorithm. Neurocomputing, 404, 247 – 256. https://doi.org/10.1016/j.neucom.2020.03.082.
  72. A survey of transfer learning. Journal of Big data, 3, Article 9. https://doi.org/10.1186/s40537-016-0043-6.
  73. A new transfer learning based on vgg-19 network for fault diagnosis. In IEEE 23rd International Conference on Computer Supported Cooperative Work in Design (CSCWD), Porto, Portugal, 2019.
  74. Arrhythmia detection using deep convolutional neural network with long duration ecg signals. Computers in Biology and Medicine, 102, 411–420. https://doi.org/10.1016/j.compbiomed.2018.09.009.
  75. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, Montreal, Canada, 2014.
  76. Application of deep learning in object detection. In IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS), Wuhan, China, 2017.
  77. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Learning Representations (ICLR), Vancouver, B.C. , Canada, 2018.
  78. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018.
  79. Deep learning based feature selection for remote sensing scene classification. IEEE Geoscience and Remote Sensing Letters, 12, 2321–2325. https://doi.org/10.1109/LGRS.2015.2475299.
Citations (29)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now