Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multiobjective Evolutionary Pruning of Deep Neural Networks with Transfer Learning for improving their Performance and Robustness (2302.10253v2)

Published 20 Feb 2023 in cs.NE and cs.AI

Abstract: Evolutionary Computation algorithms have been used to solve optimization problems in relation with architectural, hyper-parameter or training configuration, forging the field known today as Neural Architecture Search. These algorithms have been combined with other techniques such as the pruning of Neural Networks, which reduces the complexity of the network, and the Transfer Learning, which lets the import of knowledge from another problem related to the one at hand. The usage of several criteria to evaluate the quality of the evolutionary proposals is also a common case, in which the performance and complexity of the network are the most used criteria. This work proposes MO-EvoPruneDeepTL, a multi-objective evolutionary pruning algorithm. MO-EvoPruneDeepTL uses Transfer Learning to adapt the last layers of Deep Neural Networks, by replacing them with sparse layers evolved by a genetic algorithm, which guides the evolution based in the performance, complexity and robustness of the network, being the robustness a great quality indicator for the evolved models. We carry out different experiments with several datasets to assess the benefits of our proposal. Results show that our proposal achieves promising results in all the objectives, and direct relation are presented among them. The experiments also show that the most influential neurons help us explain which parts of the input images are the most relevant for the prediction of the pruned neural network. Lastly, by virtue of the diversity within the Pareto front of pruning patterns produced by the proposal, it is shown that an ensemble of differently pruned models improves the overall performance and robustness of the trained networks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Denser: deep evolutionary network structured representation. Genetic Programming and Evolvable Machines, 20, 5–35. https://doi.org/10.1007/s10710-018-9339-y.
  2. Handbook of Evolutionary Computation. (1st ed.). IOP Publishing Ltd.
  3. Evolutionary computation: an overview. In Proceedings of IEEE International Conference on Evolutionary Computation, Padua, Italy, 1996. https://doi.org/10.1109/ICEC.1996.542329.
  4. AdaEn-Net: An ensemble of adaptive 2D–3D Fully Convolutional Networks for medical image segmentation. Neural Networks, 126, 76–94. https://doi.org/10.1016/j.neunet.2020.03.007.
  5. AdaResU-Net: Multiobjective adaptive convolutional neural network for medical image segmentation. Neurocomputing, 392, 325–340. https://doi.org/10.1016/j.neucom.2019.01.110.
  6. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82–115. https://doi.org/10.1016/j.inffus.2019.12.012.
  7. Neural Architecture Search with an Efficient Multiobjective Evolutionary Framework. arXiv preprint arXiv:2011.04463, . https://arxiv.org/abs/2011.04463.
  8. Neural Architecture Search Survey: A Hardware Perspective. ACM Computing Surveys, 55, 1–36. https://doi.org/10.1145/3524500.
  9. [dataset]Sungjoon Choi (2020). Cataract Dataset. Retrieved from https://www.kaggle.com/jr2ngb/cataractdataset. Accessed September 10, 2020.
  10. Clune, J. (2019). AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv prepint arXiv:1905.10985, . https://arxiv.org/abs/1905.10985.
  11. Deb, K. (2011). Multi-objective Optimisation Using Evolutionary Algorithms: An Introduction. Springer London.
  12. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6, 182–197. https://doi.org/10.1109/4235.996017.
  13. EDEN: Evolutionary deep networks for efficient machine learning. In 2017 Pattern Recognition Association of South Africa and Robotics and Mechatronics (PRASA-RobMech), Bloemfontein, South Africa, 2017 (pp. 110–115). https://doi.org/10.1109/RoboMech.2017.8261132.
  14. Neural architecture search: A survey. The Journal of Machine Learning Research, 20, 1997–2017. http://jmlr.org/papers/v20/18-598.html.
  15. Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution. In 7th International Conference on Learning Representations (ICLR), New Orleans, LA, USA, 2019. https://doi.org/10.48550/arXiv.1804.09081.
  16. Coral species identification with texture or structure images using a two-level classifier based on Convolutional Neural Networks. Knowledge-Based Systems, 184, 104891. https://doi.org/10.1016/j.knosys.2019.104891.
  17. Learning both Weights and Connections for Efficient Neural Network. In Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal Canada, 2015. https://papers.nips.cc/paper_files/paper/2015/hash/ae0eb3eed39d2bcef4622b2499a05fe6-Abstract.html.
  18. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, . https://doi.org/10.48550/arXiv.1610.02136.
  19. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, New Orleans, USA, 2019. https://openreview.net/forum?id=HyxCxhRcY7.
  20. Sparsity in Deep Learning: Pruning and Growth for Efficient Inference and Training in Neural Networks. Journal of Machine Learning Research, 22, 1–124.
  21. ISO (2021a). Artificial Intelligence (AI) — Assessment of the robustness of neural networks — Part 1: Overview. Technical Report ISO/IEC JTC 1/SC 42 Artificial intelligence (24029-1:2021).
  22. ISO (2021b). Artificial intelligence (AI) — Assessment of the robustness of neural networks — Part 2: Methodology for the use of formal methods. Technical Report ISO/IEC JTC 1/SC 42 Artificial intelligence (24029-2).
  23. A novel deep learning based framework for the detection and classification of breast cancer using transfer learning. Pattern Recognition Letters, 125, 1–6. https://doi.org/10.1016/j.patrec.2019.03.022.
  24. ImageNet Classification with Deep Convolutional Neural Networks. Commun. ACM, 60, 84–90. https://doi.org/10.1145/3065386.
  25. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in Neural Information Processing Systems, 31. https://proceedings.neurips.cc/paper/2018/file/abdeb6f575ac5c6676b747bca8d09cc2-Paper.pdf.
  26. Enhancing the reliability of out-of-distribution image detection in neural networks. In Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, Canada, 2018. https://doi.org/10.48550/arXiv.1706.02690.
  27. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. In International Conference on Learning Representations, Vancouver, Canada, 2018. https://openreview.net/forum?id=H1VGkIxRZ.
  28. MOOD: Multi-level Out-of-distribution Detection. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, Tennessee ,2021. IEEE. https://doi.org/10.1109/cvpr46437.2021.01506.
  29. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 33, 21464–21475. https://proceedings.neurips.cc/paper/2020/hash/f5496252609c43eb8a3d147ab9b9c006-Abstract.html.
  30. DeepMaker: A multi-objective optimization framework for deep neural networks in embedded systems. Microprocessors and Microsystems, 73, 102989. https://doi.org/10.1016/j.micpro.2020.102989.
  31. Surrogate-assisted Multi-objective Neural Architecture Search for Real-time Semantic Segmentation. arXiv preprint arXiv:2208.06820, . https://arxiv.org/abs/2208.06820.
  32. Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment. arXiv preprint arXiv:2208.04321, . https://doi.org/10.48550/arXiv.2208.04321.
  33. NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, Proceedings, Part I. https://doi.org/10.1007/978-3-030-58452-8_3.
  34. NSGA-Net: Neural architecture search using multi-objective genetic algorithm. In GECCO 2019 - Proceedings of the 2019 Genetic and Evolutionary Computation Conference, Prague, Czech Republic, 2019. https://doi.org/10.24963/ijcai.2020/659.
  35. Multiobjective Evolutionary Design of Deep Convolutional Neural Networks for Image Classification. IEEE Transactions on Evolutionary Computation, 25, 277–291. https://doi.org/10.1109/TEVC.2020.3024708.
  36. EvoDeep: a new evolutionary approach for automatic deep neural networks parametrisation. Journal of Parallel and Distributed Computing, 117, 180–191. https://doi.org/10.1016/j.jpdc.2017.09.006.
  37. Lights and shadows in Evolutionary Deep Learning: Taxonomy, critical methodological analysis, cases of study, learned lessons, recommendations and challenges. Information Fusion, 67, 161–194. https://doi.org/10.1016/j.inffus.2020.10.014.
  38. Evolving deep neural networks. In Artificial intelligence in the age of neural networks and brain computing (pp. 293–312). Academic Press. https://doi.org/10.1016/B978-0-12-815480-9.00015-3.
  39. [dataset]Laurence Moroney (2019). Rock, Paper, Scissors Dataset. Retrieved from http://www.laurencemoroney.com/rock-paper-scissors-dataset/. Accessed September 10, 2020.
  40. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering, 22, 1345–1359. https://doi.org/10.1109/TKDE.2009.191.
  41. Open-World Machine Learning: Applications, Challenges, and Opportunities. ACM Comput. Surv., (pp. 1–36). https://doi.org/10.1145/3561381.
  42. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, Stockholm, Sweden, 2018. http://proceedings.mlr.press/v80/pham18a.html.
  43. EvoPruneDeepTL: An evolutionary pruning model for transfer learning based deep neural networks. Neural Networks, 158, 59–82. https://doi.org/10.1016/j.neunet.2022.10.011.
  44. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 2019. https://doi.org/10.1609/aaai.v33i01.33014780.
  45. AutoML-zero: evolving machine learning algorithms from scratch. In International Conference on Machine Learning, 2020. https://doi.org/10.48550/arXiv.2003.03384.
  46. [dataset]Virtual Russian Museum (2018). Art Images: Drawing/Painting/Sculptures/Engravings. Retrieved from https://www.kaggle.com/thedownhill/art-images-drawings-painting-sculpture-engraving, Accessed September 10, 2020.
  47. A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges. arXiv preprint arXiv:2110.14051, . https://doi.org/10.48550/arXiv.2110.14051.
  48. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In 2017 IEEE International Conference on Computer Vision (ICCV), Venezia, Italy, 2017. https://doi.org/10.1109/ICCV.2017.74.
  49. PlantDoc: A Dataset for Visual Plant Disease Detection. In 7th ACM IKDD CoDS and 25th COMAD, Hyderabad, India, 2020. https://doi.org/10.1145/3371158.3371196.
  50. Data-free Parameter Pruning for Deep Neural Networks. In Proceedings of the British Machine Vision Conference (BMVC), Swansea, UK, 2015. https://dblp.org/rec/journals/corr/SrinivasB15.bib.
  51. Evolving neural networks through augmenting topologies. Evolutionary Computation, 10, 99–127. https://doi.org/10.1162/106365602320169811.
  52. Evolution of Deep Convolutional Neural Networks Using Cartesian Genetic Programming. Evolutionary Computation, 28, 141–163. https://doi.org/10.1162/evco_a_00253.
  53. A citrus fruits and leaves dataset for detection and classification of citrus diseases through machine learning. Data in Brief, 26, Article 104340. https://doi.org/10.1016/j.dib.2019.104340.
  54. Hybrid evolutionary approach for Devanagari handwritten numeral recognition using Convolutional Neural Network. Procedia Computer Science, 125, 525–532. https://doi.org/10.1016/j.procs.2017.12.068.
  55. Acceleration of LSTM With Structured Pruning Method on FPGA. IEEE Access, 7, 62930–62937. 10.1109/ACCESS.2019.2917312.
  56. A Computationally Efficient Evolutionary Algorithm for Multiobjective Network Robustness Optimization. IEEE Transactions on Evolutionary Computation, 25, 419–432. https://doi.org/10.1109/TEVC.2020.3048174.
  57. Network pruning using sparse learning and genetic algorithm. Neurocomputing, 404, 247–256. https://doi.org/10.1016/j.neucom.2020.03.082.
  58. MOO-DNAS: Efficient Neural Network Design via Differentiable Architecture Search Based on Multi-Objective Optimization. IEEE Access, 10, 14195–14207. https://doi.org/10.1109/ACCESS.2022.3148323.
  59. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334, . https://doi.org/10.48550/arxiv.2110.11334.
  60. CARS: Continuous Evolution for Efficient Neural Architecture Search. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Virtual, June, 2020. https://doi.org/10.1109/CVPR42600.2020.00190.
  61. Zhou, Z.-H. (2022). Open-environment machine learning. National Science Review, 9, 1–11. https://doi.org/10.1093/nsr/nwac123.
  62. Neural Architecture Search with Reinforcement Learning. arXiv preprint arXiv:1611.01578, . https://arxiv.org/abs/1611.01578.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now