Papers
Topics
Authors
Recent
Search
2000 character limit reached

Graph Expansion in Pruned Recurrent Neural Network Layers Preserve Performance

Published 17 Mar 2024 in cs.LG, cs.CV, and cs.NE | (2403.11100v1)

Abstract: Expansion property of a graph refers to its strong connectivity as well as sparseness. It has been reported that deep neural networks can be pruned to a high degree of sparsity while maintaining their performance. Such pruning is essential for performing real time sequence learning tasks using recurrent neural networks in resource constrained platforms. We prune recurrent networks such as RNNs and LSTMs, maintaining a large spectral gap of the underlying graphs and ensuring their layerwise expansion properties. We also study the time unfolded recurrent network graphs in terms of the properties of their bipartite layers. Experimental results for the benchmark sequence MNIST, CIFAR-10, and Google speech command data show that expander graph properties are key to preserving classification accuracy of RNN and LSTM.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. J. Abderramán Marrero and D.A. Aiat Hadj. Improving formulas for the eigenvalues of finite block-toeplitz tridiagonal matrices. Applied Mathematics and Computation, 382:125324, Oct 2020. URL http://dx.doi.org/10.1016/j.amc.2020.125324.
  2. λ𝜆\lambdaitalic_λ1, isoperimetric inequalities for graphs, and superconcentrators. Journal of Combinatorial Theory, Series B, 38(1):73–88, 1985.
  3. Bruno B. Averbeck. Pruning recurrent neural networks replicates adolescent changes in working memory and reinforcement learning. Proceedings of the National Academy of Sciences, 119(22):e2121331119, 2022. doi: 10.1073/pnas.2121331119. URL https://www.pnas.org/doi/abs/10.1073/pnas.2121331119.
  4. Arindam Biswas. On a cheeger type inequality in cayley graphs of finite groups. European Journal of Combinatorics, 81:298–308, October 2019. doi: 10.1016/j.ejc.2019.06.009. URL https://doi.org/10.1016/j.ejc.2019.06.009.
  5. A Cheeger type inequality in finite Cayley sum graphs. Algebraic Combinatorics, 4(3):517–531, 2021. doi: 10.5802/alco.166. URL https://alco.centre-mersenne.org/articles/10.5802/alco.166/.
  6. Expansion in Cayley graphs, Cayley sum graphs and their twists. arXiv e-prints, art. arXiv:2103.05935, March 2021. doi: 10.48550/arXiv.2103.05935.
  7. Spectra of twists of cayley and cayley sum graphs. Advances in Applied Mathematics, 132:102272, January 2022. doi: 10.1016/j.aam.2021.102272. URL https://doi.org/10.1016/j.aam.2021.102272.
  8. A spectral bound for vertex-transitive graphs and their spanning subgraphs. Algebraic Combinatorics, 6(3):689–706, 2023. doi: 10.5802/alco.278. URL https://alco.centre-mersenne.org/articles/10.5802/alco.278/.
  9. What is the state of neural network pruning? CoRR, abs/2003.03033, 2020.
  10. Sparse neural networks topologies. CoRR, abs/1706.05683, 2017.
  11. Expansion in finite simple groups of lie type. Journal of the European Mathematical Society, 17(6):1367–1434, 2015. doi: 10.4171/jems/533. URL https://doi.org/10.4171/jems/533.
  12. Gabriel Béna and Dan F. M. Goodman. Extreme sparsity gives rise to functional specialization. arXiv, 2021.
  13. Dilated recurrent neural networks. Advances in neural information processing systems, 30, 2017.
  14. Recurrent neural network pruning using dynamical systems and iterative fine-tuning. Neural Networks, 143:475–488, 2021.
  15. Rgp: Neural network pruning through regular graph with edges swapping. IEEE Transactions on Neural Networks and Learning Systems, pp.  1–13, 2023. doi: 10.1109/TNNLS.2023.3280899.
  16. From block-toeplitz matrices to differential equations on graphs: towards a general theory for scalable masked transformers. In Proceedings of the 39th International Conference on Machine Learning, pp.  3962–3983, 2022.
  17. Fan Chung. A generalized alon-boppana bound and weak ramanujan graphs. The Electronic Journal of Combinatorics, 23(3), July 2016. doi: 10.37236/5933. URL https://doi.org/10.37236/5933.
  18. Audio lottery: Speech recognition made ultra-lightweight, noise-robust, and transferable. In International Conference on Learning Representations, 2021.
  19. Jozef Dodziuk. Difference equations, isoperimetric inequality and transience of certain random walks. Transactions of the American Mathematical Society, 284(2):787–794, 1984.
  20. Power law in deep neural networks: Sparse network generation and continual learning with preferential attachment. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  21. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018.
  22. Spectral pruning for recurrent neural networks. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pp.  3458–3482, 2022.
  23. Pruning recurrent neural networks for improved generalization performance. IEEE transactions on neural networks, 5(5):848–851, 1994.
  24. Stephen Grossberg. Recurrent neural networks. Scholarpedia, 8(2):1888, 2013.
  25. Ese: Efficient speech recognition engine with sparse lstm on fpga. In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pp.  75–84, 2017.
  26. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  27. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  28. Detecting modularity in deep neural networks. arXiv preprint arXiv:2110.08058, 2021.
  29. Shlomo Hoory. A lower bound on the spectral radius of the universal cover of a graph. Journal of Combinatorial Theory, Series B, 93(1):33–43, January 2005. doi: 10.1016/j.jctb.2004.06.001. URL https://doi.org/10.1016/j.jctb.2004.06.001.
  30. Expander graphs and their applications. Bulletin of the American Mathematical Society, 43(4):439–561, 2006.
  31. Winning the lottery with neurobiology: faster learning on many cognitive tasks with fixed sparse RNNs. arXiv, 2022.
  32. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017.
  33. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv:1504.00941, 2015.
  34. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  35. Stage-wise magnitude-based pruning for recurrent neural networks. IEEE Transactions on Neural Networks and Learning Systems, pp.  1–15, 2022a.
  36. Stage-wise magnitude-based pruning for recurrent neural networks. IEEE Transactions on Neural Networks and Learning Systems, pp.  1–15, 2022b. doi: 10.1109/TNNLS.2022.3184730.
  37. Efficient recurrent neural networks using structured matrices in fpgas. CoRR, abs/1803.07661, 2018.
  38. Intrinsically sparse long short-term memory networks. arXiv preprint arXiv:1901.09208, 2019.
  39. Andrew W. E McDonald and Ali Shokoufandeh. Sparse super-regular networks. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pp.  1764–1770, 2019.
  40. Yuji Nakatsukasa. Eigenvalue perturbation bounds for Hermitian block tridiagonal matrices, Jan 2012. URL http://dx.doi.org/10.1016/j.apnum.2011.09.010.
  41. Exploring sparsity in recurrent neural networks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=BylSPv9gx.
  42. A study on the ramanujan graph property of winning lottery tickets. In International Conference on Machine Learning, ICML, volume 162, pp.  17186–17201, 2022.
  43. Deep expander networks: Efficient deep networks from graph theory. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  44. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229, 2013.
  45. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  46. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  1–9, 2015.
  47. Sparse activity and sparse connectivity in supervised learning. Journal of Machine Learning Research, 14(33):1091–1143, 2013.
  48. Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition, 2018.
  49. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pp.  7105–7114. PMLR, 2019.
  50. Graph structure of neural networks. In International Conference on Machine Learning, pp.  10881–10891, 2020.
  51. M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp.  818–833. Springer, 2014.
  52. One-shot pruning of recurrent neural networks by jacobian spectrum evaluation. CoRR, abs/1912.00120, 2019.
  53. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.
  54. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  8697–8710, 2018.
  55. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=r1Ue8Hcxg.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up