Papers
Topics
Authors
Recent
Search
2000 character limit reached

In value-based deep reinforcement learning, a pruned network is a good network

Published 19 Feb 2024 in cs.LG and cs.AI | (2402.12479v3)

Abstract: Recent work has shown that deep reinforcement learning agents have difficulty in effectively using their network parameters. We leverage prior insights into the advantages of sparse training techniques and demonstrate that gradual magnitude pruning enables value-based agents to maximize parameter effectiveness. This results in networks that yield dramatic performance improvements over traditional networks, using only a small fraction of the full network parameters.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (79)
  1. An optimistic perspective on offline reinforcement learning. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 104–114. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/v119/agarwal20c.html.
  2. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304–29320, 2021.
  3. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 28955–28971. Curran Associates, Inc., 2022.
  4. Single-shot pruning for offline reinforcement learning. arXiv preprint arXiv:2112.15579, 2021.
  5. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  6. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, jun 2013. 10.1613/jair.3912.
  7. A distributional perspective on reinforcement learning. In ICML, 2017.
  8. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588:77 – 82, 2020.
  9. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
  10. Towards deeper deep reinforcement learning with spectral normalization. Advances in Neural Information Processing Systems, 34:8242–8255, 2021.
  11. Jax: composable transformations of python+ numpy programs. 2018.
  12. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110, 2018.
  13. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning, pages 1373–1383. PMLR, 2021.
  14. Small batch deep reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=wPqEvmwFEh.
  15. Stabilizing off-policy deep reinforcement learning from pixels. In International Conference on Machine Learning, pages 2784–2810. PMLR, 2022.
  16. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048–2056. PMLR, 2020.
  17. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=OpC-9aBBVJe.
  18. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pages 1407–1416. PMLR, 2018.
  19. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943–2952. PMLR, 2020.
  20. Proto-value networks: Scaling representation learning with auxiliary tasks. In Submitted to The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=oGDKSt9JrZi. under review.
  21. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47–53, 2022.
  22. Revisiting fundamentals of experience replay. In International Conference on Machine Learning, pages 3061–3071. PMLR, 2020.
  23. Noisy networks for exploration. 2018.
  24. Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050–1059. PMLR, 2016.
  25. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019.
  26. The state of sparse training in deep reinforcement learning. In International Conference on Machine Learning, pages 7766–7792. PMLR, 2022.
  27. Automatic noise filtering with dynamic sparse training in deep reinforcement learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’23, page 1932–1941, Richland, SC, 2023. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450394321.
  28. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018.
  29. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023.
  30. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
  31. Array programming with numpy. Nature, 585(7825):357–362, 2020.
  32. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.
  33. Dropout q-functions for doubly efficient reinforcement learning. In International Conference on Learning Representations, 2021.
  34. J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, 9(03):90–95, 2007.
  35. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. pmlr, 2015.
  36. Model-based reinforcement learning for atari. International Conference on Learning Representations, 2020.
  37. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649, 2020.
  38. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020.
  39. Implicit under-parameterization inhibits data-efficient deep reinforcement learning. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=O9bnihsFfXU.
  40. Dr3: Value-based deep reinforcement learning requires explicit regularization. arXiv preprint arXiv:2112.04716, 2021b.
  41. Offline q-learning on diverse multi-task data both scales and generalizes. In The Eleventh International Conference on Learning Representations, 2022.
  42. Plastic: Improving input and label plasticity for sample efficient reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  43. Jaxpruner: A concise library for sparsity research. In Conference on Parsimony and Learning, pages 515–528. PMLR, 2024.
  44. Curvature explains loss of plasticity. arXiv preprint arXiv:2312.00246, 2023.
  45. Regularization matters in policy optimization-an empirical study on continuous control. In International Conference on Learning Representations, 2020.
  46. D. Livne and K. Cohen. Pops: Policy pruning and shrinking for deep reinforcement learning. IEEE Journal of Selected Topics in Signal Processing, 14(4):789–801, May 2020. ISSN 1941-0484. 10.1109/jstsp.2020.2967566. URL http://dx.doi.org/10.1109/JSTSP.2020.2967566.
  47. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=ZkC8wKoLbQ7.
  48. Understanding plasticity in neural networks. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
  49. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, Feb. 2015.
  50. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):2383, 2018.
  51. The primacy bias in deep reinforcement learning. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 16828–16847. PMLR, 17–23 Jul 2022.
  52. Deep reinforcement learning with plasticity injection. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=jucDLW6G9l.
  53. T. E. Oliphant. Python for scientific computing. Computing in Science & Engineering, 9(3):10–20, 2007. 10.1109/MCSE.2007.58.
  54. The difficulty of passive learning in deep reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=nPHA8fGicZk.
  55. Training larger networks for deep reinforcement learning. arXiv preprint arXiv:2102.07920, 2021.
  56. Prioritized experience replay. CoRR, abs/1511.05952, 2016.
  57. D. Schmidt and T. Schmied. Fast and data-efficient training of rainbow: an experimental study on atari. arXiv preprint arXiv:2111.10247, 2021.
  58. Kickstarting deep reinforcement learning. arXiv preprint arXiv:1803.03835, 2018.
  59. Bigger, better, faster: Human-level atari with human-level efficiency. In International Conference on Machine Learning, pages 30365–30380. PMLR, 2023.
  60. D2rl: Deep dense architectures in reinforcement learning. arXiv preprint arXiv:2010.09163, 2020.
  61. Dynamic sparse training for deep reinforcement learning. arXiv preprint arXiv:2106.04217, 2021.
  62. The dormant neuron phenomenon in deep reinforcement learning. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 32145–32168. PMLR, 23–29 Jul 2023. URL https://proceedings.mlr.press/v202/sokar23a.html.
  63. Observational overfitting in reinforcement learning. arXiv preprint arXiv:1912.02975, 2019.
  64. R. S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3(1):9–44, Aug. 1988.
  65. Investigating multi-task pretraining and generalization in reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
  66. RLx2: Training a sparse deep reinforcement learning model from scratch. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=DJEEqoAq7to.
  67. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012.
  68. Deep reinforcement learning with double q-learning. In Proceedings of the Thirthieth AAAI Conference On Artificial Intelligence (AAAI), 2016, 2016. cite arxiv:1509.06461Comment: AAAI 2016.
  69. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02648, 2018.
  70. When to use parametric models in reinforcement learning? Advances in Neural Information Processing Systems, 32, 2019.
  71. G. Van Rossum and F. L. Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
  72. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.
  73. On lottery tickets and minimal task representations in deep reinforcement learning. In International Conference on Learning Representations, 2021.
  74. Dueling network architectures for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, volume 48, pages 1995–2003, 2016.
  75. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
  76. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. In International Conference on Learning Representations, 2019.
  77. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:1804.06893, 2018.
  78. Accelerating the deep reinforcement learning with neural network compression. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2019.
  79. M. Zhu and S. Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, 2017.
Citations (13)
View on Semantic Scholar

Summary

  • The paper demonstrates that gradual magnitude pruning systematically trims redundant parameters to boost deep RL performance.
  • It employs a methodical pruning approach across diverse architectures, achieving notable gains in efficiency and stability.
  • The findings offer practical benefits for resource-constrained settings and suggest new avenues for multi-task, efficient RL research.

Gradual Magnitude Pruning Enhances Deep Reinforcement Learning Networks

Introduction to Sparse Training in Deep RL

The recent literature has spotlighted a pressing issue within the field of deep reinforcement learning (RL): Agents notoriously underutilize their network parameters, hampering scalability and efficiency. This challenge is significant, particularly in an era that demands RL applications to operate over increasingly complex domains. Amidst this backdrop, the emergence of sparse training techniques offers a beacon of hope. Sparse training, notably through gradual magnitude pruning, emerges as a transformative strategy for enhancing the utilization of network parameters, thereby propelling the performance of RL agents to new heights. By rigorously adhering to this methodology, researchers have unlocked dramatic performance gains, setting the stage for a deeper exploration of "scaling laws" within the context of deep RL.

The Pruning Paradigm Shift

Gradual magnitude pruning (GMP) stands out for its methodical approach to sparsifying networks, meticulously trimming the least significant parameters over the course of training. This strategic reduction not only streamlines networks for enhanced computational efficiency but also fosters a refined network architecture that surprisingly outperforms its dense counterpart. The application of GMP across various deep RL agents and architectures—spanning online to offline settings—has consistently demonstrated superior performance, championing the notion of a "scaling law" for deep RL. These observations are pivotal, underscoring GMP's versatility and potential as a universally beneficial technique in the optimization of deep RL networks.

Insights into Performance Gains and Practical Implications

The exploration of GMP reveals intriguing dynamics within the architecture of deep RL agents. Notably, the technique affords considerable performance improvements, particularly when applied to agents employing ResNet architecture. This enhancement is attributed to the increased parameter efficiency and implicit regularization effects induced by the pruning process. Furthermore, GMP exhibits a notable impact on the stability and efficacy of networks in prolonged training scenarios, suggesting a resilience that could be particularly advantageous in applications requiring extensive learning periods.

From a practical standpoint, the ability of GMP to maintain, and in some cases enhance, agent performance with significantly reduced parameter counts offers compelling implications for the deployment of RL in resource-constrained environments. The reduced model complexity aligns well with the requirements of edge computing devices and applications where computational resources are at a premium.

Forward-looking Perspectives and Potential Avenues of Research

The findings presented catalyze a series of questions and opportunities for future research. Investigating the integration of GMP into advanced multi-task, sample-efficient, and generalization-focused RL agents emerges as a promising avenue. Additionally, the stability characteristics of pruned networks beckon a closer examination within the context of methodologies emphasizing fine-tuning or network reincarnation.

Moreover, the evolving landscape of hardware accelerators designed to optimize sparse network training presents an exciting frontier. The synergy between such technological advancements and GMP could potentially unlock new efficiencies and capabilities in the training of deep RL agents.

Concluding Thoughts

In summary, the implementation of gradual magnitude pruning heralds a paradigmatic shift in the optimization of deep reinforcement learning networks. By championing parameter efficiency and unveiling performance enhancements, this technique sets a compelling precedent for future explorations into non-standard network architectures. As the AI community continues on its quest to refine and elevate deep RL, the insights gleaned from this study provide a robust foundation upon which to build more efficient, scalable, and effective RL agents.

Acknowledgements

The work under review illuminates the collective effort and intellectual rigor of its contributors. It also highlights the indispensable role of open-source tools and the vibrant Python community in facilitating cutting-edge research in artificial intelligence and machine learning.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 13 tweets with 345 likes about this paper.

Sign Up for Free