Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Noisy Spiking Actor Network for Exploration (2403.04162v1)

Published 7 Mar 2024 in cs.LG and cs.NE

Abstract: As a general method for exploration in deep reinforcement learning (RL), NoisyNet can produce problem-specific exploration strategies. Spiking neural networks (SNNs), due to their binary firing mechanism, have strong robustness to noise, making it difficult to realize efficient exploration with local disturbances. To solve this exploration problem, we propose a noisy spiking actor network (NoisySAN) that introduces time-correlated noise during charging and transmission. Moreover, a noise reduction method is proposed to find a stable policy for the agent. Extensive experimental results demonstrate that our method outperforms the state-of-the-art performance on a wide range of continuous control tasks from OpenAI gym.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (23)
  1. Fully spiking actor network with intra-layer connections for reinforcement learning. arXiv preprint arXiv:2401.05444, 2024.
  2. Neural networks and neuroscience-inspired computer vision. Current Biology, 24(18):921–929, 2014.
  3. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
  4. Noisy networks for exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=rywHCPkAW.
  5. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.  1587–1596. PMLR, 2018.
  6. Continuous deep q-learning with model-based acceleration. In International conference on machine learning, pp.  2829–2838. PMLR, 2016.
  7. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.
  8. Nrowan-dqn: A stable noisy network with noise reduction and online weight adjustment for exploration. Expert Systems with Applications, 203:117343, 2022.
  9. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  10. Action noise in off-policy deep reinforcement learning: Impact on exploration and performance. arXiv preprint arXiv:2206.03787, 2022.
  11. A noise-based novel strategy for faster snn training. Neural Computation, 35(9):1593–1608, 2023.
  12. Exploiting noise as a resource for computation and learning in spiking neural networks. Patterns, 4(10):100831, 2023. doi: 10.1016/J.PATTER.2023.100831. URL https://doi.org/10.1016/j.patter.2023.100831.
  13. Improved robustness of reinforcement learning policies upon conversion to spiking neuronal network platforms applied to atari breakout game. Neural Networks, 120:108–115, 2019.
  14. Parameter space noise for exploration. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreview.net/forum?id=ByBAl2eAZ.
  15. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607–617, 2019.
  16. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020.
  17. Neuronal variability: noise or part of the signal? Nature Reviews Neuroscience, 6(5):389–397, 2005.
  18. Reinforcement learning: An introduction. MIT press, 2018.
  19. Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.  6090–6097. IEEE, 2020.
  20. Deep reinforcement learning with population-coded spiking neural network for continuous control. In Conference on Robot Learning, pp.  2016–2029. PMLR, 2021.
  21. Membrane noise. Progress in biophysics and molecular biology, 28:189–265, 1974.
  22. Multiscale dynamic coding improved spiking actor network for reinforcement learning. In Thirty-sixth AAAI conference on artificial intelligence, 2022.
  23. A novel noise injection-based training scheme for better model robustness. arXiv preprint arXiv:2302.10802, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now