Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Event-Enhanced Multi-Modal Spiking Neural Network for Dynamic Obstacle Avoidance (2310.02361v1)

Published 3 Oct 2023 in cs.RO

Abstract: Autonomous obstacle avoidance is of vital importance for an intelligent agent such as a mobile robot to navigate in its environment. Existing state-of-the-art methods train a spiking neural network (SNN) with deep reinforcement learning (DRL) to achieve energy-efficient and fast inference speed in complex/unknown scenes. These methods typically assume that the environment is static while the obstacles in real-world scenes are often dynamic. The movement of obstacles increases the complexity of the environment and poses a great challenge to the existing methods. In this work, we approach robust dynamic obstacle avoidance twofold. First, we introduce the neuromorphic vision sensor (i.e., event camera) to provide motion cues complementary to the traditional Laser depth data for handling dynamic obstacles. Second, we develop an DRL-based event-enhanced multimodal spiking actor network (EEM-SAN) that extracts information from motion events data via unsupervised representation learning and fuses Laser and event camera data with learnable thresholding. Experiments demonstrate that our EEM-SAN outperforms state-of-the-art obstacle avoidance methods by a significant margin, especially for dynamic obstacle avoidance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. A cerebellar-based solution to the nondeterministic time delay problem in robotic control. Science Robotics 6, 58 (2021), eabf2756.
  2. Riku Arakawa and Shintaro Shiba. 2020. Exploration of reinforcement learning for event camera using car-like robots. arXiv preprint arXiv:2004.00801 (2020).
  3. A neuromorphic controller for a robotic vehicle equipped with a dynamic vision sensor. Robotics Science and Systems, RSS 2017 (2017).
  4. Error-backpropagation in temporally encoded networks of spiking neurons. Neurocomputing (2002).
  5. Finite meta-dynamic neurons in spiking neural networks for spatio-temporal learning. arXiv preprint arXiv:2010.03140 (2020).
  6. Travis DeWolf. 2021. Spiking neural networks take control. Science Robotics 6, 58 (2021), eabk3268.
  7. Biologically Inspired Dynamic Thresholds for Spiking Neural Networks. In Advances in Neural Information Processing Systems.
  8. How fast is too fast? the role of perception latency in high-speed sense and avoid. IEEE Robotics and Automation Letters 4, 2 (2019), 1884–1891.
  9. Reverse Curriculum Generation for Reinforcement Learning. (2017).
  10. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence (2020).
  11. Wulfram Gerstner. 1995. Time structure of the activity in neural network models. Physical review E (1995).
  12. Wulfram Gerstner and Werner M Kistler. 2002. Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press.
  13. Motif-Topology and Reward-Learning Improved Spiking Neural Network for Efficient Multi-Sensory Integration. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 8917–8921.
  14. Fully spiking variational autoencoder. In Proceedings of the AAAI Conference on Artificial Intelligence.
  15. Complementary Multi-Modal Sensor Fusion for Resilient Robot Pose Estimation in Subterranean Environments. In IEEE International Conference on Unmanned Aircraft Systems 2020.
  16. Spiking neural network (snn) with memristor synapses having non-linear weight update. Frontiers in computational neuroscience 15 (2021), 646125.
  17. Rate Coding Or Direct Coding: Which One Is Better For Accurate, Robust, And Energy-Efficient Spiking Neural Networks?. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 71–75.
  18. Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
  19. Nathan Koenig and Andrew Howard. 2004. Design and use paradigms for gazebo, an open-source multi-robot simulator. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566), Vol. 3. IEEE, 2149–2154.
  20. A 128 ×\times× 128 120 dB 15 μ𝜇\muitalic_μs Latency Asynchronous Temporal Contrast Vision Sensor. IEEE Journal of Solid-State Circuits 43, 2 (2008), 566–576.
  21. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015).
  22. Event-Based Multimodal Spiking Neural Network with Attention Mechanism. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 8922–8926.
  23. Amarnath Mahadevuni and Peng Li. 2017. Navigating mobile robots to target in near shortest time using reinforcement learning with spiking neural networks. In International Joint Conference on Neural Networks (IJCNN). 2243–2250.
  24. David Marr and Tomaso Poggio. 1979. A computational theory of human stereo vision. Proceedings of the Royal Society of London. Series B. Biological Sciences 204, 1156 (1979), 301–328.
  25. Deep polarization reconstruction with PDAVIS events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  26. Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12444–12453.
  27. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529–533.
  28. FastSLAM: A factored solution to the simultaneous localization and mapping problem. Aaai/iaai 593598 (2002).
  29. Towards evasive maneuvers with quadrotors using dynamic vision sensors. In 2015 European Conference on Mobile Robots (ECMR). IEEE, 1–8.
  30. Reinforced imitation: Sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations. IEEE Robotics and Automation Letters 3, 4 (2018), 4423–4430.
  31. Nitin Rathi and Kaushik Roy. 2018. STDP based unsupervised multimodal learning with cross-modal processing in spiking neural networks. IEEE Transactions on Emerging Topics in Computational Intelligence 5, 1 (2018), 143–153.
  32. Nitin Rathi and Kaushik Roy. 2020. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008.03658 (2020).
  33. A Neuro-inspired Approach to Intelligent Collision Avoidance and Navigation. In IEEE/AIAA 39th Digital Avionics Systems Conference.
  34. Evdodgenet: Deep dynamic obstacle dodging with event cameras. In 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 10651–10657.
  35. Finding the gap: neuromorphic motion vision in cluttered environments. arXiv preprint arXiv:2102.08417 (2021).
  36. Learning to time-decode in spiking neural networks through the information bottleneck. arXiv preprint arXiv:2106.01177 (2021).
  37. Ioana Sporea and André Grüning. 2013. Supervised learning in multilayer spiking neural networks. Neural computation 25, 2 (2013), 473–509.
  38. Gesture similarity analysis on event data using a hybrid guided variational auto encoder. arXiv preprint arXiv:2104.00165 (2021).
  39. Wensheng Sun and Dennis L Barbour. 2017. Rate, not selectivity, determines neuronal population coding accuracy in auditory cortex. PLoS biology 15, 11 (2017), e2002459.
  40. RoboCup-Rescue Disaster Simulator Architecture. In Robot Soccer World Cup.
  41. Doron Tal and Eric L Schwartz. 1997. Computing with the leaky integrate-and-fire neuron: logarithmic computation and multiplication. Neural computation 9, 2 (1997), 305–318.
  42. Biologically inspired sleep algorithm for variational auto-encoders. In International Symposium on Visual Computing. Springer, 54–67.
  43. Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 6090–6097.
  44. Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control. In Conference on Robot Learning.
  45. LTMD: Learning Improvement of Spiking Neural Networks with Learnable Thresholding Neurons and Moderate Dropout. In Advances in Neural Information Processing Systems.
  46. Learning to navigate through complex dynamic environment with modular deep reinforcement learning. IEEE Transactions on Games 10, 4 (2018), 400–412.
  47. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience 12 (2018), 331.
  48. Attention spiking neural networks. arXiv preprint arXiv:2209.13929 (2022).
  49. GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks. In Advances in Neural Information Processing Systems.
  50. Improving multi-layer spiking neural networks by incorporating brain-inspired rules. Science China Information Sciences 60, 5 (2017), 1–11.
  51. Multiscale Dynamic Coding improved Spiking Actor Network for Reinforcement Learning. AAAI (2022).
  52. In the blink of an eye: Event-based emotion recognition. In ACM SIGGRAPH 2023 Conference Proceedings. 1–11.
  53. Spiking Transformers for Event-Based Single Object Tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  54. A Closed-Loop Perception, Decision-Making and Reasoning Mechanism for Human-Like Navigation. arXiv preprint arXiv:2207.11901 (2022).
  55. Learning to Navigate in a VUCA Environment: Hierarchical Multi-expert Approach. In IEEE/RSJ International Conference on Intelligent Robots and Systems. 9254–9261.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now