Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Embodied Neuromorphic Artificial Intelligence for Robotics: Perspectives, Challenges, and Research Development Stack (2404.03325v2)

Published 4 Apr 2024 in cs.RO, cs.AI, cs.LG, and cs.NE

Abstract: Robotic technologies have been an indispensable part for improving human productivity since they have been helping humans in completing diverse, complex, and intensive tasks in a fast yet accurate and efficient way. Therefore, robotic technologies have been deployed in a wide range of applications, ranging from personal to industrial use-cases. However, current robotic technologies and their computing paradigm still lack embodied intelligence to efficiently interact with operational environments, respond with correct/expected actions, and adapt to changes in the environments. Toward this, recent advances in neuromorphic computing with Spiking Neural Networks (SNN) have demonstrated the potential to enable the embodied intelligence for robotics through bio-plausible computing paradigm that mimics how the biological brain works, known as "neuromorphic AI". However, the field of neuromorphic AI-based robotics is still at an early stage, therefore its development and deployment for solving real-world problems expose new challenges in different design aspects, such as accuracy, adaptability, efficiency, reliability, and security. To address these challenges, this paper will discuss how we can enable embodied neuromorphic AI for robotic systems through our perspectives: (P1) Embodied intelligence based on effective learning rule, training mechanism, and adaptability; (P2) Cross-layer optimizations for energy-efficient neuromorphic computing; (P3) Representative and fair benchmarks; (P4) Low-cost reliability and safety enhancements; (P5) Security and privacy for neuromorphic computing; and (P6) A synergistic development for energy-efficient and robust neuromorphic-based robotics. Furthermore, this paper identifies research challenges and opportunities, as well as elaborates our vision for future research development toward embodied neuromorphic AI for robotics.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. C. Bartolozzi, G. Indiveri, and E. Donati, “Embodied neuromorphic intelligence,” Nature communications, vol. 13, no. 1, p. 1024, 2022.
  2. K. Roy, A. Jaiswal, and P. Panda, “Towards spike-based machine intelligence with neuromorphic computing,” Nature, vol. 575, no. 7784, pp. 607–617, 2019.
  3. C. D. Schuman et al., “Opportunities for neuromorphic computing algorithms and applications,” Nature Computational Science, vol. 2, no. 1, pp. 10–19, 2022.
  4. R. V. W. Putra and M. Shafique, “Fspinn: An optimization framework for memory-efficient and energy-efficient spiking neural networks,” IEEE Trans. on Computer-Aided Design of Integrated Circuits and Systems (TCAD), vol. 39, no. 11, pp. 3601–3613, 2020.
  5. R. V. W. Putra and M. Shafique, “lpspikecon: Enabling low-precision spiking neural network processing for efficient unsupervised continual learning on autonomous agents,” in Int. Joint Conf. on Neural Networks (IJCNN), 2022, pp. 1–8.
  6. N. Rathi et al., “Exploring neuromorphic computing based on spiking neural networks: Algorithms to hardware,” ACM Comput. Surv. (CSUR), vol. 55, no. 12, March 2023.
  7. A. Sironi et al., “Hats: Histograms of averaged time surfaces for robust event-based object classification,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 1731–1740.
  8. A. Safa et al., “Fusing event-based camera and radar for slam using spiking neural networks with continual stdp learning,” in 2023 IEEE Int. Conf. on Robotics and Automation (ICRA), 2023, pp. 2782–2788.
  9. P. Diehl and M. Cook, “Unsupervised learning of digit recognition using spike-timing-dependent plasticity,” Frontiers in Computational Neuroscience, vol. 9, p. 99, 2015.
  10. A. Marchisio et al., “R-snn: An analysis and design methodology for robustifying spiking neural networks against adversarial attacks through noise filters for dynamic vision sensors,” in IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2021, pp. 6315–6321.
  11. R. V. W. Putra and M. Shafique, “Spikedyn: A framework for energy-efficient spiking neural networks with continual and unsupervised learning capabilities in dynamic environments,” in 58th ACM/IEEE Design Automation Conf. (DAC), 2021, pp. 1057–1062.
  12. A. Basu et al., “Spiking neural network integrated circuits: A review of trends and future directions,” in IEEE Custom Integrated Circuits Conf. (CICC), 2022, pp. 1–8.
  13. N. Zhu, Z. Xi, C. Wu, F. Zhong, R. Qi, H. Chen, S. Xu, and W. Ji, “Inductive conformal prediction enhanced lstm-snn network: Applications to birds and uavs recognition,” IEEE Geoscience and Remote Sensing Letters, vol. 21, pp. 1–5, 2024.
  14. L. Steffen et al., “Benchmarking highly parallel hardware for spiking neural networks in robotics,” Frontiers in Neuroscience, vol. 15, 2021.
  15. R. Kreiser et al., “Pose estimation and map formation with spiking neural networks: towards neuromorphic slam,” in IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2018, pp. 2159–2166.
  16. ——, “Error estimation and correction in a spiking neural network for map formation in neuromorphic hardware,” in IEEE Int. Conf. on Robotics and Automation (ICRA), 2020, pp. 6134–6140.
  17. X. Zhang et al., “Dynamic obstacle avoidance for unmanned aerial vehicle using dynamic vision sensor,” in Int. Conf. on Artificial Neural Networks (ICANN).   Springer, 2023, pp. 161–173.
  18. W. Yu et al., “Fault-tolerant attitude tracking control driven by spiking nns for unmanned aerial vehicles,” IEEE Trans. on Neural Networks and Learning Systems (TNNLS), pp. 1–13, 2023.
  19. A. Viale et al., “Carsnn: An efficient spiking neural network for event-based autonomous cars on the loihi neuromorphic research processor,” in Int. Joint Conf. on Neural Networks (IJCNN), 2021, pp. 1–10.
  20. Z. Bing et al., “Indirect and direct training of spiking neural networks for end-to-end control of a lane-keeping vehicle,” Neural Networks, vol. 121, pp. 21–36, 2020.
  21. R. de Azambuja et al., “Graceful degradation under noise on brain inspired robot controllers,” in Neural Information Processing.   Cham: Springer International Publishing, 2016, pp. 195–204.
  22. I. Krauhausen et al., “Organic neuromorphic electronics for sensorimotor integration and learning in robotics,” Science Advances, vol. 7, no. 50, p. eabl5068, 2021.
  23. B. Rueckauer et al., “Conversion of continuous-valued deep networks to efficient event-driven networks for image classification,” Frontiers in Neuroscience (FNINS), vol. 11, p. 682, 2017.
  24. E. O. Neftci, H. Mostafa, and F. Zenke, “Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks,” IEEE Signal Processing Magazine, vol. 36, no. 6, pp. 51–63, 2019.
  25. J. Kaiser et al., “Synaptic plasticity dynamics for deep continuous local learning (decolle),” Frontiers in Neuroscience, vol. 14, 2020.
  26. R. V. W. Putra and M. Shafique, “Mantis: Enabling energy-efficient autonomous mobile agents with spiking neural networks,” in 9th Int. Conf. on Automation, Robotics and Applications, 2023, pp. 197–201.
  27. ——, “Q-spinn: A framework for quantizing spiking neural networks,” in Int. Joint Conf. on Neural Networks (IJCNN), 2021, pp. 1–8.
  28. S. Sen et al., “Approximate computing for spiking neural networks,” in DATE, March 2017, pp. 193–198.
  29. S. S. Chowdhury, N. Rathi, and K. Roy, “Towards ultra low latency spiking neural networks for vision and sequential tasks using temporal pruning,” in European Conf. on Computer Vision (ECCV).   Springer, 2022, pp. 709–726.
  30. R. V. W. Putra and M. Shafique, “Topspark: A timestep optimization methodology for energy-efficient spiking neural networks on autonomous mobile agents,” in IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2023, pp. 3561–3567.
  31. B. Na et al., “Autosnn: Towards energy-efficient spiking neural networks,” in Int. Conf. on Machine Learning, 2022, pp. 16 253–16 269.
  32. Y. Kim et al., “Neural architecture search for spiking neural networks,” in European Conf. on Computer Vision (ECCV), 2022, pp. 36–56.
  33. R. V. W. Putra and M. Shafique, “Spikenas: A fast memory-aware neural architecture search framework for spiking neural network systems,” arXiv preprint arXiv:2402.11322, 2024.
  34. P. Achararit et al., “Apnas: Accuracy-and-performance-aware neural architecture search for neural hardware accelerators,” IEEE Access, vol. 8, pp. 165 319–165 334, 2020.
  35. S. Krithivasan et al., “Dynamic spike bundling for energy-efficient spiking neural networks,” in IEEE/ACM Int. Symp. on Low Power Electronics and Design (ISLPED), July 2019, pp. 1–6.
  36. R. V. W. Putra, M. A. Hanif, and M. Shafique, “Sparkxd: A framework for resilient and energy-efficient spiking neural network inference using approximate dram,” in 58th ACM/IEEE Design Automation Conference (DAC), 2021, pp. 379–384.
  37. G. Srinivasan et al., “Spike timing dependent plasticity based enhanced self-learning for efficient pattern recognition in spiking neural networks,” in Int. Joint Conf. on Neural Networks, 2017, pp. 1847–1854.
  38. R. V. W. Putra, M. A. Hanif, and M. Shafique, “Enforcesnn: Enabling resilient and energy-efficient spiking neural network inference considering approximate drams for embedded systems,” Frontiers in Neuroscience (FNINS), vol. 16, p. 937782, 2022.
  39. ——, “Drmap: A generic dram data mapping policy for energy-efficient processing of convolutional neural networks,” in 57th ACM/IEEE Design Automation Conf. (DAC), 2020, pp. 1–6.
  40. ——, “Romanet: Fine-grained reuse-driven off-chip memory access management and data organization for deep neural network accelerators,” IEEE Trans. on Very Large Scale Integration Systems (TVLSI), vol. 29, no. 4, pp. 702–715, 2021.
  41. K. Asifuzzaman et al., “A survey on processing-in-memory techniques: Advances and challenges,” Memories-Materials, Devices, Circuits and Systems (Memori), vol. 4, p. 100022, 2023.
  42. M. Shafique et al., “Towards energy-efficient and secure edge ai: A cross-layer framework iccad special session paper,” in IEEE/ACM Int. Conf. On Computer Aided Design (ICCAD), 2021, pp. 1–9.
  43. R. V. W. Putra, M. A. Hanif, and M. Shafique, “Softsnn: Low-cost fault tolerance for spiking neural network accelerators under soft errors,” in 59th ACM/IEEE Design Automation Conf. (DAC), 2022, pp. 151–156.
  44. G. Brockman et al., “Openai gym,” arXiv preprint:1606.01540, 2016.
  45. E. Krotkov et al., “The darpa robotics challenge finals: Results and perspectives,” The DARPA robotics challenge finals: Humanoid robots to the rescue, pp. 1–26, 2018.
  46. Andrew et al., “Low cost localisation for agricultural robotics,” in Australasian Conf. on Robotics and Automation, 2013, pp. 1–8.
  47. V. Kumar et al., “Robohive: A unified framework for robot learning,” Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2024.
  48. G. Gallego et al., “Event-based vision: A survey,” IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI), vol. 44, no. 1, pp. 154–180, 2022.
  49. T. Schoepe et al., “Event-based sound source localization in neuromorphic systems,” Authorea Preprints, 2023.
  50. R. Muthusamy et al., “Neuromorphic event-based slip detection and suppression in robotic grasping and manipulation,” IEEE Access, vol. 8, pp. 153 364–153 384, 2020.
  51. C. Cadena et al., “Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age,” IEEE Trans. on Robotics (TRO), vol. 32, no. 6, pp. 1309–1332, 2016.
  52. Y. Xing et al., “Firenose on mobile robot in harsh environments,” IEEE Sensors Journal (JSEN), vol. 19, no. 24, pp. 12 418–12 431, 2019.
  53. S. A. Hasib et al., “A comprehensive review of available battery datasets, rul prediction approaches, and advanced battery management,” IEEE Access, vol. 9, pp. 86 166–86 193, 2021.
  54. R. V. W. Putra, M. A. Hanif, and M. Shafique, “Respawn: Energy-efficient fault-tolerance for spiking neural networks considering unreliable memories,” in IEEE/ACM Int. Conf. On Computer Aided Design (ICCAD), 2021, pp. 1–9.
  55. ——, “Rescuesnn: enabling reliable executions on spiking neural network accelerators under permanent faults,” Frontiers in Neuroscience (FNINS), vol. 17, 2023.
  56. R. El-Allami et al., “Securing deep spiking neural networks against adversarial attacks through inherent structural parameters,” in Design, Automation & Test in Europe Conf. & Exhibition, 2021, pp. 774–779.
  57. Y. Kim, Y. Venkatesha, and P. Panda, “Privatesnn: Privacy-preserving spiking neural networks,” in Thirty-Sixth AAAI Conf. on Artificial Intelligence (AAAI).   AAAI Press, 2022, pp. 1192–1200.
  58. F. Nikfam et al., “A homomorphic encryption framework for privacy-preserving spiking neural networks,” Inf., vol. 14, no. 10, p. 537, 2023.
  59. A. Marchisio et al., “Is spiking secure? a comparative study on the security vulnerabilities of spiking and deep neural networks,” in Int. Joint Conf. on Neural Networks (IJCNN), 2020, pp. 1–8.
  60. V. Venceslai et al., “Neuroattack: Undermining spiking neural networks security through externally triggered bit-flips,” in Int. Joint Conf. on Neural Networks (IJCNN), 2020, pp. 1–8.
  61. A. Marchisio et al., “Dvs-attacks: Adversarial attacks on dynamic vision sensors for spiking neural networks,” in Int. Joint Conf. on Neural Networks (IJCNN), 2021, pp. 1–9.
  62. S. Dave et al., “Special session: Towards an agile design methodology for efficient, reliable, and secure ML systems,” in 40th IEEE VLSI Test Symposium (VTS), 2022, pp. 1–14.
  63. A. Marchisio et al., “Rohnas: A neural architecture search framework with conjoint optimization for adversarial robustness and hardware efficiency of convolutional and capsule networks,” IEEE Access, vol. 10, pp. 109 043–109 055, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Rachmad Vidya Wicaksana Putra (30 papers)
  2. Alberto Marchisio (56 papers)
  3. Fakhreddine Zayer (10 papers)
  4. Jorge Dias (30 papers)
  5. Muhammad Shafique (204 papers)
Citations (5)
View on Semantic Scholar

Summary

Embodied Neuromorphic Artificial Intelligence for Robotics: Analyzing the Challenges and Proposing a Comprehensive Research Development Stack

Introduction

Recent advancements have sparked significant interest in integrating neuromorphic computing with robotics to achieve embodied intelligence. However, deploying neuromorphic AI in robotics introduces complex design challenges across accuracy, adaptability, efficiency, reliability, and security. This paper explores the current state, identifies key obstacles, and proposes a research development stack tailored to advance embodied neuromorphic AI in robotics.

Neuromorphic AI-based Robotics

System Overview

Neuromorphic AI-based robotics merges neuromorphic computing with spiking neural networks (SNNs) to enable sophisticated interaction and adaptability within dynamic environments. This integration demands efficient coordination among sensors, computational systems, and actuators, with a keen focus on energy-efficient processing and robustness against reliability and security threats.

Current Trends and Challenges

Despite the promising advantages presented by SNNs, such as ultra-low power consumption, efficient learning mechanisms, and their ability to cater to dynamic environments, the neuromorphic AI-based robotics field is in its nascent stages. Most existing works prioritize learning quality, with little emphasis on benchmarks and considerations for reliability and security. The paper pinpoints multiple challenges, including the need for intelligent systems capable of achieving goals, adaptability, energy-efficient computing, the absence of benchmarks, and measures for reliability, safety, security, and privacy.

Neuromorphic Computing with SNNs

SNNs stand out for their bio-plausible computation model that closely mimics the brain's functionalities, offering potential for efficient spatio-temporal learning. Key components of an SNN, such as network architecture, neuron model, spike/neural coding, and learning rules, are discussed, emphasizing their significance in embodied intelligence.

Perspectives on Embodied Neuromorphic AI Development

Embodied Intelligence

Achieving embodied intelligence requires specialized SNN processing tailored to specific application use-cases, prioritizing learning quality and adaptability. Unsupervised continual learning emerges as a vital tool for adapting to dynamic environments, underscoring the necessity for effective learning rules and mechanisms.

HW/SW-level Optimization

The energy efficiency of neuromorphic computing can be maximized through cross-layer optimizations that span both hardware and software aspects. Techniques like model compression, approximate computations, and hardware-specific optimizations are crucial for enhancing the energy efficiency of neuromorphic processors.

Benchmarks for Robotics

The establishment of representative and fair benchmarks is essential for the continuous development of neuromorphic-based robotics. These benchmarks should cover various robotic tasks and evaluate performance, accuracy, memory footprint, and energy consumption.

Reliability and Safety

Addressing reliability and safety concerns requires the development of low-cost techniques capable of mitigating hardware faults and ensuring safe operations. This involves fault-aware training and mapping techniques, along with strategies to counter transient faults and device aging.

Security and Privacy

Enhancing the security and privacy of neuromorphic computing systems is paramount, given their susceptibility to adversarial attacks and privacy breaches. Lightweight and efficient defense mechanisms are warranted to safeguard against potential threats without compromising computational resources.

Synergistic Development Approach

A synergistic development framework that encompasses holistic optimizations, fault mitigation strategies, and security measures is pivotal for achieving efficient and robust neuromorphic-based robotic systems. This requires a collaborative effort across hardware and software layers, addressing performance, reliability, and security in unison.

Conclusion

The paper articulates the intricacies involved in advancing neuromorphic AI-based robotics, shedding light on significant challenges and proposing a comprehensive research development stack. It emphasizes the importance of embodied intelligence, cross-layer optimizations, reliable benchmarking, as well as reliability and security enhancements. This analysis paves the way for future research endeavors aimed at realizing the full potential of embodied neuromorphic AI in robotics.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube