Detection of Fast-Moving Objects with Neuromorphic Hardware (2403.10677v2)

Abstract: Neuromorphic Computing (NC) and Spiking Neural Networks (SNNs) in particular are often viewed as the next generation of Neural Networks (NNs). NC is a novel bio-inspired paradigm for energy efficient neural computation, often relying on SNNs in which neurons communicate via spikes in a sparse, event-based manner. This communication via spikes can be exploited by neuromorphic hardware implementations very effectively and results in a drastic reductions of power consumption and latency in contrast to regular GPU-based NNs. In recent years, neuromorphic hardware has become more accessible, and the support of learning frameworks has improved. However, available hardware is partially still experimental, and it is not transparent what these solutions are effectively capable of, how they integrate into real-world robotics applications, and how they realistically benefit energy efficiency and latency. In this work, we provide the robotics research community with an overview of what is possible with SNNs on neuromorphic hardware focusing on real-time processing. We introduce a benchmark of three popular neuromorphic hardware devices for the task of event-based object detection. Moreover, we show that an SNN on a neuromorphic hardware is able to run in a challenging table tennis robot setup in real-time.

• The paper presents pioneering work integrating spiking neural networks with event-based cameras to detect fast-moving objects in a robotic table tennis setup.
• It adapts three SNN frameworks (sinabs, MetaTF, Lava) on distinct neuromorphic devices, revealing notable differences in inference times due to hardware interface delays.
• The findings highlight that optimized hardware-software integration is crucial for achieving robust, real-time performance in neuromorphic robotics.

Spiking Neural Networks for Fast-Moving Object Detection: An Evaluation on Neuromorphic Hardware

This paper presents an exploration into the use of Spiking Neural Networks (SNNs) in conjunction with event-based cameras for the detection of fast-moving objects, with a specific application to robotic table tennis. The primary focus is the deployment of SNNs on neuromorphic hardware to assess their practical viability in real-time robotic systems. Specifically, the authors evaluate the performance of SNNs processed on three state-of-the-art neuromorphic edge devices—DynapCNN, Akida, and Loihi2—paired with event-based cameras, which offer asynchronous data capture complementing the event-driven nature of SNNs.

Methodology and Frameworks

The paper leverages three SNN frameworks: sinabs for SynSense’s DynapCNN, MetaTF for BrainChip’s Akida, and Lava for Intel’s Loihi2. Each framework requires adaptations in the SNN architectures to comply with specific hardware constraints, such as restrictions on layer types and input/output resolutions. The method entails directly training the SNNs, thereby circumventing the accuracy drop associated with converted ANN-to-SNN approaches. For benchmarking, the authors provide a real-time application scenario, integrating SNN-based object detection with a robotic arm in a table tennis setup.

Key Findings

The paper reports comparative metrics on error rates and inference times for each hardware configuration. Notably, the accuracy of the various SNN approaches remains robust, with error margins on par with traditional frame-based detection methods. However, significant variation is observed in the inference times and overall processing latencies, influenced by the integration and interface of each neuromorphic device with the system:

• BrainChip Akida: Shows a balanced profile with a mean forward pass time of approximately 2.20 ms and an inference time of 0.89 ms, offering solid real-time performance supported by its PCIe interface.
• SynSense's DynapCNN: Despite an efficient chip inference time of 0.82 ms, the total delay extends to 46.04 ms, attributed to USB-related data transfer delays.
• Intel's Loihi2: While achieving high accuracy, its deployment through a virtual machine incurs significant communication overhead, resulting in non-optimal real-time execution.

These findings underscore the critical factor of hardware configuration in system latency, rather than the processing capabilities of the neuromorphic chips alone.

Practical Implications and Future Prospects

This research indicates promising directions for integrating SNNs and event-based cameras in high-speed robotics applications. The real-time demonstration within a table tennis robot shows feasibility; however, optimization in hardware-software integration is pivotal for broader applicability. The authors argue for ongoing development in consolidating event-based sensors with neuromorphic processors on a singular die, as this could potentially obviate current connectivity bottlenecks and advance the adoption of neuromorphic computing in complex, real-time systems.

Conclusion

This paper contributes valuable insights into the performance and implementation challenges of SNNs on neuromorphic hardware for practical robotics applications. By providing comprehensive benchmarks across multiple frameworks and hardware setups, this work serves as a critical reference point for future research focused on deploying neuromorphic solutions in dynamic environments. Advancements in this domain could lead to more energy-efficient, responsive, and robust robotics systems, accelerating the integration of biologically inspired computation in real-world applications.