Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

EventRPG: Event Data Augmentation with Relevance Propagation Guidance (2403.09274v1)

Published 14 Mar 2024 in cs.CV

Abstract: Event camera, a novel bio-inspired vision sensor, has drawn a lot of attention for its low latency, low power consumption, and high dynamic range. Currently, overfitting remains a critical problem in event-based classification tasks for Spiking Neural Network (SNN) due to its relatively weak spatial representation capability. Data augmentation is a simple but efficient method to alleviate overfitting and improve the generalization ability of neural networks, and saliency-based augmentation methods are proven to be effective in the image processing field. However, there is no approach available for extracting saliency maps from SNNs. Therefore, for the first time, we present Spiking Layer-Time-wise Relevance Propagation rule (SLTRP) and Spiking Layer-wise Relevance Propagation rule (SLRP) in order for SNN to generate stable and accurate CAMs and saliency maps. Based on this, we propose EventRPG, which leverages relevance propagation on the spiking neural network for more efficient augmentation. Our proposed method has been evaluated on several SNN structures, achieving state-of-the-art performance in object recognition tasks including N-Caltech101, CIFAR10-DVS, with accuracies of 85.62% and 85.55%, as well as action recognition task SL-Animals with an accuracy of 91.59%. Our code is available at https://github.com/myuansun/EventRPG.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. A low power, fully event-based gesture recognition system. In CVPR, July 2017.
  2. On Pixel-wise Explanations for Non-linear Classifier Decisions by Layer-wise Relevance Propagation. PloS One, 10(7):e0130140, 2015.
  3. Time-ordered recent event (tore) volumes for event cameras. IEEE TPAMI, 45(2):2519–2532, 2022.
  4. Simultaneous Optical Flow and Intensity Estimation from An Event Camera. In CVPR, pp.  884–892, 2016.
  5. E3D: Event-Based 3D Shape Reconstruction. arXiv preprint arXiv:2012.05214, 2020.
  6. A spatial–channel–temporal-fused attention for spiking neural networks. TNNLS, 2023.
  7. Grad-cam++: Generalized Gradient-based Visual Explanations for Deep Convolutional Networks. In WACV, pp.  839–847. IEEE, 2018.
  8. Esvio: Event-based stereo visual inertial odometry. IEEE Robotics and Automation Letters, 2023.
  9. Temporal efficient training of spiking neural network via gradient re-weighting. In ICLR, 2022.
  10. Spikingjelly. https://github.com/fangwei123456/spikingjelly, 2020. Accessed: 2022-09-19.
  11. Deep residual learning in spiking neural networks. NeurIPS, 34:21056–21069, 2021a.
  12. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In ICCV, pp.  2661–2671, 2021b.
  13. Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. In CVPRW, pp.  178–178. IEEE, 2004.
  14. Event-based Vision: A Survey. IEEE TPAMI, 44(1):154–180, 2020.
  15. End-to-end Learning of Representations for Asynchronous Event-based Data. In ICCV, pp.  5633–5643, 2019.
  16. E-raft: Dense Optical Flow from Event Cameras. In 3DV, pp.  197–206. IEEE, 2021.
  17. EventDrop: Data Augmentation for Event-based Learning. In IJCAI, 2021. URL https://arxiv.org/abs/2106.05836.
  18. Understanding individual decisions of cnns via contrastive backpropagation. In ACCV, pp.  119–134. Springer, 2018.
  19. Augmix: A Simple Data Processing Method to Improve Robustness and Uncertainty. In ICLR, 2019.
  20. Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup. In ICML, 2020.
  21. N-imagenet: Towards robust, fine-grained object recognition with event cameras. In ICCV, pp.  2146–2156, 2021.
  22. Optimizing deeper spiking neural networks for dynamic vision sensing. Neural Networks, 144:686–698, 2021a.
  23. Visual explanations from spiking neural networks using inter-spike intervals. Scientific reports, 11(1):1–14, 2021b.
  24. Beyond classification: Directly training spiking neural networks for semantic segmentation. Neuromorphic Computing and Engineering, 2022.
  25. Bio-inspired Stereo Vision System with Silicon Retina Imagers. In International Conference on Computer Vision Systems, pp. 174–183. Springer, 2009.
  26. Learning multiple layers of features from tiny images. 2009.
  27. Hots: a hierarchy of event-based time-surfaces for pattern recognition. IEEE TPAMI, 39(7):1346–1359, 2016.
  28. Spike-flownet: Event-based Optical Flow Estimation with Energy-efficient Hybrid Neural Networks. In ECCV, pp.  366–382. Springer, 2020.
  29. Relevance-cam: Your Model Already Knows Where to Look. In CVPR, pp.  14944–14953, 2021.
  30. Cifar10-dvs: an Event-stream Dataset for Object Classification. Frontiers in Neuroscience, 11:309, 2017.
  31. Neuromorphic Data Augmentation for Training Spiking Neural Networks. In ECCV, 2022. URL https://arxiv.org/abs/2203.06145.
  32. Wolfgang Maass. Networks of Spiking Neurons: the Third Generation of Neural Network Models. Neural Networks, 10(9):1659–1671, 1997.
  33. Misha Mahowald. The Silicon Retina. In An Analog VLSI System for Stereoscopic Vision, pp.  4–65. Springer, 1994.
  34. Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation. In CVPR, pp.  12444–12453, 2022.
  35. Explaining Nonlinear Classification Decisions with Deep Taylor Decomposition. Pattern Recognition, 65:211–222, 2017.
  36. Converting Static Image Datasets to Spiking Neuromorphic Datasets using Saccades. Frontiers in Neuroscience, 9:437, 2015.
  37. EventNeRF: Neural Radiance Fields from a Single Colour Event Camera. arXiv preprint arXiv:2206.11896, 2022.
  38. Event transformer. a sparse-aware solution for efficient event data processing. In CVPR, pp.  2677–2686, 2022.
  39. Grad-cam: Visual Explanations from Deep Networks via Gradient-based Localization. In ICCV, pp.  618–626, 2017.
  40. Eventmix: An efficient data augmentation strategy for event-based learning. Information Sciences, 644:119170, 2023.
  41. Hats: Histograms of averaged time surfaces for robust event-based object classification. In CVPR, pp.  1731–1740, 2018.
  42. 4.1 A 640×\times× 480 Dynamic Vision Sensor with a 9μ𝜇\muitalic_μm Pixel and 300Meps Address-event Representation. In ISSCC, pp.  66–67. IEEE, 2017.
  43. Event-based Motion Segmentation by Motion Compensation. In ICCV, pp.  7244–7253, 2019.
  44. SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization. In ICLR, 2020.
  45. Sl-animals-dvs: event-driven sign language animals dataset. Pattern Analysis and Applications, pp.  1–16, 2021.
  46. Score-CAM: Score-weighted Visual Explanations for Convolutional Neural Networks. In CVPRW, pp.  24–25, 2020.
  47. Stereo hybrid event-frame (shef) cameras for 3d perception. In IROS, pp.  9758–9764. IEEE, 2021.
  48. Cutmix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In ICCV, pp.  6023–6032, 2019.
  49. Mixup: Beyond Empirical Risk Minimization. In ICLR, 2018.
  50. Going deeper with directly-trained larger spiking neural networks. In AAAI, volume 35, pp.  11062–11070, 2021.
  51. Learning Deep Features for Discriminative Localization. In CVPR, pp.  2921–2929, 2016.
  52. Event-based Motion Segmentation with Spatio-temporal Graph Cuts. TNNLS, 2021.
  53. DEVO: Depth-Event Camera Visual Odometry in Challenging Conditions. ICRA, 2022.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now