Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning to Robustly Reconstruct Low-light Dynamic Scenes from Spike Streams (2401.10461v2)

Published 19 Jan 2024 in cs.CV

Abstract: As a neuromorphic sensor with high temporal resolution, spike camera can generate continuous binary spike streams to capture per-pixel light intensity. We can use reconstruction methods to restore scene details in high-speed scenarios. However, due to limited information in spike streams, low-light scenes are difficult to effectively reconstruct. In this paper, we propose a bidirectional recurrent-based reconstruction framework, including a Light-Robust Representation (LR-Rep) and a fusion module, to better handle such extreme conditions. LR-Rep is designed to aggregate temporal information in spike streams, and a fusion module is utilized to extract temporal features. Additionally, we have developed a reconstruction benchmark for high-speed low-light scenes. Light sources in the scenes are carefully aligned to real-world conditions. Experimental results demonstrate the superiority of our method, which also generalizes well to real spike streams. Related codes and proposed datasets will be released after publication.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. A 240 × 180 130 db 3 μ⁢s𝜇s\mu\rm sitalic_μ roman_s latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits (JSSC), 49(10):2333–2341, 2014.
  2. Self-supervised mutual learning for dynamic scene reconstruction of spiking camera. In Proceedings of the International Joint Conference on Artificial Intelligence, IJCAI, pages 2859–2866, 2022.
  3. Activity-driven, event-based vision sensors. IEEE International Symposium on Circuits and Systems (ISCAS), pages 2426–2429, 2010.
  4. High-speed scene reconstruction from low-light spike streams. In 2022 IEEE International Conference on Visual Communications and Image Processing (VCIP), pages 1–5. IEEE, 2022.
  5. Optimal biasing and physical limits of dvs event noise. arXiv preprint arXiv:2304.04019, 2023.
  6. Zero-reference deep curve estimation for low-light image enhancement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1780–1789, 2020.
  7. Optical flow estimation for spiking camera. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 17844–17853, 2022.
  8. 1000×\times× faster camera and machine vision with ordinary devices. Engineering, 2022.
  9. Enlightengan: Deep light enhancement without paired supervision. IEEE transactions on image processing, 30:2340–2349, 2021.
  10. Retinomorphic sensing: A novel paradigm for future multimedia computing. In Proceedings of the ACM International Conference on Multimedia (ACMMM), page 144–152, 2021.
  11. Low-light image and video enhancement using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(12):9396–9416, 2021.
  12. Learning to enhance low-light image via zero-reference deep curve estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4225–4238, 2021.
  13. Photon-limited object detection using non-local feature matching and knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3976–3987, 2021.
  14. Uncertainty guided depth fusion for spike camera. arXiv preprint arXiv:2208.12653, 2022.
  15. A 128 × 128 120 db 15 μ⁢s𝜇s\mu\rm sitalic_μ roman_s latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits (JSSC), 43(2):566–576, 2008.
  16. Unsupervised spike depth estimation via cross-modality cross-domain knowledge transfer. arXiv preprint arXiv:2208.12527, 2022.
  17. Yolo in the dark-domain adaptation method for merging multiple models. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16, pages 345–359. Springer, 2020.
  18. Research on improved yolov5 for low-light environment object detection. Electronics, 12(14):3089, 2023.
  19. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
  20. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560, 2018.
  21. Tom D Wilson. On user studies and information needs. Journal of documentation, 37(1):3–15, 1981.
  22. A bio-inspired multi-exposure fusion framework for low-light image enhancement. arXiv preprint arXiv:1711.00591, 2017.
  23. Spike-based optical flow estimation via contrastive learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  24. Learning temporal-ordered representation for spike streams based on discrete wavelet transforms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 137–147, 2023.
  25. Spike transformer: Monocular depth estimation for spiking camera. In European Conference on Computer Vision (ECCV), 2022.
  26. High-speed motion scene reconstruction for spike camera via motion aligned filtering. In International Symposium on Circuits and Systems (ISCAS), pages 1–5, 2020.
  27. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11996–12005, 2021.
  28. Spikingsim: A bio-inspired spiking simulator. In 2022 IEEE International Symposium on Circuits and Systems (ISCAS), pages 3003–3007. IEEE, 2022.
  29. Learninng optical flow from continuous spike streams. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
  30. High-speed image reconstruction through short-term plasticity for spiking cameras. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6358–6367, 2021.
  31. A retina-inspired sampling method for visual texture reconstruction. In IEEE International Conference on Multimedia and Expo (ICME), pages 1432–1437, 2019.
  32. Ultra-high temporal resolution visual reconstruction from a fovea-like spike camera via spiking neuron model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):1233–1249, 2022.
  33. Neuspike-net: High speed video reconstruction via bio-inspired neuromorphic cameras. In IEEE International Conference on Computer Vision (ICCV), pages 2400–2409, 2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now