Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Unsupervised Optical Flow Estimation with Dynamic Timing Representation for Spike Camera (2307.06003v1)

Published 12 Jul 2023 in cs.CV

Abstract: Efficiently selecting an appropriate spike stream data length to extract precise information is the key to the spike vision tasks. To address this issue, we propose a dynamic timing representation for spike streams. Based on multi-layers architecture, it applies dilated convolutions on temporal dimension to extract features on multi-temporal scales with few parameters. And we design layer attention to dynamically fuse these features. Moreover, we propose an unsupervised learning method for optical flow estimation in a spike-based manner to break the dependence on labeled data. In addition, to verify the robustness, we also build a spike-based synthetic validation dataset for extreme scenarios in autonomous driving, denoted as SSES dataset. It consists of various corner cases. Experiments show that our method can predict optical flow from spike streams in different high-speed scenes, including real scenes. For instance, our method gets $15\%$ and $19\%$ error reduction from the best spike-based work, SCFlow, in $\Delta t=10$ and $\Delta t=20$ respectively which are the same settings as the previous works.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Event-lstm: An unsupervised and asynchronous learning-based representation for event-based data. IEEE Robotics and Automation Letters, 7(2):4678–4685, 2022.
  2. A database and evaluation methodology for optical flow. International journal of computer vision, 92(1):1–31, 2011.
  3. A differentiable recurrent surface for asynchronous event-based data. In ECCV, pages 136–152, 2020.
  4. Spatio-temporal recurrent networks for event-based optical flow estimation. AAAI, 2021.
  5. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017.
  6. End-to-end learning of representations for asynchronous event-based data. In ICCV, pages 5633–5643, 2019.
  7. Self-supervised learning of event-based optical flow with spiking neural networks. In NeurIPS, volume 34, 2021.
  8. Scflow: Optical flow estimation for spiking camera. CVPR, 2021.
  9. Liteflownet: A lightweight convolutional neural network for optical flow estimation. In CVPR, pages 8981–8989, 2018.
  10. Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness. In ECCV, pages 3–10, 2016.
  11. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In CVPR, pages 9000–9008, 2018.
  12. Learning to estimate hidden motions with global motion aggregation. In ICCV, pages 9772–9781, 2021.
  13. Learning optical flow from a few matches. In CVPR, pages 16592–16600, 2021.
  14. What matters in unsupervised optical flow. In ECCV, pages 557–572, 2020.
  15. What matters in unsupervised optical flow. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV, volume 12347 of Lecture Notes in Computer Science, pages 557–572, 2020.
  16. Spike-flownet: event-based optical flow estimation with energy-efficient hybrid neural networks. In ECCV, pages 366–382, 2020.
  17. Learning by analogy: reliable supervision from transformations for unsupervised optical flow estimation. In CVPR, pages 6489–6498, 2020.
  18. Selflow: Self-supervised learning of optical flow. In CVPR, pages 4571–4580, 2019.
  19. Upflow: Upsampling pyramid for unsupervised optical flow learning. In CVPR, pages 1045–1054, 2021.
  20. Event-based vision meets deep learning on steering prediction for self-driving cars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5419–5427, 2018.
  21. Unflow: Unsupervised learning of optical flow with a bidirectional census loss. In AAAI, volume 32, 2018.
  22. Real-time visual-inertial odometry for event cameras using keyframe-based nonlinear optimization. 2017.
  23. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  24. Hats: Histograms of averaged time surfaces for robust event-based object classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1731–1740, 2018.
  25. Smurf: Self-teaching multi-frame unsupervised raft with full-image warping. In CVPR, pages 3887–3896, 2021.
  26. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In CVPR, pages 8934–8943, 2018.
  27. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, pages 402–419, 2020.
  28. Representation learning for event-based visuomotor policies. NeurIPS, 34, 2021.
  29. Displacement-invariant matching cost learning for accurate optical flow estimation. NeurIPS, 33:15220–15231, 2020.
  30. High-resolution optical flow from 1d attention and correlation. In ICCV, pages 10498–10507, 2021.
  31. Volumetric correspondence networks for optical flow. NeurIPS, 5:12, 2019.
  32. Dystab: Unsupervised object segmentation via dynamic-static bootstrapping. In CVPR, pages 2826–2836, 2021.
  33. Separable flow: Learning motion cost volumes for optical flow estimation. In ICCV, pages 10807–10817, 2021.
  34. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In CVPR, pages 11996–12005, 2021.
  35. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 11996–12005, 2021.
  36. Maskflownet: Asymmetric feature matching with learnable occlusion mask. In CVPR, pages 6278–6287, 2020.
  37. High-speed image reconstruction through short-term plasticity for spiking cameras. In CVPR, pages 6358–6367, 2021.
  38. Motion-attentive transition for zero-shot video object segmentation. In AAAI, volume 34, pages 13066–13073, 2020.
  39. Event-based feature tracking with probabilistic data association. In IEEE International Conference on Robotics and Automation (ICRA), pages 4465–4470, 2017.
  40. Ev-flownet: Self-supervised optical flow estimation for event-based cameras. arXiv preprint arXiv:1802.06898, 2018.
  41. Unsupervised event-based learning of optical flow, depth, and egomotion. In CVPR, pages 989–997, 2019.
  42. A retina-inspired sampling method for visual texture reconstruction. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pages 1432–1437, 2019.
  43. Retina-like visual image reconstruction via spiking neural model. In CVPR, pages 1438–1446, 2020.
  44. Flow-guided feature aggregation for video object detection. In CVPR, pages 408–417, 2017.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now