Papers
Topics
Authors
Recent
Search
2000 character limit reached

Unveiling the Potential of Spike Streams for Foreground Occlusion Removal from Densely Continuous Views

Published 3 Jul 2023 in cs.CV | (2307.00821v1)

Abstract: The extraction of a clean background image by removing foreground occlusion holds immense practical significance, but it also presents several challenges. Presently, the majority of de-occlusion research focuses on addressing this issue through the extraction and synthesis of discrete images from calibrated camera arrays. Nonetheless, the restoration quality tends to suffer when faced with dense occlusions or high-speed motions due to limited perspectives and motion blur. To successfully remove dense foreground occlusion, an effective multi-view visual information integration approach is required. Introducing the spike camera as a novel type of neuromorphic sensor offers promising capabilities with its ultra-high temporal resolution and high dynamic range. In this paper, we propose an innovative solution for tackling the de-occlusion problem through continuous multi-view imaging using only one spike camera without any prior knowledge of camera intrinsic parameters and camera poses. By rapidly moving the spike camera, we continually capture the dense stream of spikes from the occluded scene. To process the spikes, we build a novel model \textbf{SpkOccNet}, in which we integrate information of spikes from continuous viewpoints within multi-windows, and propose a novel cross-view mutual attention mechanism for effective fusion and refinement. In addition, we contribute the first real-world spike-based dataset \textbf{S-OCC} for occlusion removal. The experimental results demonstrate that our proposed model efficiently removes dense occlusions in diverse scenes while exhibiting strong generalization.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. A 240×\times× 180 130 db 3 μ𝜇\muitalic_μs latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits 49, 10 (2014), 2333–2341.
  2. Pulse-modulation Imaging—Review and Performance Analysis. IEEE Transactions on Biomedical Circuits and Systems 5, 1 (2011), 64–82.
  3. Self-supervised mutual learning for dynamic scene reconstruction of spiking camera. IJCAI.
  4. Spike camera and its coding methods. DCC (2017).
  5. An Efficient Coding Method for Spike Camera Using Inter-Spike Intervals. In 2019 Data Compression Conference (DCC). IEEE, 568–568.
  6. Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 1 (2020), 154–180.
  7. Event-Based Vision: A Survey. IEEE Transactions on PAMI 44, 1 (2022), 154–180. https://doi.org/10.1109/TPAMI.2020.3008413
  8. Optical flow estimation for spiking camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17844–17853.
  9. A dynamic vision sensor with direct logarithmic output and full-frame picture-on-demand. In ISCAS. 1–4.
  10. 1000× Faster Camera and Machine Vision with Ordinary Devices. Engineering (2022).
  11. I See-Through You: A Framework for Removing Foreground Occlusion in Both Sparse and Dense Light Field Images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 229–238.
  12. Mask4D: 4D convolution network for light field occlusion removal. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2480–2484.
  13. Synthetic aperture imaging with events and frames. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17735–17744.
  14. A 128×\times×128 120 dB 15μ𝜇\muitalic_μs Latency Asynchronous Temporal Contrast Vision Sensor. IEEE Journal of Solid-State Circuits 43, 2 (2008), 566–576.
  15. Event-driven sensing for efficient perception: Vision and audition algorithms. IEEE Signal Processing Magazine 36, 6 (2019), 29–37.
  16. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision. 10012–10022.
  17. Richard H. Masland. 2012. The Neuronal Organization of the Retina. Neuron 76, 2 (2012), 266–280.
  18. Synthetic aperture imaging using pixel labeling via energy minimization. Pattern Recognition 46, 1 (2013), 174–187.
  19. Reconstructing occluded surfaces using synthetic apertures: Stereo, focus and robust measures. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), Vol. 2. IEEE, 2331–2338.
  20. Using plane+ parallax for calibrating dense camera arrays. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Vol. 1. IEEE, I–I.
  21. Learning stereo depth estimation with bio-inspired spike cameras. In 2022 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 1–6.
  22. DeOccNet: Learning to see through foreground occlusions in light fields. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 118–127.
  23. Heinz Wässle. 2004. Parallel processing in the mammalian retina. Nature Reviews Neuroscience 5, 10 (2004), 747–757.
  24. Learning super-resolution reconstruction for high temporal resolution spike stream. IEEE Transactions on Circuits and Systems for Video Technology (2021).
  25. All-in-focus synthetic aperture imaging. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13. Springer, 1–15.
  26. Learning to See Through with Events. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).
  27. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5728–5739.
  28. Spike Transformer: Monocular Depth Estimation for Spiking Camera. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII. Springer, 34–52.
  29. Light field occlusion removal network via foreground location and background recovery. Signal Processing: Image Communication 109 (2022), 116853.
  30. Removing Foreground Occlusions in Light Field using Micro-lens Dynamic Filter.. In IJCAI. 1302–1308.
  31. Event-based synthetic aperture imaging with a hybrid network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14235–14244.
  32. Synthetic aperture photography using a moving camera-IMU system. Pattern Recognition 62 (2017), 175–188.
  33. Super resolve dynamic scene from continuous spike streams. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2533–2542.
  34. Spk2imgnet: Learning to reconstruct dynamic scene from continuous spike stream. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11996–12005.
  35. Learning optical flow from continuous spike streams. Advances in Neural Information Processing Systems 35 (2022), 7905–7920.
  36. Spike-Based Motion Estimation for Object Tracking Through Bio-Inspired Unsupervised Learning. IEEE Transactions on Image Processing 32 (2023), 335–349. https://doi.org/10.1109/TIP.2022.3228168
  37. High-speed image reconstruction through short-term plasticity for spiking cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6358–6367.
  38. A retina-inspired sampling method for visual texture reconstruction. In 2019 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 1432–1437.
  39. Retina-like visual image reconstruction via spiking neural model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1438–1446.
  40. NeuSpike-Net: High Speed Video Reconstruction via Bio-inspired Neuromorphic Cameras. In International Conference on Computer Vision. 2400–2409.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up