Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Event-based Shape from Polarization with Spiking Neural Networks (2312.16071v1)

Published 26 Dec 2023 in cs.NE, cs.AI, cs.GR, and cs.LG

Abstract: Recent advances in event-based shape determination from polarization offer a transformative approach that tackles the trade-off between speed and accuracy in capturing surface geometries. In this paper, we investigate event-based shape from polarization using Spiking Neural Networks (SNNs), introducing the Single-Timestep and Multi-Timestep Spiking UNets for effective and efficient surface normal estimation. Specificially, the Single-Timestep model processes event-based shape as a non-temporal task, updating the membrane potential of each spiking neuron only once, thereby reducing computational and energy demands. In contrast, the Multi-Timestep model exploits temporal dynamics for enhanced data extraction. Extensive evaluations on synthetic and real-world datasets demonstrate that our models match the performance of state-of-the-art Artifical Neural Networks (ANNs) in estimating surface normals, with the added advantage of superior energy efficiency. Our work not only contributes to the advancement of SNNs in event-based sensing but also sets the stage for future explorations in optimizing SNN architectures, integrating multi-modal data, and scaling for applications on neuromorphic hardware.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (83)
  1. Poisson surface reconstruction. In Proceedings of the fourth Eurographics symposium on Geometry processing, volume 7, page 0, 2006.
  2. Niid-net: adapting surface normal knowledge for intrinsic image decomposition in indoor scenes. IEEE Transactions on Visualization and Computer Graphics, 26(12):3434–3445, 2020.
  3. Recovering surface normal and arbitrary images: A dual regression network for photometric stereo. IEEE Transactions on Image Processing, 30:3676–3690, 2021.
  4. Multimodal feature-based surface material classification. IEEE transactions on haptics, 10(2):226–239, 2016.
  5. Fast and accurate computation of surface normals from range images. In 2011 IEEE International Conference on Robotics and Automation, pages 3084–3091. IEEE, 2011.
  6. Jason Geng. Structured-light 3d surface imaging: a tutorial. Advances in optics and photonics, 3(2):128–160, 2011.
  7. Berthold KP Horn. Shape from shading: A method for obtaining the shape of a smooth opaque object from one view. 1970.
  8. Using shadows in finding surface orientations. Computer Vision, Graphics, and Image Processing, 22(1):145–176, 1983.
  9. Andrew P Witkin. Recovering surface shape and orientation from texture. Artificial intelligence, 17(1-3):17–45, 1981.
  10. Carlos Hernández. Stereo and silhouette fusion for 3d object modeling from uncalibrated images under circular motion. These de doctorat, École Nationale Supérieure des Télécommunications, 2, 2004.
  11. Robert J Woodham. Photometric method for determining surface orientation from multiple images. Optical engineering, 19(1):139–144, 1980.
  12. Reconstruction of specular surfaces using polarization imaging. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 1, pages I–I. IEEE, 2001.
  13. Polarized 3d: High-quality depth sensing with polarization cues. In Proceedings of the IEEE International Conference on Computer Vision, pages 3370–3378, 2015.
  14. Constraining object features using a polarization reflectance model. Phys. Based Vis. Princ. Pract. Radiom, 1:167, 1993.
  15. Linear differential constraints for photo-polarimetric height estimation. In Proceedings of the IEEE international conference on computer vision, pages 2279–2287, 2017.
  16. Deep shape from polarization. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIV 16, pages 554–571. Springer, 2020.
  17. Shape from polarization for complex scenes in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12632–12641, 2022.
  18. Depth sensing using geometrically constrained polarization normals. International Journal of Computer Vision, 125:34–51, 2017.
  19. High-sensitivity analysis of polarization by surface reflection. Machine Vision and Applications, 29:1171–1189, 2018.
  20. Lawrence B Wolff. Polarization vision: a new sensory approach to image understanding. Image and Vision computing, 15(2):81–93, 1997.
  21. Lucid vision phoenix polarization camera. https://thinklucid.com/product/phoenix-5-0-mp-polarized-model/. 2018.
  22. Event-based shape from polarization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1547–1556, 2023.
  23. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 2020.
  24. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607–617, 2019.
  25. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3. Citeseer, 2013.
  26. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853, 2015.
  27. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.
  28. Wulfram Gerstner. Time structure of the activity in neural network models. Physical review E, 51(1):738, 1995.
  29. Larry F Abbott. Lapicque’s introduction of the integrate-and-fire model neuron (1907). Brain research bulletin, 50(5-6):303–304, 1999.
  30. Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press, 2002.
  31. Deep learning. nature, 521(7553):436–444, 2015.
  32. Spikformer: When spiking neural network meets transformer. arXiv preprint arXiv:2209.15425, 2022.
  33. Spiking transformers for event-based single object tracking. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 8801–8810, 2022.
  34. Spiking graph convolutional networks. arXiv preprint arXiv:2205.02767, 2022.
  35. Spikegpt: Generative pre-trained language model with spiking neural networks. arXiv preprint arXiv:2302.13939, 2023.
  36. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  37. William A Shurcliff. Polarized light: production and use. Harvard University Press, 1962.
  38. Edward Collett. Field guide to polarization. Spie Bellingham, WA, 2005.
  39. Recent progress in shape from polarization. Advances in Photometric 3D-Reconstruction, pages 177–203, 2020.
  40. Polarization-based transparent surface modeling from two views. In Proceedings Ninth IEEE International Conference on Computer Vision, pages 1381–1386. IEEE, 2003.
  41. Surface normal estimation of black specular objects from multiview polarization images. Optical Engineering, 56(4):041303–041303, 2017.
  42. Polarimetric dense monocular slam. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3857–3866, 2018.
  43. Depth from a polarisation+ rgb stereo pair. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7586–7595, 2019.
  44. Shape from polarization: a method for solving zenithal angle ambiguity. Optics letters, 37(20):4218–4220, 2012.
  45. Improved diffuse reflection models for computer vision. International Journal of Computer Vision, 30:55–71, 1998.
  46. Shape-from-polarisation: a nonlinear least squares approach. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 2969–2976, 2017.
  47. Height-from-polarisation with unknown lighting or albedo. IEEE transactions on pattern analysis and machine intelligence, 41(12):2875–2888, 2018.
  48. N Justin Marshall. A unique colour and polarization vision system in mantis shrimps. Nature, 333(6173):557–560, 1988.
  49. Pdavis: Bio-inspired polarization event camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3962–3971, 2023.
  50. Unsupervised event-based learning of optical flow, depth, and egomotion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 989–997, 2019.
  51. Ed Bullmore and Olaf Sporns. The economy of brain network organization. Nature Reviews Neuroscience, 13(5):336–349, 2012.
  52. Distributed hierarchical processing in the primate cerebral cortex. In Cereb cortex. Citeseer, 1991.
  53. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science, 345(6197):668–673, 2014.
  54. Advancing neuromorphic computing with loihi: A survey of results and outlook. Proceedings of the IEEE, 109(5):911–934, 2021.
  55. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1):54–66, 2015.
  56. Going deeper in spiking neural networks: Vgg and residual architectures. Frontiers in neuroscience, 13:95, 2019.
  57. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
  58. Lisnn: Improving spiking neural networks with lateral interactions for robust object recognition. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 1519–1525. International Joint Conferences on Artificial Intelligence Organization, 7 2020. Main track.
  59. Stereospike: Depth learning with a spiking neural network. IEEE Access, 10:127428–127439, 2022.
  60. Mss-depthnet: Depth prediction with multi-step spiking neural network. arXiv preprint arXiv:2211.12156, 2022.
  61. Spiking nerf: Making bio-inspired neural networks see through the real world. arXiv preprint arXiv:2309.10987, 2023.
  62. High speed and high dynamic range video with an event camera. IEEE transactions on pattern analysis and machine intelligence, 43(6):1964–1980, 2019.
  63. Time lens++: Event-based frame interpolation with parametric non-linear flow and multi-scale fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17755–17764, 2022.
  64. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2661–2671, 2021.
  65. Deconvolution and checkerboard artifacts. Distill, 1(10):e3, 2016.
  66. Integrating non-spiking interneurons in spiking neural networks. Frontiers in neuroscience, 15:633945, 2021.
  67. Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Transactions on Neural Networks and Learning Systems, 33(11):6249–6262, 2021.
  68. Robert Hecht-Nielsen. Theory of the backpropagation neural network. In Neural networks for perception, pages 65–93. Elsevier, 1992.
  69. Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550–1560, 1990.
  70. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51–63, 2019.
  71. Spikingjelly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances, 9(40):eadi1480, 2023.
  72. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  73. Training deep spiking neural networks. arXiv preprint arXiv:2006.04436, 2020.
  74. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  75. Mitsuba 3 renderer, 2022. https://mitsuba-renderer.org.
  76. Esim: an open event camera simulator. In Conference on robot learning, pages 969–982. PMLR, 2018.
  77. A 1280x720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86 um pixels, 1.066 geps readout, programmable event-rate controller and compressive data-formatting pipeline. In IEEE International Solid-State Circuits Conference, 2020.
  78. Breakthrough photography x4 polarizer. https://breakthrough.photography/products/x4-circular-polarizer.
  79. Esl: Event-based structured light. In 2021 International Conference on 3D Vision (3DV), pages 1165–1174. IEEE, 2021.
  80. Direct method for shape recovery from polarization and shading. In 2012 19th IEEE International Conference on Image Processing, pages 1769–1772. IEEE, 2012.
  81. Exploiting spiking dynamics with spatial-temporal feature normalization in graph learning. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3207–3213. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Main Track.
  82. Spike-flownet: event-based optical flow estimation with energy-efficient hybrid neural networks. In European Conference on Computer Vision, pages 366–382. Springer, 2020.
  83. Mark Horowitz. 1.1 computing’s energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10–14. IEEE, 2014.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now