Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Time-Efficient Light-Field Acquisition Using Coded Aperture and Events (2403.07244v1)

Published 12 Mar 2024 in cs.CV and eess.IV

Abstract: We propose a computational imaging method for time-efficient light-field acquisition that combines a coded aperture with an event-based camera. Different from the conventional coded-aperture imaging method, our method applies a sequence of coding patterns during a single exposure for an image frame. The parallax information, which is related to the differences in coding patterns, is recorded as events. The image frame and events, all of which are measured in a single exposure, are jointly used to computationally reconstruct a light field. We also designed an algorithm pipeline for our method that is end-to-end trainable on the basis of deep optics and compatible with real camera hardware. We experimentally showed that our method can achieve more accurate reconstruction than several other imaging methods with a single exposure. We also developed a hardware prototype with the potential to complete the measurement on the camera within 22 msec and demonstrated that light fields from real 3-D scenes can be obtained with convincing visual quality. Our software and supplementary video are available from our project website.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Single lens stereo with a plenoptic camera. IEEE transactions on pattern analysis and machine intelligence, 14(2):99–106, 1992.
  2. Gradient-index lens-array method based on real-time integral photography for three-dimensional images. Applied optics, 37(11):2034–2045, 1998.
  3. A 240 × 180 130 dB 3 μ⁢s𝜇𝑠\mu sitalic_μ italic_s latency global shutter spatiotemporal vision sensor. IEEE Journal of Solid-State Circuits, 49:2333–2341, 2014.
  4. Immersive light field video with a layered mesh representation. In ACM Transactions on Graphics (Proc. SIGGRAPH), 2020.
  5. Pi-GAN: Periodic implicit generative adversarial networks for 3D-Aware image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  6. Generative novel view synthesis with 3D-aware diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4217–4229, 2023.
  7. LFGAN: 4D light field synthesis from a single RGB image. ACM Trans. Multimedia Comput. Commun. Appl., 16(1), 2020.
  8. Multipoint measuring system for video and sound - 100-camera and microphone system. In IEEE International Conference on Multimedia and Expo, pages 437–440, 2006.
  9. Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):154–180, 2022.
  10. Deep spatial-angular regularization for light field imaging, denoising, and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6094–6110, 2022.
  11. A dataset and evaluation methodology for depth estimation on 4D light fields. In Asian Conference on Computer Vision, 2016.
  12. The light field stereoscope: immersive computer graphics via factored near-eye light field displays with focus cues. ACM Transactions on Graphics, 34(4):60, 2015.
  13. Deepbinarymask: Learning a binary mask for video compressive sensing, 2016.
  14. Learning to capture light fields through a coded aperture camera. In European Conference on Computer Vision, pages 418–434, 2018.
  15. Learning-based view synthesis for light field cameras. ACM Transactions on Graphics, 35(6), 2016.
  16. Edge-aware bidirectional diffusion for dense depth estimation from light fields. In British Machine Vision Conference (BMVC), 2021.
  17. Real-time 3D reconstruction and 6-DoF tracking with an event camera. In European Conference on Computer Vision (ECCV), pages 349–364, 2016.
  18. Additive light field displays: realization of augmented reality with holographic optical elements. ACM Transactions on Graphics, 35(4):1–13, 2016.
  19. MINE: Towards continuous depth MPI with NeRF for novel view synthesis. In International Conference on Computer Vision, 2021.
  20. Synthesizing light field from a single image with variable MPI and two network fusion. ACM Transactions on Graphics, 2020.
  21. End-to-end video compressive sensing using anderson-accelerated unrolled networks. In International Conference on Computational Photography, pages 137–148, 2020.
  22. Programmable aperture photography: multiplexed light field acquisition. ACM Transactions on Graphics, 27(3):1–10, 2008.
  23. Zhengyu Liang. BasicLFSR (open source light field toolbox for super-resolution). https://github.com/ZhengyuLiang24/BasicLFSR, 2021.
  24. Deformable neural radiance fields using RGB and event cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3590–3600, 2023.
  25. Light field distortion feature for transparent object recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2786–2793, 2013.
  26. Compressive light field photography using overcomplete dictionaries and optimized projections. ACM Transactions on Graphics, 32(4):1–12, 2013.
  27. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics, 38:1–14, 2019.
  28. Acquiring a dynamic light field through a single-shot coded image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  29. Fast and accurate reconstruction of compressed color light field. In International Conference on Computational Photography, pages 1–11, 2018.
  30. Programmable aperture camera using LCoS. In European Conference on Computer Vision, pages 337–350, 2010.
  31. Ren Ng. Digital light field photography. PhD thesis, Stanford University, 2006.
  32. Light field photography with a hand-held plenoptic camera. Computer Science Technical Report CSTR, 2(11):1–11, 2005.
  33. Deeply learned filter response functions for hyperspectral reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4767–4776, 2018.
  34. 3D ken burns effect from a single image. ACM Transactions on Graphics, 38(6):184:1–184:15, 2019.
  35. EMVS: Event-based multi-view stereo – 3D reconstruction with an event camera in real-time. International Journal of Computer Vision, 126(12):1394–1414, 2018a.
  36. ESIM: an open event camera simulator. Conf. on Robotics Learning (CoRL), 2018b.
  37. EventNeRF: Neural radiance fields from a single colour event camera. In Computer Vision and Pattern Recognition (CVPR), 2023.
  38. Acquiring dynamic light fields through coded aperture camera. In European Conference on Computer Vision, pages 368–385, 2020.
  39. EPINET: A fully-convolutional neural network using epipolar geometry for depth from light field images. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4748–4757, 2018.
  40. TransCAIP: A live 3D TV system using a camera array and an integral photography display with interactive control of viewing parameters. IEEE Transactions on Visualization and Computer Graphics, 15(5):841–852, 2009.
  41. Factor modulation for single-shot light-field acquisition. In IEEE International Conference on Image Processing, pages 3253–3257, 2021.
  42. Time-multiplexed coded aperture and coded focal stack -comparative study on snapshot compressive light field imaging. IEICE Transactions on Information and Systems, E105.D(10):1679–1690, 2022.
  43. GRF: Learning a general radiance field for 3D representation and rendering. In International Conference on Computer Vision, 2021.
  44. Single-view view synthesis with multiplane images. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  45. A unified learning based framework for light field reconstruction from coded projections. IEEE Transactions on Computational Imaging, 6:304–316, 2019.
  46. Time-multiplexed coded aperture imaging: Learned coded aperture and pixel exposures for compressive imaging systems. In International Conference on Computer Vision, 2021.
  47. Dappled photography: Mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Transactions on Graphics, 26(3):69, 2007.
  48. Deep optics for video snapshot compressive imaging. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10646–10656, 2023.
  49. A 4D light-field dataset and CNN architectures for material recognition. In European Conference on Computer Vision, 2016.
  50. Tensor displays: Compressive light field synthesis using multilayer displays with directional backlighting. ACM Transactions on Graphics, 31(4):1–11, 2012.
  51. High performance imaging using large camera arrays. ACM Transactions on Graphics, 24(3):765–776, 2005.
  52. Phasecam3D ― learning phase masks for passive single view depth estimation. In International Conference on Computational Photography, pages 1–12, 2019.
  53. SinNeRF: Training neural radiance fields on complex scenes from a single image. In European Conference on Computer Vision, 2022.
  54. Joint optimization for compressive video sensing and reconstruction under hardware constraints. In European Conference on Computer Vision, 2018.
  55. pixelNeRF: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  56. Semi-dense 3D reconstruction with a stereo event camera. In European Conference on Computer Vision, pages 242–258, 2018.
  57. Event-based stereo visual odometry. IEEE Transactions on Robotics, 37(5):1433–1450, 2021.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now