Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Neural radiance fields-based holography [Invited] (2403.01137v2)

Published 2 Mar 2024 in cs.CV, cs.GR, and eess.IV

Abstract: This study presents a novel approach for generating holograms based on the neural radiance fields (NeRF) technique. Generating three-dimensional (3D) data is difficult in hologram computation. NeRF is a state-of-the-art technique for 3D light-field reconstruction from 2D images based on volume rendering. The NeRF can rapidly predict new-view images that do not include a training dataset. In this study, we constructed a rendering pipeline directly from a 3D light field generated from 2D images by NeRF for hologram generation using deep neural networks within a reasonable time. The pipeline comprises three main components: the NeRF, a depth predictor, and a hologram generator, all constructed using deep neural networks. The pipeline does not include any physical calculations. The predicted holograms of a 3D scene viewed from any direction were computed using the proposed pipeline. The simulation and experimental results are presented.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. P. S. Hilaire, S. A. Benton, and M. Lucente, “Synthetic aperture holography: a novel approach to three-dimensional displays,” \JournalTitleJOSA A 9, 1969–1977 (1992).
  2. D. Blinder, A. Ahar, S. Bettens, et al., “Signal processing challenges for digital holographic video display systems,” \JournalTitleSignal Processing: Image Communication 70, 114–130 (2019).
  3. Y. Takaki and K. Fujii, “Viewing-zone scanning holographic display using a mems spatial light modulator,” \JournalTitleOptics Express 22, 24713–24721 (2014).
  4. H. Sasaki, K. Yamamoto, K. Wakunami, et al., “Large size three-dimensional video by electronic holography using multiple spatial light modulators,” \JournalTitleScientific reports 4, 6177 (2014).
  5. J. Li, Q. Smithwick, and D. Chu, “Full bandwidth dynamic coarse integral holographic displays with large field of view using a large resonant scanner and a galvanometer scanner,” \JournalTitleOptics Express 26, 17459–17476 (2018).
  6. J. Park, K. Lee, and Y. Park, “Ultrathin wide-angle large-area digital 3d holographic display using a non-periodic photon sieve,” \JournalTitleNature communications 10, 1304 (2019).
  7. S. Tay, P.-A. Blanche, R. Voorakaranam, et al., “An updatable holographic three-dimensional display,” \JournalTitleNature 451, 694–698 (2008).
  8. P.-A. Blanche, A. Bablumian, R. Voorakaranam, et al., “Holographic three-dimensional telepresence using large-area photorefractive polymer,” \JournalTitleNature 468, 80–83 (2010).
  9. H. Takagi, K. Nakamura, T. Goto, et al., “Magneto-optic spatial light modulator with submicron-size magnetic pixels for wide-viewing-angle holographic displays,” \JournalTitleOptics Letters 39, 3344–3347 (2014).
  10. M. Makowski, J. Bomba, A. Frej, et al., “Dynamic complex opto-magnetic holography,” \JournalTitleNature Communications 13, 7286 (2022).
  11. T. Yamaguchi, G. Okabe, and H. Yoshikawa, “Real-time image plane full-color and full-parallax holographic video display system,” \JournalTitleOptical Engineering 46, 125801–125801 (2007).
  12. K. Matsushima and S. Nakahara, “Extremely high-definition full-parallax computer-generated hologram created by the polygon-based method,” \JournalTitleApplied optics 48, H54–H63 (2009).
  13. F. Wang, T. Ito, and T. Shimobaba, “High-speed rendering pipeline for polygon-based holograms,” \JournalTitlePhotonics Research 11, 313–328 (2023).
  14. K. Wakunami and M. Yamaguchi, “Calculation for computer generated hologram using ray-sampling plane,” \JournalTitleOptics Express 19, 9086–9101 (2011).
  15. Y. Zhao, L. Cao, H. Zhang, et al., “Accurate calculation of computer-generated holograms using angular-spectrum layer-oriented method,” \JournalTitleOptics express 23, 25440–25449 (2015).
  16. M. Yamaguchi, “Light-field and holographic three-dimensional displays,” \JournalTitleJOSA A 33, 2348–2364 (2016).
  17. R. Horisaki, R. Takagi, and J. Tanida, “Deep-learning-generated holography,” \JournalTitleApplied optics 57, 3859–3863 (2018).
  18. H. Goi, K. Komuro, and T. Nomura, “Deep-learning-based binary hologram,” \JournalTitleApplied Optics 59, 7103–7108 (2020).
  19. M. H. Eybposh, N. W. Caira, M. Atisa, et al., “Deepcgh: 3d computer-generated holography using deep learning,” \JournalTitleOptics Express 28, 26636–26650 (2020).
  20. L. Shi, B. Li, C. Kim, et al., “Towards real-time photorealistic 3d holography with deep neural networks,” \JournalTitleNature 591, 234–239 (2021).
  21. S. Choi, M. Gopakumar, Y. Peng, et al., “Neural 3d holography: learning accurate wave propagation models for 3d holographic virtual and augmented reality displays,” \JournalTitleACM Transactions on Graphics (TOG) 40, 1–12 (2021).
  22. L. Shi, B. Li, and W. Matusik, “End-to-end learning of 3d phase-only holograms for holographic display,” \JournalTitleLight: Science & Applications 11, 247 (2022).
  23. C. Chang, B. Dai, D. Zhu, et al., “From picture to 3d hologram: end-to-end learning of real-time 3d photorealistic hologram generation from 2d image input,” \JournalTitleOptics Letters 48, 851–854 (2023).
  24. Y. Ishii, F. Wang, H. Shiomi, et al., “Multi-depth hologram generation from two-dimensional images by deep learning,” \JournalTitleOptics and Lasers in Engineering 170, 107758 (2023).
  25. B. Mildenhall, P. P. Srinivasan, M. Tancik, et al., “Nerf: Representing scenes as neural radiance fields for view synthesis,” \JournalTitleCommunications of the ACM 65, 99–106 (2021).
  26. R. Ranftl, K. Lasinger, D. Hafner, et al., “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” \JournalTitleIEEE transactions on pattern analysis and machine intelligence 44, 1623–1637 (2020).
  27. S. Chan, H.-Y. Shum, and K.-T. Ng, “Image-based rendering and synthesis,” \JournalTitleIEEE Signal Processing Magazine 24, 22–33 (2007).
  28. O. Özyeşil, V. Voroninski, R. Basri, and A. Singer, “A survey of structure from motion*.” \JournalTitleActa Numerica 26, 305–364 (2017).
  29. A. Vaswani, N. Shazeer, N. Parmar, et al., “Attention is all you need,” \JournalTitleAdvances in neural information processing systems 30 (2017).
  30. J. L. Schönberger and J.-M. Frahm, “Structure-from-motion revisited,” in Conference on Computer Vision and Pattern Recognition (CVPR), (2016).
  31. C.-K. Hsueh and A. A. Sawchuk, “Computer-generated double-phase holograms,” \JournalTitleApplied optics 17, 3874–3883 (1978).
  32. X. Sui, Z. He, G. Jin, et al., “Band-limited double-phase method for enhancing image sharpness in complex modulated computer-generated holograms,” \JournalTitleOptics Express 29, 2597–2612 (2021).
  33. K. Matsushima and T. Shimobaba, “Band-limited angular spectrum method for numerical simulation of free-space propagation in far and near fields,” \JournalTitleOptics express 17, 19662–19673 (2009).
  34. S. M. Pizer, E. P. Amburn, J. D. Austin, et al., “Adaptive histogram equalization and its variations,” \JournalTitleComputer vision, graphics, and image processing 39, 355–368 (1987).
  35. T. Yamaguchi, T. Fujii, and H. Yoshikawa, “Fast calculation method for computer-generated cylindrical holograms,” \JournalTitleApplied Optics 47, D63–D70 (2008).
  36. M. L. Tachiki, Y. Sando, M. Itoh, and T. Yatagai, “Fast calculation method for spherical computer-generated holograms,” \JournalTitleApplied optics 45, 3527–3533 (2006).
  37. A. Yu, V. Ye, M. Tancik, and A. Kanazawa, “pixelnerf: Neural radiance fields from one or few images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2021), pp. 4578–4587.
  38. R. Zhu, L. Chen, and H. Zhang, “Computer holography using deep neural network with fourier basis,” \JournalTitleOptics Letters 48, 2333–2336 (2023).

Summary

  • The paper introduces a three-stage pipeline that leverages NeRF for 3D light-field reconstruction, depth prediction, and hologram generation.
  • It validates the approach with simulations and real-world experiments using park scene datasets and powerful hardware like AMD Ryzen 5 and Nvidia RTX 2080 SUPER.
  • The study highlights challenges in NeRF optimization and depth range, paving the way for future advancements to streamline holographic display technologies.

Innovations in Hologram Generation with Neural Radiance Fields-based Holography

Introduction to Neural Radiance Fields-based Holography

Recently, a paper focused on neural radiance fields (NeRF) as a basis for hologram generation. This novel methodology leverages the NeRF technology, a cutting-edge approach for creating 3D light-field reconstructions from 2D images, and does not rely on traditional 3D cameras or graphics techniques. The core of this innovation is a pipeline that directly translates a 3D light field generated from 2D images into holograms using deep neural networks. This process circumvents the need for intricate physical calculations typically associated with holography, offering a more streamlined and time-efficient approach.

The Proposed Pipeline

The paper outlines a three-stage pipeline for creating holograms:

  1. NeRF for 3D Light Field Reconstruction: Initially, NeRF processes 2D images to predict new-view images, crafting a 3D light field without requiring the images to belong to the initial training dataset.
  2. Depth Prediction: Subsequently, a depth predictor (specifically, MiDaS) analyzes the synthesized images to estimate depth maps.
  3. Hologram Generation: Ultimately, the "Tensor Holography" method employs both the RGB images and depth maps to generate the final hologram.

This approach is noted for its ability to rapidly generate new views and subsequently predict holograms within a reasonable timeframe, representing a significant step forward in holographic technology.

Simulation and Experimental Results

The paper conducted simulations and real-world experiments to validate the effectiveness of the proposed pipeline. Utilizing datasets like park scene photographs, the authors demonstrated that the pipeline could generate holograms with accurate 3D representations. The research utilized an AMD Ryzen 5 4500 CPU and an Nvidia GeForce RTX 2080 SUPER GPU, emphasizing the computational efficiency of the process.

Challenges and Future Directions

Despite its innovative approach, the paper acknowledges several limitations. The initial optimization of NeRF, while producing high-quality images, remains time-consuming and demands a considerable number of input images. Additionally, the depth range of the produced holograms is somewhat constrained, hinting at the necessity for further advancements in hologram predictors and their capabilities to render deeper 3D scenes.

The future of this research is headed towards simplifying the pipeline by potentially eliminating the need for a separate depth prediction network, thus accelerating the hologram prediction process. Recent advancements in NeRF and alternative deep learning strategies for hologram generation suggest promising avenues for overcoming the current limitations.

Conclusion

The exploration of NeRF-based holography presents a notable advancement in hologram generation technology. By merging NeRF with deep learning approaches for depth prediction and hologram generation, the paper introduces a method that could significantly reduce the complexities and computational demands traditionally associated with producing holograms. Despite facing challenges such as optimization times and depth representation nuances, the suggested pipeline offers substantial groundwork for future improvements in holographic display technologies.

Funding: This research was supported by the Japan Society for the Promotion of Science and the IAAR Research Support Program at Chiba University, Japan, highlighting the academic and potential commercial interest in developing advanced holographic display technologies.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets