Neural radiance fields-based holography [Invited]

Abstract: This study presents a novel approach for generating holograms based on the neural radiance fields (NeRF) technique. Generating three-dimensional (3D) data is difficult in hologram computation. NeRF is a state-of-the-art technique for 3D light-field reconstruction from 2D images based on volume rendering. The NeRF can rapidly predict new-view images that do not include a training dataset. In this study, we constructed a rendering pipeline directly from a 3D light field generated from 2D images by NeRF for hologram generation using deep neural networks within a reasonable time. The pipeline comprises three main components: the NeRF, a depth predictor, and a hologram generator, all constructed using deep neural networks. The pipeline does not include any physical calculations. The predicted holograms of a 3D scene viewed from any direction were computed using the proposed pipeline. The simulation and experimental results are presented.

• The paper introduces a three-stage pipeline that leverages NeRF for 3D light-field reconstruction, depth prediction, and hologram generation.
• It validates the approach with simulations and real-world experiments using park scene datasets and powerful hardware like AMD Ryzen 5 and Nvidia RTX 2080 SUPER.
• The study highlights challenges in NeRF optimization and depth range, paving the way for future advancements to streamline holographic display technologies.

Innovations in Hologram Generation with Neural Radiance Fields-based Holography

Introduction to Neural Radiance Fields-based Holography

Recently, a study focused on neural radiance fields (NeRF) as a basis for hologram generation. This novel methodology leverages the NeRF technology, a cutting-edge approach for creating 3D light-field reconstructions from 2D images, and does not rely on traditional 3D cameras or graphics techniques. The core of this innovation is a pipeline that directly translates a 3D light field generated from 2D images into holograms using deep neural networks. This process circumvents the need for intricate physical calculations typically associated with holography, offering a more streamlined and time-efficient approach.

The Proposed Pipeline

The paper outlines a three-stage pipeline for creating holograms:

1. NeRF for 3D Light Field Reconstruction: Initially, NeRF processes 2D images to predict new-view images, crafting a 3D light field without requiring the images to belong to the initial training dataset.
2. Depth Prediction: Subsequently, a depth predictor (specifically, MiDaS) analyzes the synthesized images to estimate depth maps.
3. Hologram Generation: Ultimately, the "Tensor Holography" method employs both the RGB images and depth maps to generate the final hologram.

This approach is noted for its ability to rapidly generate new views and subsequently predict holograms within a reasonable timeframe, representing a significant step forward in holographic technology.

Simulation and Experimental Results

The study conducted simulations and real-world experiments to validate the effectiveness of the proposed pipeline. Utilizing datasets like park scene photographs, the authors demonstrated that the pipeline could generate holograms with accurate 3D representations. The research utilized an AMD Ryzen 5 4500 CPU and an Nvidia GeForce RTX 2080 SUPER GPU, emphasizing the computational efficiency of the process.

Challenges and Future Directions

Despite its innovative approach, the study acknowledges several limitations. The initial optimization of NeRF, while producing high-quality images, remains time-consuming and demands a considerable number of input images. Additionally, the depth range of the produced holograms is somewhat constrained, hinting at the necessity for further advancements in hologram predictors and their capabilities to render deeper 3D scenes.

The future of this research is headed towards simplifying the pipeline by potentially eliminating the need for a separate depth prediction network, thus accelerating the hologram prediction process. Recent advancements in NeRF and alternative deep learning strategies for hologram generation suggest promising avenues for overcoming the current limitations.

Conclusion

The exploration of NeRF-based holography presents a notable advancement in hologram generation technology. By merging NeRF with deep learning approaches for depth prediction and hologram generation, the study introduces a method that could significantly reduce the complexities and computational demands traditionally associated with producing holograms. Despite facing challenges such as optimization times and depth representation nuances, the suggested pipeline offers substantial groundwork for future improvements in holographic display technologies.

Funding: This research was supported by the Japan Society for the Promotion of Science and the IAAR Research Support Program at Chiba University, Japan, highlighting the academic and potential commercial interest in developing advanced holographic display technologies.