RRM: Relightable assets using Radiance guided Material extraction (2407.06397v1)

Abstract: Synthesizing NeRFs under arbitrary lighting has become a seminal problem in the last few years. Recent efforts tackle the problem via the extraction of physically-based parameters that can then be rendered under arbitrary lighting, but they are limited in the range of scenes they can handle, usually mishandling glossy scenes. We propose RRM, a method that can extract the materials, geometry, and environment lighting of a scene even in the presence of highly reflective objects. Our method consists of a physically-aware radiance field representation that informs physically-based parameters, and an expressive environment light structure based on a Laplacian Pyramid. We demonstrate that our contributions outperform the state-of-the-art on parameter retrieval tasks, leading to high-fidelity relighting and novel view synthesis on surfacic scenes.

• The paper introduces an end-to-end model that extracts physically-based material properties to generate relightable neural radiance fields from 2D images.
• It employs a physically-aware radiance module and a Laplacian pyramid for environment maps to robustly capture high-frequency specular effects in glossy scenes.
• Experiments show improved normal estimation (MAE ~2.26°) and superior relighting performance, highlighting its potential for VR and advanced graphics applications.

RRM: Relightable Assets using Radiance Guided Material Extraction

Diego Gomez et al. introduce "RRM: Relightable assets using Radiance guided Material extraction," a method that has made significant strides in the generation of relightable NeRFs (Neural Radiance Fields) from a collection of 2D images. This paper demonstrates substantial improvements in handling glossy scenes, a limitation observed in previous approaches. The authors present an end-to-end optimizable model able to recover materials, geometry, and environment lighting even in complex scenarios involving reflective objects.

The crux of the proposed method lies in a two-pronged architecture combining a physically-aware radiance field with an expressive environment lighting structure based on a Laplacian Pyramid. This hybrid approach enables retrieving physically-based parameters, optimizing relighting fidelity, and accomplishing novel view synthesis.

Key Contributions

1. Physically-Aware Radiance Module:

The authors integrate a physically-aware radiance module to extract coarse scene geometry, including surface normals and a sense of roughness. This module segregates the predicted radiance into view-dependent and view-independent components, which becomes foundational in handling high-specularity scenarios.

2. Laplacian Pyramid for Environment Maps:

A novel representation of environment maps using a Laplacian Pyramid coupled with Multiple Importance Sampling (MIS) allows the model to capture high-frequency specular effects. This structure provides robust performance in scenes with complex reflective properties, thus outperforming traditional methods like Spherical Gaussians (SG) in terms of detail extraction and convergence speed.

3. Radiance Field Guided Learning:

The method utilizes a radiance field not only for rendering but also as a supervisory signal to inform the learning process of physically-based parameters. This innovative strategy facilitates the breakdown of incoming light into diffuse and glossy components, leading to higher-quality parameter retrieval.

Numerical Results and Comparisons

The paper's methodology significantly enhances normal estimation, reflecting in a marginal Mean Angular Error (MAE) of approximately 2.26°, vastly superior to the 4.39° exhibited by TensoIR. It also achieves commendable performance on the novel view synthesis task, albeit slightly less than NMF (PSNR of 31.64 vs. 33.60).

In the context of relighting, the proposed method consistently outperforms both state-of-the-art approaches, NMF and TensoIR, particularly excelling in scenes dominated by glossy materials. For instance, the method achieves a PSNR of 25.84 on the Shiny Blender dataset, surpassing NMF's 25.50 and demonstrating superior SSIM and LPIPS scores.

Practical and Theoretical Implications

Practically, the RRM method is poised to benefit applications requiring high-fidelity 3D reconstructions from 2D images, such as virtual reality content creation, computer graphics, and immersive media. The end-to-end trainable pipeline ensures scalability and adaptability, making it suitable for various real-world scenarios involving complex materials and lighting environments.

Theoretically, this paper pushes the boundaries of NeRF applications by efficiently integrating volumetric and physically-based rendering. It underscores the necessity of multi-faceted approaches in addressing intrinsic ambiguities within inverse rendering problems, specifically the disentanglement of complex light interactions within a scene.

Future Directions

Possible augmentations to this work could involve enhancing environment representations to include local light sources, rather than restricting to far-field approximations. Further exploration in accommodating semi-transmissive and subsurface scattering effects would broaden the scope of its applicability. Additionally, leveraging optimized NeRF libraries and integrating mesh-based representations could refine computational efficiency and broaden deployment contexts.

In summary, Gomez et al. present a pioneering approach to relightable NeRFs that adeptly bridges physically-based rendering with volumetric techniques, setting a new standard in material extraction from radiance fields. This paper's method not only contributes to novel theoretical insights but also offers practical solutions with promising real-world implications.