Neural Light Transport for Relighting and View Synthesis (2008.03806v3)

Abstract: The light transport (LT) of a scene describes how it appears under different lighting and viewing directions, and complete knowledge of a scene's LT enables the synthesis of novel views under arbitrary lighting. In this paper, we focus on image-based LT acquisition, primarily for human bodies within a light stage setup. We propose a semi-parametric approach to learn a neural representation of LT that is embedded in the space of a texture atlas of known geometric properties, and model all non-diffuse and global LT as residuals added to a physically-accurate diffuse base rendering. In particular, we show how to fuse previously seen observations of illuminants and views to synthesize a new image of the same scene under a desired lighting condition from a chosen viewpoint. This strategy allows the network to learn complex material effects (such as subsurface scattering) and global illumination, while guaranteeing the physical correctness of the diffuse LT (such as hard shadows). With this learned LT, one can relight the scene photorealistically with a directional light or an HDRI map, synthesize novel views with view-dependent effects, or do both simultaneously, all in a unified framework using a set of sparse, previously seen observations. Qualitative and quantitative experiments demonstrate that our neural LT (NLT) outperforms state-of-the-art solutions for relighting and view synthesis, without separate treatment for both problems that prior work requires.

• The paper proposes Neural Light Transport (NLT), a semi-parametric method using a neural network embedded in a texture atlas to efficiently capture and render light transport for relighting and novel view synthesis.
• The methodology integrates a neural network with traditional graphics techniques, including a residuals model for complex lighting effects and a two-branch framework processing observations for novel queries.
• Quantitatively, NLT surpasses existing methods in SSIM, PSNR, and LPIPS metrics, achieving higher rendering fidelity that captures fine material details, with applications in areas like virtual telepresence and digital actors.

Neural Light Transport for Relighting and View Synthesis

The paper "Neural Light Transport for Relighting and View Synthesis" presented by Zhang et al. proposes a novel method for efficiently capturing and rendering the light transport (LT) characteristics of human bodies, focusing particularly on reconstructing scenes with a light stage setup. The authors introduce a semi-parametric approach that constructs a neural representation of light transport embedded within a texture atlas. This approach facilitates relighting under arbitrary lighting conditions and synthesizes novel views concurrently, using previously captured observations fused effectively.

Methodology and Framework

In addressing the challenges of capturing LT, the paper critiques existing methods for their sparse sampling limitations and lack of adaptability. The proposed Neural Light Transport (NLT) method combines a neural network with traditional computer graphics techniques. Specifically, the approach integrates a residuals model to account for non-diffuse and global light effects atop a diffuse base rendering. The methodology leverages the texture atlas space to train a neural network that interpolates a 6D LT function consistently across different lighting and viewing angles.

The framework employs two primary branches: observation paths and a query path. The observation paths process multi-view, one-light-at-a-time (OLAT) images to extract relevant features, which are then pooled and fed into the query path. The query path synthesizes new images based on desired lighting and view directions, greatly extending the versatility of traditional static-based rendering.

Practical and Theoretical Implications

Quantitatively, the paper demonstrates that NLT outperforms existing methodologies in terms of SSIM, PSNR, and LPIPS metrics, particularly excelling in capturing fine material effects like specular highlights and subsurface scattering. The visual quality and rendering fidelity achieved reflect the potential of leveraging neural networks for relighting applications, enabling photorealistic virtual renderings to be synthesized from sparse input data.

Theoretically, the paper posits that combining physically-based approaches with neural networks serves to mitigate the inadequacies of both extremes—empirical models often fail to capture complex reflectance phenomena, while purely data-driven approaches lack the constraints and extrapolative ability of physical models. The integration within the texture atlas space further aligns with classical computer graphics by maintaining compatibility with standard rendering pipelines when projecting from texture to camera space.

Future Developments and Considerations

Moving forward, the paper suggests several areas for enhancement, including dynamic scene rendering and improving resolution for high-frequency detail reproduction. Moreover, extending the framework to accommodate generalization across different scene geometries or dynamic poses could increase its applicability in diverse real-world scenarios. The method aligns with advancing machine learning tasks reliant on high-quality image datasets, such as virtual telepresence and digital actors, by providing a robust mechanism to synthetically generate relit subjects.

In conclusion, the research presented by Zhang et al. substantially contributes to computer graphics through its innovative integration of neural networks within a traditional rendering context, providing new opportunities for photorealistic rendering and view synthesis capabilities.