Single-Image SVBRDF Capture with a Rendering-Aware Deep Network (1810.09718v1)

Abstract: Texture, highlights, and shading are some of many visual cues that allow humans to perceive material appearance in single pictures. Yet, recovering spatially-varying bi-directional reflectance distribution functions (SVBRDFs) from a single image based on such cues has challenged researchers in computer graphics for decades. We tackle lightweight appearance capture by training a deep neural network to automatically extract and make sense of these visual cues. Once trained, our network is capable of recovering per-pixel normal, diffuse albedo, specular albedo and specular roughness from a single picture of a flat surface lit by a hand-held flash. We achieve this goal by introducing several innovations on training data acquisition and network design. For training, we leverage a large dataset of artist-created, procedural SVBRDFs which we sample and render under multiple lighting directions. We further amplify the data by material mixing to cover a wide diversity of shading effects, which allows our network to work across many material classes. Motivated by the observation that distant regions of a material sample often offer complementary visual cues, we design a network that combines an encoder-decoder convolutional track for local feature extraction with a fully-connected track for global feature extraction and propagation. Many important material effects are view-dependent, and as such ambiguous when observed in a single image. We tackle this challenge by defining the loss as a differentiable SVBRDF similarity metric that compares the renderings of the predicted maps against renderings of the ground truth from several lighting and viewing directions. Combined together, these novel ingredients bring clear improvement over state of the art methods for single-shot capture of spatially varying BRDFs.

• The paper's main contribution is a rendering-aware deep network that estimates diffuse albedo, specular albedo, roughness, and normals from a single image.
• It leverages a procedural dataset of 200,000 augmented samples and employs a rendering loss to improve material appearance over traditional pixel-wise errors.
• The enhanced encoder-decoder architecture with global feature fusion significantly outperforms existing methods in both synthetic tests and qualitative evaluations.

Single-Image SVBRDF Capture with a Rendering-Aware Deep Network

The paper presents a novel approach to capturing spatially-varying bidirectional reflectance distribution functions (SVBRDFs) from a single image, using a deep neural network specifically designed for this complex task. The authors propose a method that leverages a rendering-aware architecture to effectively tackle the ill-posed inverse problem of estimating SVBRDFs from a single flash-lit photograph. Their approach addresses the longstanding challenge of material capture in computer graphics through several technical innovations in both data acquisition and network design.

The primary contribution of the work is the introduction of a deep network capable of synthesizing diffuse albedo, specular albedo, specular roughness, and normal maps from a single image of a flat surface. The authors employ a procedural dataset comprised of artist-created SVBRDFs and utilize physically-based rendering to produce a diverse set of training images. By augmenting this dataset through parameter perturbations and mixing, the training set is amplified to approximately 200,000 realistic samples.

A key innovation is the incorporation of a rendering loss function, comparing the appearances of predicted and ground truth SVBRDFs under various lighting and viewing configurations. This approach avoids the pitfalls of pixel-wise error minimization, which often fail to account for the interactions between reflectance parameters. Instead, the rendering loss aligns the predicted maps with the perceptual appearance of the actual materials.

The authors further enhance the network’s performance by enriching the traditional encoder-decoder (U-Net) architecture with a global features network. This secondary track is designed to amalgamate local and global information, overcoming challenges related to distant feature interactions. The network thereby facilitates the transmission of critical information across the image, addressing shortfalls observed in architectures restricted to local feature extraction.

Quantitatively, the proposed method outperforms existing techniques, as demonstrated in comparisons with algorithms like those by Li et al. [2017] and Aiţala et al. [2016]. On synthetic data, the proposed network yields lower RMSE in diffuse albedo, normal, and rendered appearance evaluations. Qualitatively, it more accurately reproduces specular and roughness attributes, particularly in challenging scenarios.

The implications of this research are multi-faceted. Practically, the method promises improvements in digital content creation, enabling artists and developers to quickly capture and reproduce material properties from single images without extensive, costly measurement equipment. Theoretically, this work advances the understanding of how deep networks can integrate and exploit complex datasets, influencing future research on material capture and image-based rendering.

Future developments might explore extending the approach to handle materials with anisotropic properties or those involving complex subsurface scattering, which fall outside the bounds of the current framework. Additionally, the rendering-aware concept could be expanded to other domains within computer vision, where inverse problems prevail.

Overall, this research marks a substantial step in the ongoing refinement of SVBRDF capture techniques, offering a viable solution to single-image material reconstruction with implications for both practical applications and theoretical explorations in computer graphics and machine learning.