NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images (2305.17398v1)

Abstract: We present a neural rendering-based method called NeRO for reconstructing the geometry and the BRDF of reflective objects from multiview images captured in an unknown environment. Multiview reconstruction of reflective objects is extremely challenging because specular reflections are view-dependent and thus violate the multiview consistency, which is the cornerstone for most multiview reconstruction methods. Recent neural rendering techniques can model the interaction between environment lights and the object surfaces to fit the view-dependent reflections, thus making it possible to reconstruct reflective objects from multiview images. However, accurately modeling environment lights in the neural rendering is intractable, especially when the geometry is unknown. Most existing neural rendering methods, which can model environment lights, only consider direct lights and rely on object masks to reconstruct objects with weak specular reflections. Therefore, these methods fail to reconstruct reflective objects, especially when the object mask is not available and the object is illuminated by indirect lights. We propose a two-step approach to tackle this problem. First, by applying the split-sum approximation and the integrated directional encoding to approximate the shading effects of both direct and indirect lights, we are able to accurately reconstruct the geometry of reflective objects without any object masks. Then, with the object geometry fixed, we use more accurate sampling to recover the environment lights and the BRDF of the object. Extensive experiments demonstrate that our method is capable of accurately reconstructing the geometry and the BRDF of reflective objects from only posed RGB images without knowing the environment lights and the object masks. Codes and datasets are available at https://github.com/liuyuan-pal/NeRO.

• The paper presents NeRO, a novel method that reconstructs geometry and BRDF of reflective objects using a two-stage neural rendering approach.
• It employs neural SDF with split-sum approximation and integrated directional encoding to capture accurate geometry and shading without object masks.
• BRDF is refined using Monte Carlo sampling and dedicated MLPs, yielding superior relighting accuracy and outperforming established methods on key metrics.

Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images

The paper presents "NeRO," a novel method for reconstructing the geometry and BRDF of reflective objects using multiview images. Reflective object reconstruction is particularly challenging due to the nature of specular reflections, which violate multiview consistency — a critical assumption for conventional reconstruction methods.

Methodology Overview

NeRO employs a neural rendering approach, leveraging recent advancements in this area to infer reflective properties of objects under unknown lighting conditions. The paper identifies limitations in existing methods, primarily their reliance on direct lighting models and object masks, and proposes a two-step approach to address these issues.

1. Light Representation and Two-Stage Approach:
   • Stage I: Focuses on geometry reconstruction. It employs split-sum approximation combined with integrated directional encoding to approximate shading from both direct and indirect lights in the absence of object masks. This stage uses a neural SDF representation to accurately capture geometry.
   • Stage II: Dedicated to BRDF refinement. With geometry fixed from Stage I, Monte Carlo sampling is utilized to enhance the accuracy of environment light and BRDF estimation.
2. Light and Occlusion Modeling:
   • Direct and indirect lights are modeled using two separate MLPs. Occlusion probabilities, predicted by a dedicated MLP, determine whether direct or indirect lights are used, enhancing the accuracy of occluded surface regions.
3. Key Innovations:
   • The use of a bounding sphere for defining light categories.
   • A novel light representation that accounts for spatial variation and occlusion.
   • A two-stage pipeline refining geometry and BRDF sequentially.

Results and Comparisons

The extensive experiments conducted demonstrate that NeRO substantially outperforms existing methods like NeuS, COLMAP, and others, particularly on challenging datasets with strong reflective surfaces. Notable results include:

• Accurate reconstruction without reliance on object masks.
• Improved BRDF estimation leading to photorealistic relighting capabilities.
• Superior performance metrics such as Chamfer Distance and PSNR in both synthetic and real environments.

Implications and Future Work

The implications of this research are twofold:

• Theoretical:
  • It underscores the potential of split-sum approximations and integrated encodings in handling complex light-object interactions.
  • It opens pathways to further integrate neural rendering techniques in classical multiview reconstruction problems.
• Practical:
  • The method's ability to handle unknown light conditions and lack of object masks broadens its applicability, particularly in fields requiring precision modeling from limited data inputs.

Future Work:

The paper suggests further exploration into enhancing the robustness of the two-stage method and extending applications to more diverse environments. These improvements could lead to more adaptive algorithms capable of generalizing across varying reflective characteristics and complex scenes.

In conclusion, this paper contributes significantly to the field of neural rendering and 3D reconstruction, particularly in handling reflective surfaces—a persistent challenge in computer vision. NeRO's methodological innovations and promising results position it as a foundational tool for future investigations and applications in geometry and material reconstruction.