DeepShaRM: Multi-View Shape and Reflectance Map Recovery Under Unknown Lighting (2310.17632v1)
Abstract: Geometry reconstruction of textureless, non-Lambertian objects under unknown natural illumination (i.e., in the wild) remains challenging as correspondences cannot be established and the reflectance cannot be expressed in simple analytical forms. We derive a novel multi-view method, DeepShaRM, that achieves state-of-the-art accuracy on this challenging task. Unlike past methods that formulate this as inverse-rendering, i.e., estimation of reflectance, illumination, and geometry from images, our key idea is to realize that reflectance and illumination need not be disentangled and instead estimated as a compound reflectance map. We introduce a novel deep reflectance map estimation network that recovers the camera-view reflectance maps from the surface normals of the current geometry estimate and the input multi-view images. The network also explicitly estimates per-pixel confidence scores to handle global light transport effects. A deep shape-from-shading network then updates the geometry estimate expressed with a signed distance function using the recovered reflectance maps. By alternating between these two, and, most important, by bypassing the ill-posed problem of reflectance and illumination decomposition, the method accurately recovers object geometry in these challenging settings. Extensive experiments on both synthetic and real-world data clearly demonstrate its state-of-the-art accuracy.
- Deep 3D Capture: Geometry and Reflectance From Sparse Multi-View Images. In Proc. CVPR, pages 5959–5968, 2020.
- Invertible Neural BRDF for Object Inverse Rendering. TPAMI, 44(12):9380–9395, 2022.
- Multi-View 3D Reconstruction of a Texture-Less Smooth Surface of Unknown Generic Reflectance. In Proc. CVPR, pages 16226–16235, 2021.
- Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning. TPAMI, 40(8):1932–1947, 2018.
- Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. In Proc. NeurIPS, 2022.
- Calculating the reflectance map. Applied optics, 18(11):1770–1779, 1979.
- James T. Kajiya. The rendering equation. In Proc. SIGGRAPH, pages 143–150. ACM, 1986.
- Modular primitives for high-performance differentiable rendering. ACM TOG, 39(6):194:1–194:14, 2020.
- NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images. ACM TOG, 2023.
- Reflectance and Illumination Recovery in the Wild. TPAMI, 38(1):129–141, 2016.
- SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views. In Proc. ECCV, pages 210–227, 2022.
- Marching cubes: A high resolution 3d surface construction algorithm. In Proc. SIGGRAPH, page 163–169, New York, NY, USA, 1987. Association for Computing Machinery.
- A Level Set Theory for Neural Implicit Evolution Under Explicit Flows. In Proc. ECCV, pages 711–729. Springer, 2022.
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Proc. ECCV, 2020.
- Practical SVBRDF Acquisition of 3D Objects with Unstructured Flash Photography. ACM TOG, 37(6):267, 2018.
- UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction. In Proc. ICCV, pages 5569–5579. IEEE, 2021.
- Shape and Reflectance Estimation in the Wild. TPAMI, 38(2):376–389, 2015.
- A signal-processing framework for reflection. ACM TOG, 23(4):1004–1042, 2004.
- VolRecon: Volume Rendering of Signed Ray Distance Functions for Generalizable Multi-View Reconstruction. In Proc. CVPR, pages 16685–16695, 2023.
- Differentiable Signed Distance Function Rendering. ACM TOG, 41(4):125:1–125:18, 2022.
- NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In Proc. NeurIPS, pages 27171–27183, 2021.
- nLMVS-Net: Deep Non-Lambertian Multi-View Stereo. In Proc. WACV, pages 3037–3046, 2023.
- Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. In Proc. NeurIPS, pages 2492–2502, 2020.
- Volume Rendering of Neural Implicit Surfaces. In Proc. NeurIPS, pages 4805–4815, 2021.
- PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting. In Proc. CVPR, pages 5453–5462, 2021.
- IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images. In Proc. CVPR, pages 5555–5564. IEEE, 2022.