Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DeepShaRM: Multi-View Shape and Reflectance Map Recovery Under Unknown Lighting (2310.17632v1)

Published 26 Oct 2023 in cs.CV

Abstract: Geometry reconstruction of textureless, non-Lambertian objects under unknown natural illumination (i.e., in the wild) remains challenging as correspondences cannot be established and the reflectance cannot be expressed in simple analytical forms. We derive a novel multi-view method, DeepShaRM, that achieves state-of-the-art accuracy on this challenging task. Unlike past methods that formulate this as inverse-rendering, i.e., estimation of reflectance, illumination, and geometry from images, our key idea is to realize that reflectance and illumination need not be disentangled and instead estimated as a compound reflectance map. We introduce a novel deep reflectance map estimation network that recovers the camera-view reflectance maps from the surface normals of the current geometry estimate and the input multi-view images. The network also explicitly estimates per-pixel confidence scores to handle global light transport effects. A deep shape-from-shading network then updates the geometry estimate expressed with a signed distance function using the recovered reflectance maps. By alternating between these two, and, most important, by bypassing the ill-posed problem of reflectance and illumination decomposition, the method accurately recovers object geometry in these challenging settings. Extensive experiments on both synthetic and real-world data clearly demonstrate its state-of-the-art accuracy.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (26)
  1. Deep 3D Capture: Geometry and Reflectance From Sparse Multi-View Images. In Proc. CVPR, pages 5959–5968, 2020.
  2. Invertible Neural BRDF for Object Inverse Rendering. TPAMI, 44(12):9380–9395, 2022.
  3. Multi-View 3D Reconstruction of a Texture-Less Smooth Surface of Unknown Generic Reflectance. In Proc. CVPR, pages 16226–16235, 2021.
  4. Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning. TPAMI, 40(8):1932–1947, 2018.
  5. Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising. In Proc. NeurIPS, 2022.
  6. Calculating the reflectance map. Applied optics, 18(11):1770–1779, 1979.
  7. James T. Kajiya. The rendering equation. In Proc. SIGGRAPH, pages 143–150. ACM, 1986.
  8. Modular primitives for high-performance differentiable rendering. ACM TOG, 39(6):194:1–194:14, 2020.
  9. NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images. ACM TOG, 2023.
  10. Reflectance and Illumination Recovery in the Wild. TPAMI, 38(1):129–141, 2016.
  11. SparseNeuS: Fast Generalizable Neural Surface Reconstruction from Sparse Views. In Proc. ECCV, pages 210–227, 2022.
  12. Marching cubes: A high resolution 3d surface construction algorithm. In Proc. SIGGRAPH, page 163–169, New York, NY, USA, 1987. Association for Computing Machinery.
  13. A Level Set Theory for Neural Implicit Evolution Under Explicit Flows. In Proc. ECCV, pages 711–729. Springer, 2022.
  14. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Proc. ECCV, 2020.
  15. Practical SVBRDF Acquisition of 3D Objects with Unstructured Flash Photography. ACM TOG, 37(6):267, 2018.
  16. UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction. In Proc. ICCV, pages 5569–5579. IEEE, 2021.
  17. Shape and Reflectance Estimation in the Wild. TPAMI, 38(2):376–389, 2015.
  18. A signal-processing framework for reflection. ACM TOG, 23(4):1004–1042, 2004.
  19. VolRecon: Volume Rendering of Signed Ray Distance Functions for Generalizable Multi-View Reconstruction. In Proc. CVPR, pages 16685–16695, 2023.
  20. Differentiable Signed Distance Function Rendering. ACM TOG, 41(4):125:1–125:18, 2022.
  21. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In Proc. NeurIPS, pages 27171–27183, 2021.
  22. nLMVS-Net: Deep Non-Lambertian Multi-View Stereo. In Proc. WACV, pages 3037–3046, 2023.
  23. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. In Proc. NeurIPS, pages 2492–2502, 2020.
  24. Volume Rendering of Neural Implicit Surfaces. In Proc. NeurIPS, pages 4805–4815, 2021.
  25. PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting. In Proc. CVPR, pages 5453–5462, 2021.
  26. IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images. In Proc. CVPR, pages 5555–5564. IEEE, 2022.
Citations (1)

Summary

We haven't generated a summary for this paper yet.