Papers
Topics
Authors
Recent
Search
2000 character limit reached

NeRF as a Non-Distant Environment Emitter in Physics-based Inverse Rendering

Published 7 Feb 2024 in cs.CV and cs.GR | (2402.04829v2)

Abstract: Physics-based inverse rendering enables joint optimization of shape, material, and lighting based on captured 2D images. To ensure accurate reconstruction, using a light model that closely resembles the captured environment is essential. Although the widely adopted distant environmental lighting model is adequate in many cases, we demonstrate that its inability to capture spatially varying illumination can lead to inaccurate reconstructions in many real-world inverse rendering scenarios. To address this limitation, we incorporate NeRF as a non-distant environment emitter into the inverse rendering pipeline. Additionally, we introduce an emitter importance sampling technique for NeRF to reduce the rendering variance. Through comparisons on both real and synthetic datasets, our results demonstrate that our NeRF-based emitter offers a more precise representation of scene lighting, thereby improving the accuracy of inverse rendering.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Differentiable Rendering of Neural SDFs through Reparameterization. In SIGGRAPH Asia 2022 Conference Papers, SA 2022, Daegu, Republic of Korea, December 6-9, 2022, Soon Ki Jung, Jehee Lee, and Adam W. Bargteil (Eds.). ACM, 5:1–5:9. https://doi.org/10.1145/3550469.3555397
  2. Unbiased warped-area sampling for differentiable rendering. ACM Trans. Graph. 39, 6 (2020), 245:1–245:18. https://doi.org/10.1145/3414685.3417833
  3. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. IEEE, 5460–5469. https://doi.org/10.1109/CVPR52688.2022.00539
  4. Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. IEEE, 19640–19648. https://doi.org/10.1109/ICCV51070.2023.01804
  5. Neural Reflectance Fields for Appearance Acquisition. CoRR abs/2008.03824 (2020). arXiv:2008.03824 https://arxiv.org/abs/2008.03824
  6. Deep Reflectance Volumes: Relightable Reconstructions from Multi-view Photometric Images. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III (Lecture Notes in Computer Science, Vol. 12348), Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer, 294–311. https://doi.org/10.1007/978-3-030-58580-8_18
  7. NeRD: Neural Reflectance Decomposition from Image Collections. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. IEEE, 12664–12674. https://doi.org/10.1109/ICCV48922.2021.01245
  8. Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 10691–10704. https://proceedings.neurips.cc/paper/2021/hash/58ae749f25eded36f486bc85feb3f0ab-Abstract.html
  9. Parameter-space ReSTIR for Differentiable and Inverse Rendering. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH 2023, Los Angeles, CA, USA, August 6-10, 2023, Erik Brunvand, Alla Sheffer, and Michael Wimmer (Eds.). ACM, 18:1–18:10. https://doi.org/10.1145/3588432.3591512
  10. Zhaoxi Chen and Ziwei Liu. 2022. Relighting4D: Neural Relightable Human from Videos. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV (Lecture Notes in Computer Science, Vol. 13674), Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer, 606–623. https://doi.org/10.1007/978-3-031-19781-9_35
  11. Jorge Condor and Adrián Jarabo. 2022. A Learned Radiance-Field Representation for Complex Luminaires. In 33rd Eurographics Symposium on Rendering, EGSR 2022 - Symposium Track, Prague, Czech Republic, 4-6 July 2022, Abhijeet Ghosh and Li-Yi Wei (Eds.). Eurographics Association, 49–58. https://doi.org/10.2312/SR.20221155
  12. Paul E. Debevec and Jitendra Malik. 1997. Recovering high dynamic range radiance maps from photographs. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1997, Los Angeles, CA, USA, August 3-8, 1997, G. Scott Owen, Turner Whitted, and Barbara Mones-Hattal (Eds.). ACM, 369–378. https://doi.org/10.1145/258734.258884
  13. Param Hanji and Rafal K. Mantiuk. 2023. Robust Estimation of Exposure Ratios in Multi-Exposure Image Stacks. IEEE Trans. Computational Imaging 9 (2023), 721–731. https://doi.org/10.1109/TCI.2023.3301338
  14. Noise-Aware Merging of High Dynamic Range Image Stacks Without Camera Calibration. In Computer Vision - ECCV 2020 Workshops - Glasgow, UK, August 23-28, 2020, Proceedings, Part III (Lecture Notes in Computer Science, Vol. 12537), Adrien Bartoli and Andrea Fusiello (Eds.). Springer, 376–391. https://doi.org/10.1007/978-3-030-67070-2_23
  15. Wenzel Jakob. 2012. Numerically stable sampling of the von Mises-Fisher distribution on Sˆ2 (and other tricks). Interactive Geometry Lab, ETH Zürich, Tech. Rep (2012), 6.
  16. DR.JIT: a just-in-time compiler for differentiable rendering. ACM Trans. Graph. 41, 4 (2022), 124:1–124:19. https://doi.org/10.1145/3528223.3530099
  17. TensoIR: Tensorial Inverse Rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. IEEE, 165–174. https://doi.org/10.1109/CVPR52729.2023.00024
  18. MIS compensation: optimizing sampling techniques in multiple importance sampling. ACM Trans. Graph. 38, 6 (2019), 151:1–151:12. https://doi.org/10.1145/3355089.3356565
  19. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1412.6980
  20. Differentiable Monte Carlo ray tracing through edge sampling. ACM Trans. Graph. 37, 6 (2018), 222. https://doi.org/10.1145/3272127.3275109
  21. BARF: Bundle-Adjusting Neural Radiance Fields. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. IEEE, 5721–5731. https://doi.org/10.1109/ICCV48922.2021.00569
  22. ShadowNeuS: Neural SDF Reconstruction by Shadow Ray Supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023. IEEE, 175–185. https://doi.org/10.1109/CVPR52729.2023.00025
  23. NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. IEEE, 16169–16178. https://doi.org/10.1109/CVPR52688.2022.01571
  24. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 12346), Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer, 405–421. https://doi.org/10.1007/978-3-030-58452-8_24
  25. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. 41, 4 (2022), 102:1–102:15. https://doi.org/10.1145/3528223.3530127
  26. Recursive Control Variates for Inverse Rendering. ACM Trans. Graph. 42, 4 (2023), 62:1–62:13. https://doi.org/10.1145/3592139
  27. Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering. ACM Trans. Graph. 39, 4 (2020), 146. https://doi.org/10.1145/3386569.3392406
  28. Julien Philip and Valentin Deschaintre. 2023. Radiance Field Gradient Scaling for Unbiased Near-Camera Training. CoRR abs/2305.02756 (2023). https://doi.org/10.48550/ARXIV.2305.02756 arXiv:2305.02756
  29. Dynamic Mesh-Aware Radiance Fields. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. IEEE, 385–396. https://doi.org/10.1109/ICCV51070.2023.00042
  30. Neural Radiance Fields for Outdoor Scene Relighting. CoRR abs/2112.05140 (2021). arXiv:2112.05140 https://arxiv.org/abs/2112.05140
  31. NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. Computer Vision Foundation / IEEE, 7495–7504. https://doi.org/10.1109/CVPR46437.2021.00741
  32. Neural-PBIR Reconstruction of Shape, Material, and Illumination. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. IEEE, 18000–18010. https://doi.org/10.1109/ICCV51070.2023.01654
  33. Nerfstudio: A Modular Framework for Neural Radiance Field Development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH 2023, Los Angeles, CA, USA, August 6-10, 2023, Erik Brunvand, Alla Sheffer, and Michael Wimmer (Eds.). ACM, 72:1–72:12. https://doi.org/10.1145/3588432.3591516
  34. Differentiable signed distance function rendering. ACM Trans. Graph. 41, 4 (2022), 125:1–125:18. https://doi.org/10.1145/3528223.3530139
  35. Yu. M. Volin and G. M. Ostrovskii. 1985. Automatic computation of derivatives with the use of the multilevel differentiating technique—1. Algorithmic basis. Computers & Mathematics With Applications 11 (1985), 1099–1114. https://api.semanticscholar.org/CorpusID:120003155
  36. Amortizing Samples in Physics-Based Inverse Rendering Using ReSTIR. ACM Trans. Graph. 42, 6 (2023), 214:1–214:17. https://doi.org/10.1145/3618331
  37. Adaptive Shells for Efficient Neural Radiance Field Rendering. ACM Trans. Graph. 42, 6 (2023), 260:1–260:15. https://doi.org/10.1145/3618390
  38. VR-NeRF: High-Fidelity Virtualized Walkable Spaces. In SIGGRAPH Asia 2023 Conference Papers, SA 2023, Sydney, NSW, Australia, December 12-15, 2023, June Kim, Ming C. Lin, and Bernd Bickel (Eds.). ACM, 43:1–43:12. https://doi.org/10.1145/3610548.3618139
  39. Warped-Area Reparameterization of Differential Path Integrals. ACM Trans. Graph. 42, 6 (2023), 213:1–213:18. https://doi.org/10.1145/3618330
  40. PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 13661), Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (Eds.). Springer, 266–284. https://doi.org/10.1007/978-3-031-19769-7_16
  41. S33{}^{\mbox{3}}start_FLOATSUPERSCRIPT 3 end_FLOATSUPERSCRIPT-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). http://papers.nips.cc/paper_files/paper/2022/hash/0a630402ee92620dc2de3b704181de9b-Abstract-Conference.html
  42. Monte Carlo estimators for differential light transport. ACM Trans. Graph. 40, 4 (2021), 78:1–78:16. https://doi.org/10.1145/3450626.3459807
  43. Path-space differentiable rendering. ACM Trans. Graph. 39, 4 (2020), 143. https://doi.org/10.1145/3386569.3392383
  44. A differential theory of radiative transfer. ACM Trans. Graph. 38, 6 (2019), 227:1–227:16. https://doi.org/10.1145/3355089.3356522
  45. IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. IEEE, 5555–5564. https://doi.org/10.1109/CVPR52688.2022.00548
  46. PhySG: Inverse Rendering With Spherical Gaussians for Physics-Based Material Editing and Relighting. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. Computer Vision Foundation / IEEE, 5453–5462. https://doi.org/10.1109/CVPR46437.2021.00541
  47. Modeling Indirect Illumination for Inverse Rendering. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. IEEE, 18622–18631. https://doi.org/10.1109/CVPR52688.2022.01809
  48. Neural complex luminaires: representation and rendering. ACM Trans. Graph. 40, 4 (2021), 57:1–57:12. https://doi.org/10.1145/3450626.3459798
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

GitHub

  1. NeRF Emitter 

Tweets

Sign up for free to view the 3 tweets with 1 like about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free