Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

NEMTO: Neural Environment Matting for Novel View and Relighting Synthesis of Transparent Objects (2303.11963v2)

Published 21 Mar 2023 in cs.CV and cs.GR

Abstract: We propose NEMTO, the first end-to-end neural rendering pipeline to model 3D transparent objects with complex geometry and unknown indices of refraction. Commonly used appearance modeling such as the Disney BSDF model cannot accurately address this challenging problem due to the complex light paths bending through refractions and the strong dependency of surface appearance on illumination. With 2D images of the transparent object as input, our method is capable of high-quality novel view and relighting synthesis. We leverage implicit Signed Distance Functions (SDF) to model the object geometry and propose a refraction-aware ray bending network to model the effects of light refraction within the object. Our ray bending network is more tolerant to geometric inaccuracies than traditional physically-based methods for rendering transparent objects. We provide extensive evaluations on both synthetic and real-world datasets to demonstrate our high-quality synthesis and the applicability of our method.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  2. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
  3. Eikonal fields for refractive novel-view synthesis. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings, pages 1–9, 2022.
  4. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824, 2020.
  5. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVF Conference on International Conference on Computer Vision (ICCV), 2021.
  6. Neural-pil: Neural pre-integrated lighting for reflectance decomposition. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  7. Brent Burley and Walt Disney Animation Studios. Physically-based shading at disney. In Transactions on Graphics (Proceedings of SIGGRAPH), volume 2012, pages 1–7. vol. 2012, 2012.
  8. Tom-net: Learning transparent object matting from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9233–9241, 2018.
  9. Environment matting extensions: Towards higher accuracy and real-time capture. In Proceedings of the 27th annual conference on Computer Graphics and Interactive Techniques, pages 121–130, 2000.
  10. Polarimetric multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1558–1567, 2017.
  11. Christian Diller. Chamfer distance for pytorch, 2022. https://github.com/otaheri/chamfer_distance.
  12. Kaleido AI GmbH. Remove image background., 2022. https://www.remove.bg/. Accessed 2022-11.
  13. Implicit geometric regularization for learning shapes. In Proceedings of Machine Learning and Systems (ICML), pages 3569–3579, 2020.
  14. John C Hart. Sphere tracing: A geometric method for the antialiased ray tracing of implicit surfaces. The Visual Computer, 12(10):527–545, 1996.
  15. Shape and refractive index recovery from single-view polarisation images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1229–1236. IEEE, 2010.
  16. Shape and refractive index recovery from single-view polarisation images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1229–1236, 2010.
  17. Eikonal rendering: Efficient light transport in refractive objects. Transactions on Graphics (Proceedings of SIGGRAPH), 26(3):59–es, 2007.
  18. Mitsuba 3 renderer, 2022. https://mitsuba-renderer.org.
  19. James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Computer Graphics and Interactive Techniques, pages 143–150, 1986.
  20. Real shading in unreal engine 4. Proc. Physically Based Shading Theory Practice, 4(3):1, 2013.
  21. Acquiring axially-symmetric transparent objects using single-view transmission imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3559–3567, 2017.
  22. A theory of refractive and specular 3d shape by light-path triangulation. International Journal of Computer Vision, 76(1):13–29, 2008.
  23. Through the looking glass: Neural 3d reconstruction of transparent shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1262–1271, 2020.
  24. Differentiable refraction-tracing for mesh reconstruction of transparent objects. Transactions on Graphics (Proceedings of SIGGRAPH), 39(6):1–13, 2020.
  25. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210–7219, 2021.
  26. Nerf: Representing scenes as neural radiance fields for view synthesis. In Computer Vision–ECCV 2020: 15th European Conference, 23-28 August 2020, Proceeding, 2020.
  27. Inverse polarization raytracing: estimating surface shapes of transparent objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pages 910–917 vol. 2, 2005.
  28. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1–15, 2022.
  29. Extracting triangular 3d models, materials, and lighting from images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8280–8290, 2022.
  30. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 165–174, 2019.
  31. Physically based rendering: From theory to implementation. Morgan Kaufmann, 2016.
  32. Scene reassembly after multimodal digitization and pipeline evaluation using photorealistic rendering. Applied Optics, 56(27):7679–7690, 2017.
  33. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
  34. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings. ACM New York, NY, USA, 2023.
  35. Eric Veach. Robust Monte Carlo methods for light transport simulation. Stanford University, 1998.
  36. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  37. Refractive shape from light field distortion. In Proceedings of the IEEE/CVF Conference on International Conference on Computer Vision (ICCV), pages 1180–1186. IEEE, 2011.
  38. Image-based environment matting. In Proceedings, Eurographics Workshop on Rendering, pages 289–299, 2002.
  39. Surface light fields for 3d photography. In Proceedings of the 27th annual conference on Computer Graphics and Interactive Techniques, pages 287–296, 2000.
  40. Full 3d reconstruction of transparent objects. Transactions on Graphics (Proceedings of SIGGRAPH), 37(4), jul 2018.
  41. Differentiable rendering using rgbxy derivatives and optimal transport. Transactions on Graphics (Proceedings of SIGGRAPH), 41(6), dec 2022.
  42. A hybrid mesh-neural representation for 3d transparent object reconstruction. Computational Visual Media Journal (CVMJ), 2023.
  43. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems (NeurIPS), 34:4805–4815, 2021.
  44. Multiview neural surface reconstruction with implicit lighting and material. Advances in Neural Information Processing Systems (NeurIPS), 3, 2020.
  45. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5453–5462, 2021.
  46. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  47. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. Transactions on Graphics (Proceedings of SIGGRAPH), 40(6):1–18, 2021.
  48. Modeling indirect illumination for inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  49. Environment matting and compositing. In Proceedings of the 26th annual conference on Computer Graphics and Interactive Techniques, pages 205–214, 1999.
Citations (11)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now