Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Objects With Lighting: A Real-World Dataset for Evaluating Reconstruction and Rendering for Object Relighting (2401.09126v2)

Published 17 Jan 2024 in cs.CV and cs.GR

Abstract: Reconstructing an object from photos and placing it virtually in a new environment goes beyond the standard novel view synthesis task as the appearance of the object has to not only adapt to the novel viewpoint but also to the new lighting conditions and yet evaluations of inverse rendering methods rely on novel view synthesis data or simplistic synthetic datasets for quantitative analysis. This work presents a real-world dataset for measuring the reconstruction and rendering of objects for relighting. To this end, we capture the environment lighting and ground truth images of the same objects in multiple environments allowing to reconstruct the objects from images taken in one environment and quantify the quality of the rendered views for the unseen lighting environments. Further, we introduce a simple baseline composed of off-the-shelf methods and test several state-of-the-art methods on the relighting task and show that novel view synthesis is not a reliable proxy to measure performance. Code and dataset are available at https://github.com/isl-org/objects-with-lighting .

Definition Search Book Streamline Icon: https://streamlinehq.com
References (80)
  1. Reflectance modeling by neural texture synthesis. In ACM Transactions on Graphics (ToG), 2018.
  2. Deep svbrdf estimation on real materials. In 2020 International Conference on 3D Vision (3DV), pages 1157–1166. IEEE, 2020.
  3. Shape, illumination, and reflectance from shading. IEEE transactions on pattern analysis and machine intelligence, 37(8):1670–1687, 2015.
  4. Recovering intrinsic scene characteristics. Comput. Vis. Syst, 2(3-26):2, 1978.
  5. Neural Reflectance Fields for Appearance Acquisition. arXiv:2008.03824 [cs], 2020a.
  6. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In European Conference on Computer Vision, pages 294–311. Springer, 2020b.
  7. Deep 3D Capture: Geometry and Reflectance From Sparse Multi-View Images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020c.
  8. James F Blinn. Models of light reflection for computer synthesized pictures. In Proceedings of the 4th annual conference on Computer graphics and interactive techniques, pages 192–198, 1977.
  9. Deep Dual Loss BRDF Parameter Estimation. In Workshop on Material Appearance Modeling, 2018.
  10. NeRD: Neural Reflectance Decomposition from Image Collections. In IEEE International Conference on Computer Vision (ICCV), 2021a.
  11. Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition. In Advances in Neural Information Processing Systems (NeurIPS), 2021b.
  12. Brian. freepbr, 2019. https://freepbr.com.
  13. Brent Burley and Walt Disney Animation Studios. Physically-based shading at disney. In ACM SIGGRAPH, pages 1–7. vol. 2012, 2012.
  14. Geosim: Realistic video simulation via geometry-aware composition for self-driving. In CVPR, 2021.
  15. Invertible Neural BRDF for Object Inverse Rendering. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V, pages 767–783. Springer, 2020.
  16. Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
  17. A reflectance model for computer graphics. ACM Transactions on Graphics (ToG), 1982.
  18. Single-image SVBRDF capture with a rendering-aware deep network. In ACM Transactions on Graphics (ToG), 2018.
  19. Guided fine-tuning for large-scale material transfer. In Eurographics Symposium on Rendering, 2020.
  20. Appearance-from-motion: Recovering spatially varying surface reflectance under unknown lighting. ACM Transactions on Graphics (TOG), 33(6):1–12, 2014.
  21. An adaptive parameterization for efficient material acquisition and rendering. ACM Transactions on graphics (TOG), 37(6):1–14, 2018.
  22. Plenoxels: Radiance fields without neural networks. In CVPR, pages 5491–5500. IEEE, 2022.
  23. Datasheets for datasets, 2021.
  24. Yoshiharu Gotanda. Beyond a simple physically based blinn-phong model in real-time. In SIGGRAPH Physically Based Shading in Theory and Practice course, 2012.
  25. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In IEEE 12th International Conference on Computer Vision, ICCV 2009, Kyoto, Japan, September 27 - October 4, 2009, pages 2335–2342. IEEE Computer Society, 2009.
  26. The relightables: Volumetric performance capture of humans with realistic relighting. ACM Transactions on Graphics (ToG), 38(6):1–19, 2019.
  27. Shape, light, and material decomposition from images using monte carlo rendering and denoising. In NeurIPS, 2022.
  28. Generative modelling of BRDF textures from flash images. ACM Transactions on Graphics (SIGGRAPH ASIA), 2021.
  29. DeepBRDF: A Deep Representation for Manipulating Measured BRDF. Computer Graphics Forum, 39(2), 2020.
  30. Mitsuba 3 renderer, 2022. https://mitsuba-renderer.org.
  31. Large scale multi-view stereopsis evaluation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 406–413. IEEE, 2014.
  32. Tensoir: Tensorial inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  33. James T. Kajiya. The rendering equation. In SIGGRAPH, pages 143–150, 1986.
  34. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. IEEE Trans. Pattern Anal. Mach. Intell., 28(8):1335–1340, 2006.
  35. Brian Karis. Real shading in Unreal Engine 4. In SIGGRAPH Physically Based Shading in Theory and Practice course, 2013.
  36. Rendering synthetic objects into legacy photographs. ACM Transactions on Graphics (TOG), 30(6):1–12, 2011.
  37. Fast arbitrary brdf shading for low-frequency lighting using spherical harmonics. Rendering Techniques, 2(291-296):1, 2002.
  38. Uncalibrated neural inverse rendering for photometric stereo of general surfaces. In IEEE International Conference on Computer Vision (ICCV), 2021.
  39. Neroic: Neural rendering of objects from online image collections. ACM Trans. Graph., 41(4), 2022.
  40. Dimitar Lazarov. Getting more physical in call of duty: Back ops ii. In SIGGRAPH Physically Based Shading in Theory and Practice course, 2013.
  41. Planned sampling of spatially varying BRDFs. In Computer Graphics Forum, 2003.
  42. Multi-view photometric stereo: A robust solution and benchmark dataset for spatially varying isotropic materials. IEEE Trans. Image Process., 29:4159–4173, 2020.
  43. Interiornet: Mega-scale multi-sensor photo-realistic indoor scenes dataset. In British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018, page 77. BMVA Press, 2018a.
  44. Materials for masses: SVBRDF acquisition with a single mobile phone image. In European Conference on Computer Vision (ECCV), 2018b.
  45. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. In SIGGRAPH, 2023.
  46. Reflectance and illumination recovery in the wild. IEEE transactions on pattern analysis and machine intelligence, 38(1):129–141, 2015.
  47. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 21(4):163–169, 1987.
  48. A data-driven reflectance model. SIGGRAPH, 22(3):759–769, 2003.
  49. Extracting triangular 3d models, materials, and lighting from images. In CVPR, pages 8270–8280. IEEE, 2022.
  50. A dataset of multi-illumination images in the wild. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 4079–4088. IEEE, 2019.
  51. Practical SVBRDF acquisition of 3d objects with unstructured flash photography. In ACM Transactions on Graphics (SIGGRAPH ASIA), 2018.
  52. Geometrical considerations and nomenclature for reflectance. Technical report, National Bureau of Standards, 1977.
  53. Ko Nishino. Directional statistics brdf model. In 2009 IEEE 12th International Conference on Computer Vision, pages 476–483. IEEE, 2009.
  54. Determining reflectance parameters and illumination distribution from a sparse set of images for view-dependent image synthesis. In Proceedings Eighth IEEE international conference on computer vision. ICCV 2001, pages 599–606. IEEE, 2001.
  55. Bui Tuong Phong. Illumination for computer generated pictures. Communications of the ACM, 18(6):311–317, 1975.
  56. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10912–10922, 2021.
  57. Nerf for outdoor scene relighting. In European Conference on Computer Vision (ECCV), 2022.
  58. Single-shot neural relighting and SVBRDF estimation. In ECCV, 2020.
  59. Object shape and reflectance modeling from observation. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages 379–387, 1997.
  60. On joint estimation of pose, geometry and svbrdf from a handheld scanner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3493–3503, 2020.
  61. Structure-from-motion revisited. In CVPR, pages 4104–4113. IEEE Computer Society, 2016.
  62. A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo. IEEE Trans. Pattern Anal. Mach. Intell., 41(2):271–284, 2019.
  63. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021.
  64. The replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797, 2019.
  65. Neural BRDF Representation and Importance Sampling. Comput. Graph. Forum, 40(6):332–346, 2021.
  66. Relight my nerf: A dataset for novel view synthesis and relighting of real world objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20762–20772, 2023.
  67. scikit-image: Image processing in python. CoRR, abs/1407.6245, 2014.
  68. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. NeurIPS, 2021.
  69. Gregory J Ward. Measuring and modeling anisotropic reflection. In Proceedings of the 19th annual conference on Computer graphics and interactive techniques, pages 265–272, 1992.
  70. Nex: Real-time view synthesis with neural basis expansion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  71. Recovering shape and spatially-varying surface reflectance under unknown illumination. ACM Transactions on Graphics, 35(6):187, 2016.
  72. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. Computer Vision and Pattern Recognition (CVPR), 2020.
  73. Photoscene: Photorealistic material and lighting transfer for indoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18562–18571, 2022.
  74. PlenOctrees for Real-Time Rendering of Neural Radiance Fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5752–5761, 2021.
  75. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 215–224, 1999.
  76. Deep inverse rendering for practical object appearance scan with uncalibrated illumination. In Advances in Computer Graphics, 2020.
  77. PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021a.
  78. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
  79. NeRFactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Trans. Graph., 40(6):237:1–237:18, 2021b.
  80. Modeling indirect illumination for inverse rendering. In CVPR, 2022.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - isl-org/objects-with-lighting: Repository for the Objects With Lighting Dataset (47 stars)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets