Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Accidental Light Probes (2301.05211v3)

Published 12 Jan 2023 in cs.CV and cs.GR

Abstract: Recovering lighting in a scene from a single image is a fundamental problem in computer vision. While a mirror ball light probe can capture omnidirectional lighting, light probes are generally unavailable in everyday images. In this work, we study recovering lighting from accidental light probes (ALPs) -- common, shiny objects like Coke cans, which often accidentally appear in daily scenes. We propose a physically-based approach to model ALPs and estimate lighting from their appearances in single images. The main idea is to model the appearance of ALPs by photogrammetrically principled shading and to invert this process via differentiable rendering to recover incidental illumination. We demonstrate that we can put an ALP into a scene to allow high-fidelity lighting estimation. Our model can also recover lighting for existing images that happen to contain an ALP.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. https://www.einscan.com/handheld-3d-scanner/.
  2. https://remove.bg.
  3. Detailed human shape and pose from images. In CVPR, 2007.
  4. James F Blinn. Models of light reflection for computer synthesized pictures. In Proceedings of the 4th annual conference on Computer graphics and interactive techniques, pages 192–198, 1977.
  5. Brent Burley and Walt Disney Animation Studios. Physically-based shading at disney. In SIGGRAPH, 2012.
  6. From faces to outdoor light probes. In CGF, 2018.
  7. End-to-end object detection with transformers. In ECCV, 2020.
  8. Reflectance and texture of real-world surfaces. ACM Transactions On Graphics (TOG), 18(1):1–34, 1999.
  9. Paul Debevec. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In SIGGRAPH, 1998.
  10. An adaptive parameterization for efficient material acquisition and rendering. ACM Transactions on graphics (TOG), 37(6):1–14, 2018.
  11. The reproduction angular error for evaluating the performance of illuminant estimation algorithms. IEEE transactions on pattern analysis and machine intelligence, 39(7):1482–1488, 2016.
  12. Deep parametric indoor lighting estimation. In ICCV, 2019.
  13. Learning to predict indoor illumination from a single image. arXiv:1704.00090, 2017.
  14. Fast spatially-varying indoor lighting estimation. In CVPR, 2019.
  15. Modeling the interaction of light between diffuse surfaces. ACM SIGGRAPH computer graphics, 18(3):213–222, 1984.
  16. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV, 2009.
  17. Distinguishing shiny from matte. Journal of Vision, 2002.
  18. Shape, light & material decomposition from images using monte carlo rendering and denoising. arXiv:2206.03380, 2022.
  19. Eric Heitz. Sampling the ggx distribution of visible normals. Journal of Computer Graphics Techniques (JCGT), 2018.
  20. A practical model for subsurface light transport. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 511–518, 2001.
  21. James T Kajiya. The rendering equation. In SIGGRAPH, 1986.
  22. Deep sr-itm: Joint learning of super-resolution and inverse tone-mapping for 4k uhd hdr applications. In ICCV, 2019.
  23. Real-time illumination estimation from faces for coherent rendering. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 113–122. IEEE, 2014.
  24. Modular primitives for high-performance differentiable rendering. ACM TOG, 2020.
  25. Deep recursive hdri: Inverse tone mapping using generative adversarial networks. In ECCV, 2018.
  26. Deeplight: Learning illumination for unconstrained mobile mixed reality. In CVPR, 2019.
  27. Learning illumination from diverse portraits. In SIGGRAPH Asia Technical Communications. 2020.
  28. Image-based reconstruction of spatial appearance and geometric detail. ACM Transactions on Graphics (TOG), 22(2):234–257, 2003.
  29. Deepim: Deep iterative matching for 6d pose estimation. In ECCV, 2018.
  30. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2475–2484, 2020.
  31. Image-based bidirectional reflectance distribution function measurement. Applied optics, 39(16):2592–2600, 2000.
  32. Extracting triangular 3d models, materials, and lighting from images. In CVPR, 2022.
  33. Eyes for relighting. ACM TOG, 2004.
  34. Physically-inspired deep light estimation from a homogeneous-material object for mixed reality lighting. IEEE TVCG, 2020.
  35. Seeing the world in a bag of chips. In CVPR, 2020.
  36. Physically based rendering: From theory to implementation. Morgan Kaufmann, 2016.
  37. U2-net: Going deeper with nested u-structure for salient object detection. PR, 2020.
  38. A signal-processing framework for inverse rendering. In SIGGRAPH, 2001.
  39. Instant mixed reality lighting from casual scanning. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 27–36. IEEE, 2016.
  40. Christophe Schlick. An inexpensive brdf model for physically-based rendering. In Computer graphics forum, 1994.
  41. Structure-from-motion revisited. In CVPR, 2016.
  42. Neural inverse rendering of an indoor scene from a single image. In International Conference on Computer Vision (ICCV), 2019.
  43. Neural illumination: Lighting prediction for indoor environments. In CVPR, 2019.
  44. Lighthouse: Predicting lighting volumes for spatially-coherent illumination. In CVPR, 2020.
  45. Segmenter: Transformer for semantic segmentation. In ICCV, 2021.
  46. Single image portrait relighting. ACM TOG, 2019.
  47. Theory for off-specular reflection from roughened surfaces. Josa, 1967.
  48. Microfacet models for refraction through rough surfaces. In Proceedings of the 18th Eurographics conference on Rendering Techniques, 2007.
  49. Stylelight: Hdr panorama generation for lighting estimation and editing. In ECCV, 2022.
  50. Normalized object coordinate space for category-level 6d object pose and size estimation. In CVPR, 2019.
  51. Neural light field estimation for street scenes with differentiable virtual object insertion. In ECCV, 2022.
  52. Learning indoor inverse rendering with 3d spatially-varying lighting. In Proceedings of International Conference on Computer Vision (ICCV), 2021.
  53. Learning to estimate indoor lighting from 3d objects. In 3DV, 2018.
  54. Object-based illumination estimation with rendering-aware neural networks. In ECCV, 2020.
  55. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv:1711.00199, 2017.
  56. Faces as lighting probes via unsupervised deep highlight extraction. In ECCV, 2018.
  57. Inverserendernet: Learning single image inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  58. Sparse needlets for lighting estimation with spherical transport loss. In ICCV, 2021.
  59. Emlight: Lighting estimation via spherical distribution approximation. In AAAI, 2021.
  60. Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In CVPR, 2022.
  61. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. In International Conference on Learning Representations, 2021.
  62. State of the art on 3d reconstruction with rgb-d cameras. In Computer graphics forum, volume 37, pages 625–652. Wiley Online Library, 2018.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube