Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
140 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Material Palette: Extraction of Materials from a Single Image (2311.17060v1)

Published 28 Nov 2023 in cs.CV and cs.GR

Abstract: In this paper, we propose a method to extract physically-based rendering (PBR) materials from a single real-world image. We do so in two steps: first, we map regions of the image to material concepts using a diffusion model, which allows the sampling of texture images resembling each material in the scene. Second, we benefit from a separate network to decompose the generated textures into Spatially Varying BRDFs (SVBRDFs), providing us with materials ready to be used in rendering applications. Our approach builds on existing synthetic material libraries with SVBRDF ground truth, but also exploits a diffusion-generated RGB texture dataset to allow generalization to new samples using unsupervised domain adaptation (UDA). Our contributions are thoroughly evaluated on synthetic and real-world datasets. We further demonstrate the applicability of our method for editing 3D scenes with materials estimated from real photographs. The code and models will be made open-source. Project page: https://astra-vision.github.io/MaterialPalette/

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. AmbientCG. Pbr repository. https://www.ambientcg.com, 2017. Accessed: 2023-04-01.
  2. Deep svbrdf estimation on real materials. In 3DV, 2020.
  3. Intrinsic scene properties from a single rgb-d image. In CVPR, 2013.
  4. Agile depth sensing using triangulation light curtains. In ICCV, 2019.
  5. OpenSurfaces: A richly annotated catalog of surface appearance. In ACM TOG, 2013.
  6. Learning texture manifolds with the periodic spatial gan. In ICML, 2017.
  7. Wearable imagenet: Synthesizing tileable textures via dataset distillation. In CVPR-W, 2022.
  8. CGBookCase. Pbr repository. https://www.cgbookcase.com, 2019. Accessed: 2023-04-01.
  9. Single-image svbrdf capture with a rendering-aware deep network. ACM TOG, 2018.
  10. Guided fine‐tuning for large‐scale material transfer. Comput. Graph. Forum, 2020.
  11. Deep polarization imaging for 3d shape and svbrdf acquisition. In CVPR, 2021.
  12. Synthesis of complex image appearance from limited exemplars. ACM TOG, 2015.
  13. Image quilting for texture synthesis and transfer. In ACM TOG, 2001.
  14. Texture synthesis by non-parametric sampling. In ICCV, 1999.
  15. Deep inverse rendering for high-resolution svbrdf estimation from an arbitrary number of images. ACM TOG, 2019.
  16. Fast spatially-varying indoor lighting estimation. In CVPR, 2019.
  17. Texture synthesis using convolutional neural networks. In NeurIPS, 2015.
  18. Spectrophotometry: Accurate measurement of optical properties of materials. Elsevier, 2014.
  19. Brdf representation and acquisition. In Comput. Graph. Forum, 2016.
  20. Deep residual learning for image recognition, 2015.
  21. Berthold KP Horn. Determining lightness from an image. CGIP, 1974.
  22. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
  23. Decomposing single images for layered photo retouching. In Comput. Graph. Forum, 2017.
  24. Multi-view gradient consistency for svbrdf estimation of complex scenes under natural illumination. arXiv, 2022.
  25. Segment anything. In ICCV, 2023.
  26. Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML, 2013.
  27. Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. ACM TOG, 2017.
  28. Scraping textures from natural images for synthesis and editing. In ECCV, 2022a.
  29. Cgintrinsics: Better intrinsic image decomposition through physically-based rendering. In ECCV, 2018.
  30. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In CVPR, 2020.
  31. Physically-based editing of indoor scene lighting from a single image. In ECCV. Springer, 2022b.
  32. An approximate shading model with detail decomposition for object relighting. IJCV, 2019.
  33. Unsupervised learning for intrinsic image decomposition from a single image. In CVPR, 2020.
  34. Cross-task attention mechanism for dense multi-task learning. In WACV, 2023.
  35. Materia: Single image high-resolution material capture in the wild. In Comput. Graph. Forum, 2022.
  36. Share with thy neighbors: Single-view reconstruction by cross-instance consistency. In ECCV, 2022.
  37. Direct intrinsics: Learning albedo-shading decomposition by convolutional regression. In ICCV, 2015.
  38. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
  39. OpenAI. Chatgpt. chat.openai.org, 2023. Accessed: 2023-05-20.
  40. Poisson image editing. In Seminal Graphics Papers: Pushing the Boundaries. SIGGRAPH, 2023.
  41. PolyHaven. Pbr repository. https://www.polyhaven.com, 2021. Accessed: 2023-04-01.
  42. Learning transferable visual models from natural language supervision, 2021.
  43. Zero-shot text-to-image generation. In ICML, 2021.
  44. Hierarchical text-conditional image generation with clip latents. arXiv, 2022.
  45. Umat: Uncertainty-aware single image high resolution material capture. In CVPR, 2023.
  46. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  47. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
  48. Layered shape synthesis: automatic generation of control maps for non-stationary textures. ACM TOG, 2009.
  49. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
  50. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
  51. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. In ICML, 2023.
  52. Neural inverse rendering of an indoor scene from a single image. In ICCV, 2019.
  53. Materialistic: Selecting similar materials in images. In ACM TOG, 2023.
  54. Deep lambertian networks. arXiv, 2012.
  55. Surfacenet: Adversarial svbrdf estimation from a single image. In ICCV, 2021.
  56. Controlmat: A controlled generative approach to material capture. arXiv, 2023a.
  57. Matfuse: Controllable material generation with diffusion models, 2023b.
  58. Image quality assessment: from error visibility to structural similarity. T-IP, 2004.
  59. Unsupervised learning of probably symmetric deformable 3d objects from images in the wild. In CVPR, 2020.
  60. Unified perceptual parsing for scene understanding. In ECCV, 2018.
  61. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In CVPR, 2018.
  62. Photoscene: Photorealistic material and lighting transfer for indoor scenes. In CVPR, 2022.
  63. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
  64. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
  65. Glosh: Global-local spherical harmonics for intrinsic image decomposition. In ICCV, 2019.
  66. Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In CVPR, 2019.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. Material Palette