Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ReShader: View-Dependent Highlights for Single Image View-Synthesis (2309.10689v3)

Published 19 Sep 2023 in cs.CV and cs.GR

Abstract: In recent years, novel view synthesis from a single image has seen significant progress thanks to the rapid advancements in 3D scene representation and image inpainting techniques. While the current approaches are able to synthesize geometrically consistent novel views, they often do not handle the view-dependent effects properly. Specifically, the highlights in their synthesized images usually appear to be glued to the surfaces, making the novel views unrealistic. To address this major problem, we make a key observation that the process of synthesizing novel views requires changing the shading of the pixels based on the novel camera, and moving them to appropriate locations. Therefore, we propose to split the view synthesis process into two independent tasks of pixel reshading and relocation. During the reshading process, we take the single image as the input and adjust its shading based on the novel camera. This reshaded image is then used as the input to an existing view synthesis method to relocate the pixels and produce the final novel view image. We propose to use a neural network to perform reshading and generate a large set of synthetic input-reshaded pairs to train our network. We demonstrate that our approach produces plausible novel view images with realistic moving highlights on a variety of real world scenes.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824 (2020).
  2. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer, 294–311.
  3. Benedikt Bitterli. 2014. Tungsten Renderer. https://github.com/tunabrain/tungsten.
  4. Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.
  5. A. Blake. 1985. Specular Stereo. In Proceedings of the 9th International Joint Conference on Artificial Intelligence - Volume 2 (Los Angeles, California) (IJCAI’85). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 973–976.
  6. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12684–12694.
  7. GeNVS: Generative Novel View Synthesis with 3D-Aware Diffusion Models. In arXiv.
  8. Scenescape: Text-driven consistent scene generation. arXiv preprint arXiv:2302.01133 (2023).
  9. NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion. In International Conference on Machine Learning.
  10. Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Images. In ACM SIGGRAPH 2022 Conference Proceedings (Vancouver, BC, Canada) (SIGGRAPH ’22). Association for Computing Machinery, New York, NY, USA, Article 14, 8 pages. https://doi.org/10.1145/3528233.3530755
  11. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33 (2020), 6840–6851.
  12. Holopix50k: A Large-Scale In-the-wild Stereo Image Dataset. In CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Seattle, WA, 2020.
  13. SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting. In Proceedings of the IEEE International Conference on Computer Vision.
  14. James T Kajiya. 1986. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques. 143–150.
  15. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics 42, 4 (July 2023). https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  16. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR).
  17. Neural Point Catacaustics for Novel-View Synthesis of Reflections. ACM Transactions on Graphics 41, 6 (2022), Article–201.
  18. Practical 3D photography. CVPR Workshop on Computer Vision for AR/VR (2019).
  19. One Shot 3D Photography. ACM Trans. Graph. 39, 4, Article 76 (aug 2020), 13 pages. https://doi.org/10.1145/3386569.3392420
  20. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. IJCV (2020).
  21. Leia. 2023. Leia Inc. https://www.leiainc.com/.
  22. Qinbo Li and Nima Kalantari. 2020. Synthesizing Light Field From a Single Image with Variable MPI and Two Network Fusion. ACM Transactions on Graphics 39, 6 (2020). https://doi.org/10.1145/3414685.3417785
  23. Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  24. Physically-Based Editing of Indoor Scene Lighting from a Single Image. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VI. Springer, 555–572.
  25. Real-time Reflective and Refractive Novel-view Synthesis, Jan Bender, Arjan Kuijper, Tatiana von Landesberger, Holger Theisel, and Philipp Urban (Eds.). Eurographics Association, Darmstadt, Germany, 9–16. https://doi.org/10.2312/vmv.20141270
  26. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV.
  27. 3D Ken Burns Effect from a Single Image. ACM Trans. Graph. 38, 6, Article 184 (nov 2019), 15 pages. https://doi.org/10.1145/3355089.3356528
  28. Multi-View Relighting Using a Geometry-Aware Network. ACM Trans. Graph. 38, 4, Article 78 (jul 2019), 14 pages. https://doi.org/10.1145/3306346.3323013
  29. Free-viewpoint Indoor Neural Relighting from Multi-view Stereo. ACM Transactions on Graphics (2021). http://www-sop.inria.fr/reves/Basilic/2021/PMGD21
  30. DreamFusion: Text-to-3D using 2D Diffusion. arXiv (2022).
  31. Vision Transformers for Dense Prediction. ICCV (2021).
  32. Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 3 (2022).
  33. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 234–241.
  34. S. Roth and M.J. Black. 2006. Specular Flow and the Recovery of Surface Structure. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), Vol. 2. 1869–1876. https://doi.org/10.1109/CVPR.2006.290
  35. Neural Inverse Rendering of an Indoor Scene From a Single Image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
  36. Layered Depth Images. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’98). Association for Computing Machinery, New York, NY, USA, 231–242. https://doi.org/10.1145/280814.280882
  37. 3D Photography using Context-aware Layered Depth Inpainting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  38. 3D Neural Field Generation using Triplane Diffusion. arXiv preprint arXiv:2211.16677 (2022).
  39. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
  40. Image-Based Rendering for Scenes with Reflections. ACM Trans. Graph. 31, 4, Article 100 (jul 2012), 10 pages.
  41. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7495–7504.
  42. Learning to synthesize a 4D RGBD light field from a single image. In Proceedings of the IEEE International Conference on Computer Vision. 2243–2251.
  43. Richard Tucker and Noah Snavely. 2020. Single-view View Synthesis with Multiplane Images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  44. Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields. CVPR (2022).
  45. 3D Moments from Near-Duplicate Photos. In CVPR.
  46. Learning indoor inverse rendering with 3d spatially-varying lighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12538–12547.
  47. Novel view synthesis with diffusion models. arXiv preprint arXiv:2210.04628 (2022).
  48. SynSin: End-to-end View Synthesis from a Single Image. In CVPR.
  49. NeX: Real-time View Synthesis with Neural Basis Expansion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  50. Scalable neural indoor scene rendering. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1–16.
  51. Deep view synthesis from sparse photometric images. ACM Transactions on Graphics (ToG) 38, 4 (2019), 1–13.
  52. Deep image-based relighting from optimal sparse samples. ACM Transactions on Graphics (ToG) 37, 4 (2018), 1–13.
  53. pixelNeRF: Neural Radiance Fields from One or Few Images. In CVPR.
  54. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (TOG) 40, 6 (2021), 1–18.
  55. View Synthesis by Appearance Flow. In Computer Vision – ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 286–301.
  56. Path-space Motion Estimation and Decomposition for Robust Animation Filtering. Computer Graphics Forum 34, 4 (2015), 131–142.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Avinash Paliwal (9 papers)
  2. Brandon Nguyen (2 papers)
  3. Andrii Tsarov (4 papers)
  4. Nima Khademi Kalantari (19 papers)
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube