RPNR: Robust-Perception Neural Reshading (2401.14510v1)
Abstract: Augmented Reality (AR) applications necessitates methods of inserting needed objects into scenes captured by cameras in a way that is coherent with the surroundings. Common AR applications require the insertion of predefined 3D objects with known properties and shape. This simplifies the problem since it is reduced to extracting an illumination model for the object in that scene by understanding the surrounding light sources. However, it is often not the case that we have information about the properties of an object, especially when we depart from a single source image. Our method renders such source fragments in a coherent way with the target surroundings using only these two images. Our pipeline uses a Deep Image Prior (DIP) network based on a U-Net architecture as the main renderer, alongside robust-feature extracting networks that are used to apply needed losses. Our method does not require any pair-labeled data, and no extensive training on a dataset. We compare our method using qualitative metrics to the baseline methods such as Cut and Paste, Cut And Paste Neural Rendering, and Image Harmonization
- Cut-and-paste neural rendering. CoRR, abs/2010.05907, 2020.
- Edwin H Land. The retinex theory of color vision. Scientific american, 1977.
- Lightness and retinex theory. josa, 1971.
- Better intrinsic image decomposition through physically-based rendering. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
- An l 1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Transactions on Graphics (TOG), 2015.
- Revisiting deep intrinsic image decompositions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
- Self-supervised intrinsic image decomposition. In Advances in Neural Information Processing Systems, 2017.
- Intrinsic images in the wild. ACM Transactions on Graphics (TOG), 2014.
- Multi-scale image harmonization. ACM Transactions on Graphics (TOG), 29(4):1–10, 2010.
- Deep image harmonization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, page 3789–3797, 2017.
- Deep image harmonization via domain verification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, page 8394–8403, 2020.
- Fast spatially-varying indoor lighting estimation. CoRR, abs/1906.03799, 2019.
- Deep sky modeling for single image outdoor lighting estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
- Lighthouse: Predicting lighting volumes for spatially-coherent illumination. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8077–8086, 2020.
- Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv preprint arXiv, 2020.
- Single image portrait relighting. ACM Transactions on Graphics (Proceedings SIGGRAPH), 2019.
- Deep single-image portrait relighting. . In Proceedings of the IEEE International Conference on Computer Vision, 2019.
- Learning physics-guided face relighting under directional light. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5123–5132, 2020.
- Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and SVBRDF from a single image. CoRR, abs/1905.02722, 2019.
- Deep image prior. International Journal of Computer Vision, 128(7):1867–1888, Mar 2020.
- Generative adversarial networks, 2014.
- A u-net based discriminator for generative adversarial networks, 2021.
- Real-time joint semantic segmentation and depth estimation using asymmetric annotations. CoRR, abs/1809.04766, 2018.