2000 character limit reached
SEM-CS: Semantic CLIPStyler for Text-Based Image Style Transfer (2303.06334v1)
Published 11 Mar 2023 in cs.CV
Abstract: CLIPStyler demonstrated image style transfer with realistic textures using only the style text description (instead of requiring a reference style image). However, the ground semantics of objects in style transfer output is lost due to style spillover on salient and background objects (content mismatch) or over-stylization. To solve this, we propose Semantic CLIPStyler (Sem-CS) that performs semantic style transfer. Sem-CS first segments the content image into salient and non-salient objects and then transfers artistic style based on a given style text description. The semantic style transfer is achieved using global foreground loss (for salient objects) and global background loss (for non-salient objects). Our empirical results, including DISTS, NIMA and user study scores, show that our proposed framework yields superior qualitative and quantitative performance.
- “Artistic style transfer with internal-external learning and contrastive learning,” in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, Eds. 2021, vol. 34, pp. 26561–26573, Curran Associates, Inc.
- “Universal style transfer via feature transforms,” Advances in neural information processing systems, vol. 30, 2017.
- “Arbitrary style transfer with style-attentional networks,” in proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 5880–5888.
- “Learning linear transformations for fast arbitrary style transfer,” ArXiv, vol. abs/1808.04537, 2018.
- “The contextual loss for image transformation with non-aligned data,” in Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIV. 2018, vol. 11218 of Lecture Notes in Computer Science, pp. 800–815, Springer.
- “Cross-modal style transfer,” in 2018 25th IEEE International Conference on Image Processing (ICIP), 2018, pp. 2157–2161.
- “Photorealistic style transfer with screened poisson equation,” in Proceedings of the British Machine Vision Conference (BMVC), Gabriel Brostow Tae-Kyun Kim, Stefanos Zafeiriou and Krystian Mikolajczyk, Eds. September 2017, pp. 153.1–153.12, BMVA Press.
- “Style transfer using optimal transport via wasserstein distance,” in 2022 IEEE International Conference on Image Processing (ICIP), 2022, pp. 2681–2685.
- “Image style transfer using convolutional neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2414–2423.
- “A patch-based approach for artistic style transfer via constrained multi-scale image matching,” in 2022 IEEE International Conference on Image Processing (ICIP). IEEE, 2022, pp. 3490–3494.
- “Clipstyler: Image style transfer with a single text condition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 18062–18071.
- “Deep photo style transfer,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6997–7005, 2017.
- “Deepobjstyle: Deep object-based photo style transfer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2021, pp. 711–719.
- “Generative artisan: A semantic-aware and controllable clipstyler,” .
- “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431–3440.
- “Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8364–8375.
- “Image quality assessment: Unifying structure and texture similarity,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 5, pp. 2567–2581, 2020.
- “Nima: Neural image assessment,” IEEE transactions on image processing, vol. 27, no. 8, pp. 3998–4011, 2018.
- “Rethinking and improving the robustness of image style transfer,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 124–133.
- “Stylegan-nada: Clip-guided domain adaptation of image generators,” ACM Transactions on Graphics (TOG), vol. 41, no. 4, pp. 1–13, 2022.
- “Styleclip: Text-driven manipulation of stylegan imagery,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2085–2094.