Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DSI2I: Dense Style for Unpaired Image-to-Image Translation (2212.13253v3)

Published 26 Dec 2022 in cs.CV

Abstract: Unpaired exemplar-based image-to-image (UEI2I) translation aims to translate a source image to a target image domain with the style of a target image exemplar, without ground-truth input-translation pairs. Existing UEI2I methods represent style using one vector per image or rely on semantic supervision to define one style vector per object. Here, in contrast, we propose to represent style as a dense feature map, allowing for a finer-grained transfer to the source image without requiring any external semantic information. We then rely on perceptual and adversarial losses to disentangle our dense style and content representations. To stylize the source content with the exemplar style, we extract unsupervised cross-domain semantic correspondences and warp the exemplar style to the source content. We demonstrate the effectiveness of our method on four datasets using standard metrics together with a localized style metric we propose, which measures style similarity in a class-wise manner. Our results show that the translations produced by our approach are more diverse, preserve the source content better, and are closer to the exemplars when compared to the state-of-the-art methods. Project page: https://github.com/IVRL/dsi2i

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Neural best-buddies: Sparse cross-domain correspondence. ACM Transactions on Graphics (TOG), 37(4):1–14, 2018.
  2. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009.
  3. Dunit: Detection-based unsupervised image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  4787–4796, 2020.
  4. Attention-gan for object transfiguration in wild images. In Proceedings of the European conference on computer vision (ECCV), pp.  164–180, 2018.
  5. Photowct2: Compact autoencoder for photorealistic style transfer resulting from blockwise training and skip connections of high-frequency residuals. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.  2868–2877, 2022.
  6. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  7. Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
  8. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp.  2414–2423, 2016.
  9. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition, 2012.
  10. Generative adversarial networks. arXiv preprint arXiv:1406.2661, 2014.
  11. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  12. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning, pp. 1989–1998. PMLR, 2018.
  13. Qs-attn: Query-selected attention for contrastive learning in i2i translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  18291–18300, 2022.
  14. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pp.  1501–1510, 2017.
  15. Multimodal unsupervised image-to-image translation. In Proceedings of the European Conference on Computer Vision, pp.  172–189, 2018.
  16. Memory-guided unsupervised image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  6558–6567, 2021.
  17. Masked and adaptive transformer for exemplar based image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  22418–22427, 2023.
  18. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision, pp.  694–711. Springer, 2016.
  19. Exploring patch-wise semantic relation for contrastive learning in image-to-image translation tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  18260–18269, 2022.
  20. Instaformer: Instance-aware image-to-image translation with transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  18321–18331, 2022.
  21. Deformable style transfer. In Proceedings of the European Conference on Computer Vision, pp.  246–261. Springer, 2020.
  22. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  23. Style transfer by relaxed optimal transport and self-similarity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10051–10060, 2019.
  24. Diverse image-to-image translation via disentangled representations. In Proceedings of the European Conference on Computer Vision, pp.  35–51, 2018.
  25. Positional normalization. In Advances in Neural Information Processing Systems, pp. 1620–1632, 2019.
  26. Demystifying neural style transfer. arXiv preprint arXiv:1701.01036, 2017a.
  27. Universal style transfer via feature transforms. Advances in Neural Information Processing Systems, 30, 2017b.
  28. A closed-form solution to photorealistic image stylization. In Proceedings of the European Conference on Computer Vision, pp.  453–468, 2018.
  29. Sift flow: Dense correspondence across scenes and its applications. IEEE transactions on pattern analysis and machine intelligence, 33(5):978–994, 2010.
  30. Unsupervised image-to-image translation networks. Advances in neural information processing systems, 30, 2017.
  31. Learning to warp for style transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  3702–3711, 2021.
  32. Semantic correspondence as an optimal transport problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  4463–4472, 2020.
  33. Hyperpixel flow: Semantic correspondence with multi-layer neural features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  3395–3404, 2019.
  34. Instagan: Instance-aware image-to-image translation. arXiv preprint arXiv:1812.10889, 2018.
  35. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  36. Contrastive learning for unpaired image-to-image translation. In Proceedings of the European Conference on Computer Vision, pp.  319–345. Springer, 2020.
  37. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748–8763. PMLR, 2021.
  38. Playing for data: Ground truth from computer games. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Proceedings of the European Conference on Computer Vision, volume 9906 of LNCS, pp.  102–118. Springer International Publishing, 2016.
  39. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
  40. Towards instance-level image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  3683–3692, 2019.
  41. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  42. Dense contrastive learning for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  3024–3033, 2021.
  43. Industrial style transfer with large-scale geometric warping and content preservation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  7834–7843, 2022.
  44. Photorealistic style transfer via wavelet transforms. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  9036–9045, 2019.
  45. Dilated residual networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp.  472–480, 2017.
  46. Unbalanced feature transport for exemplar-based image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  15028–15038, 2021.
  47. Bi-level feature alignment for versatile image translation and manipulation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVI, pp.  224–241. Springer, 2022a.
  48. Marginal contrastive correspondence for guided image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  10663–10672, 2022b.
  49. Cross-domain correspondence learning for exemplar-based image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5143–5153, 2020.
  50. Multimodal style transfer via graph cuts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  5943–5951, 2019.
  51. The spatially-correlative loss for various image translation tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  16407–16417, 2021.
  52. Cocosnet v2: Full-resolution correspondence learning for image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  11465–11475, 2021.
  53. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp.  2223–2232, 2017.
  54. Sean: Image synthesis with semantic region-adaptive normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5104–5113, 2020.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now