Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SPColor: Semantic Prior Guided Exemplar-based Image Colorization (2304.06255v3)

Published 13 Apr 2023 in cs.CV

Abstract: Exemplar-based image colorization aims to colorize a target grayscale image based on a color reference image, and the key is to establish accurate pixel-level semantic correspondence between these two images. Previous methods search for correspondence across the entire reference image, and this type of global matching is easy to get mismatch. We summarize the difficulties in two aspects: (1) When the reference image only contains a part of objects related to target image, improper correspondence will be established in unrelated regions. (2) It is prone to get mismatch in regions where the shape or texture of the object is easily confused. To overcome these issues, we propose SPColor, a semantic prior guided exemplar-based image colorization framework. Different from previous methods, SPColor first coarsely classifies pixels of the reference and target images to several pseudo-classes under the guidance of semantic prior, then the correspondences are only established locally between the pixels in the same class via the newly designed semantic prior guided correspondence network. In this way, improper correspondence between different semantic classes is explicitly excluded, and the mismatch is obviously alleviated. Besides, to better reserve the color from reference, a similarity masked perceptual loss is designed. Noting that the carefully designed SPColor utilizes the semantic prior provided by an unsupervised segmentation model, which is free for additional manual semantic annotations. Experiments demonstrate that our model outperforms recent state-of-the-art methods both quantitatively and qualitatively on public dataset.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” in European conference on computer vision.   Springer, 2016, pp. 649–666.
  2. P. Vitoria, L. Raad, and C. Ballester, “Chromagan: Adversarial picture colorization with semantic class distribution,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020, pp. 2445–2454.
  3. S. Wan, Y. Xia, L. Qi, Y.-H. Yang, and M. Atiquzzaman, “Automated colorization of a grayscale image with seed points propagation,” IEEE Transactions on Multimedia, vol. 22, no. 7, pp. 1756–1768, 2020.
  4. M. Kumar, D. Weissenborn, and N. Kalchbrenner, “Colorization transformer,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=5NA1PinlGFu
  5. J. Li, W. Li, Z. Xu, Y. Wang, and Q. Liu, “Wavelet transform-assisted adaptive generative modeling for colorization,” IEEE Transactions on Multimedia, pp. 1–1, 2022.
  6. Y. Zhao, L.-M. Po, K.-W. Cheung, W.-Y. Yu, and Y. A. U. Rehman, “Scgan: Saliency map-guided colorization with generative adversarial network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 8, pp. 3062–3077, 2020.
  7. F.-Y. Luo, S.-L. Liu, Y.-J. Cao, K.-F. Yang, C.-Y. Xie, Y. Liu, and Y.-J. Li, “Nighttime thermal infrared image colorization with feedback-based object appearance learning,” IEEE Transactions on Circuits and Systems for Video Technology, pp. 1–1, 2023.
  8. A. Levin, D. Lischinski, and Y. Weiss, “Colorization using optimization,” in ACM SIGGRAPH 2004 Papers, 2004, pp. 689–694.
  9. Y.-C. Huang, Y.-S. Tung, J.-C. Chen, S.-W. Wang, and J.-L. Wu, “An adaptive edge detection based colorization algorithm and its applications,” in Proceedings of the 13th annual ACM international conference on Multimedia, 2005, pp. 351–354.
  10. Y. Qu, T.-T. Wong, and P.-A. Heng, “Manga colorization,” ACM Transactions on Graphics (ToG), vol. 25, no. 3, pp. 1214–1220, 2006.
  11. Q. Luan, F. Wen, D. Cohen-Or, L. Liang, Y.-Q. Xu, and H.-Y. Shum, “Natural image colorization,” in Proceedings of the 18th Eurographics conference on Rendering Techniques, 2007, pp. 309–320.
  12. K. Panetta, L. Bao, and S. Agaian, “Fast hue-division-based selective color transfer,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 9, pp. 2853–2866, 2019.
  13. R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros, “Real-time user-guided image colorization with learned deep priors,” arXiv preprint arXiv:1705.02999, 2017.
  14. E. Kim, S. Lee, J. Park, S. Choi, C. Seo, and J. Choo, “Deep edge-aware interactive colorization against color-bleeding effects,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14 667–14 676.
  15. Z. Xu, T. Wang, F. Fang, Y. Sheng, and G. Zhang, “Stylization-based architecture for fast deep exemplar colorization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9363–9372.
  16. H. Zhao, W. Wu, Y. Liu, and D. He, “Color2embed: Fast exemplar-based image colorization using color embeddings,” arXiv preprint arXiv:2106.08017, 2021.
  17. Y. Huang, S. Qiu, C. Wang, and C. Li, “Learning representations for high-dynamic-range image color transfer in a self-supervised way,” IEEE Transactions on Multimedia, vol. 23, pp. 176–188, 2021.
  18. Y. Bai, C. Dong, Z. Chai, A. Wang, Z. Xu, and C. Yuan, “Semantic-sparse colorization network for deep exemplar-based colorization,” in European Conference on Computer Vision.   Springer, 2022, pp. 505–521.
  19. X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 1501–1510.
  20. Y.-W. Tai, J. Jia, and C.-K. Tang, “Local color transfer via probabilistic segmentation by expectation-maximization,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1.   IEEE, 2005, pp. 747–754.
  21. R. Ironi, D. Cohen-Or, and D. Lischinski, “Colorization by example.” Rendering techniques, vol. 29, pp. 201–210, 2005.
  22. A. Y.-S. Chia, S. Zhuo, R. K. Gupta, Y.-W. Tai, S.-Y. Cho, P. Tan, and S. Lin, “Semantic colorization with internet images,” ACM Transactions on Graphics (TOG), vol. 30, no. 6, pp. 1–8, 2011.
  23. R. K. Gupta, A. Y.-S. Chia, D. Rajan, E. S. Ng, and H. Zhiyong, “Image colorization using similar images,” in Proceedings of the 20th ACM international conference on Multimedia, 2012, pp. 369–378.
  24. A. Bugeau, V.-T. Ta, and N. Papadakis, “Variational exemplar-based image colorization,” IEEE Transactions on Image Processing, vol. 23, no. 1, pp. 298–307, 2013.
  25. S. Wu, X. Yan, W. Liu, S. Xu, and S. Zhang, “Self-driven dual-path learning for reference-based line art colorization under limited data,” IEEE Transactions on Circuits and Systems for Video Technology, pp. 1–1, 2023.
  26. M. He, D. Chen, J. Liao, P. V. Sander, and L. Yuan, “Deep exemplar-based colorization,” ACM Transactions on Graphics (TOG), vol. 37, no. 4, pp. 1–16, 2018.
  27. B. Zhang, M. He, J. Liao, P. V. Sander, L. Yuan, A. Bermak, and D. Chen, “Deep exemplar-based video colorization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 8052–8061.
  28. P. Lu, J. Yu, X. Peng, Z. Zhao, and X. Wang, “Gray2colornet: Transfer more colors from reference image,” in Proceedings of the 28th ACM International Conference on Multimedia, 2020, pp. 3210–3218.
  29. W. Yin, P. Lu, Z. Zhao, and X. Peng, “Yes,” attention is all you need”, for exemplar based colorization,” in Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 2243–2251.
  30. M. G. Blanch, I. Khalifeh, N. E. O’Connor, and M. Mrak, “Attention-based stylisation for exemplar image colourisation,” in 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP).   IEEE, 2021, pp. 1–6.
  31. M. Hamilton, Z. Zhang, B. Hariharan, N. Snavely, and W. T. Freeman, “Unsupervised semantic segmentation by distilling feature correspondences,” arXiv preprint arXiv:2203.08414, 2022.
  32. J. Duchon, “Splines minimizing rotation-invariant semi-norms in sobolev spaces,” in Constructive theory of functions of several variables.   Springer, 1977, pp. 85–100.
  33. H. Chui and A. Rangarajan, “A new algorithm for non-rigid point matching,” in Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662), vol. 2.   IEEE, 2000, pp. 44–51.
  34. B. Li, Y.-K. Lai, M. John, and P. L. Rosin, “Automatic example-based image colorization using location-aware cross-scale matching,” IEEE Transactions on Image Processing, vol. 28, no. 9, pp. 4606–4619, 2019.
  35. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  36. X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7794–7803.
  37. J. H. Cho, U. Mall, K. Bala, and B. Hariharan, “Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 16 794–16 804.
  38. H. Caesar, J. Uijlings, and V. Ferrari, “Coco-stuff: Thing and stuff classes in context,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1209–1218.
  39. M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3213–3223.
  40. G. Long, L. Kneip, J. M. Alvarez, H. Li, X. Zhang, and Q. Yu, “Learning image matching by simply watching video,” in European Conference on Computer Vision.   Springer, 2016, pp. 434–450.
  41. M. Mathieu, C. Couprie, and Y. LeCun, “Deep multi-scale video prediction beyond mean square error,” arXiv preprint arXiv:1511.05440, 2015.
  42. S. Niklaus, L. Mai, and F. Liu, “Video frame interpolation via adaptive separable convolution,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 261–270.
  43. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition.   Ieee, 2009, pp. 248–255.
  44. G. Larsson, M. Maire, and G. Shakhnarovich, “Learning representations for automatic colorization,” in European conference on computer vision.   Springer, 2016, pp. 577–593.
  45. F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” arXiv preprint arXiv:1511.07122, 2015.
  46. Z. Huang, N. Zhao, and J. Liao, “Unicolor: A unified framework for multi-modal colorization with transformer,” ACM Transactions on Graphics (TOG), vol. 41, no. 6, pp. 1–16, 2022.
  47. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” Advances in neural information processing systems, vol. 30, 2017.
  48. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  49. J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Hornung, and L. Van Gool, “The 2017 davis challenge on video object segmentation,” arXiv preprint arXiv:1704.00675, 2017.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now