Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FreqAlign: Excavating Perception-oriented Transferability for Blind Image Quality Assessment from A Frequency Perspective (2309.17014v1)

Published 29 Sep 2023 in eess.IV

Abstract: Blind Image Quality Assessment (BIQA) is susceptible to poor transferability when the distribution shift occurs, e.g., from synthesis degradation to authentic degradation. To mitigate this, some studies have attempted to design unsupervised domain adaptation (UDA) based schemes for BIQA, which intends to eliminate the domain shift through adversarial-based feature alignment. However, the feature alignment is usually taken at the low-frequency space of features since the global average pooling operation. This ignores the transferable perception knowledge in other frequency components and causes the sub-optimal solution for the UDA of BIQA. To overcome this, from a novel frequency perspective, we propose an effective alignment strategy, i.e., Frequency Alignment (dubbed FreqAlign), to excavate the perception-oriented transferability of BIQA in the frequency space. Concretely, we study what frequency components of features are more proper for perception-oriented alignment. Based on this, we propose to improve the perception-oriented transferability of BIQA by performing feature frequency decomposition and selecting the frequency components that contained the most transferable perception knowledge for alignment. To achieve a stable and effective frequency selection, we further propose the frequency movement with a sliding window to find the optimal frequencies for alignment, which is composed of three strategies, i.e., warm up with pre-training, frequency movement-based selection, and perturbation-based finetuning. Extensive experiments under different domain adaptation settings of BIQA have validated the effectiveness of our proposed method. The code will be released at https://github.com/lixinustc/Openworld-IQA.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (95)
  1. H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, and M. Marchand, “Domain-adversarial neural networks,” arXiv preprint arXiv:1412.4446, 2014.
  2. G. Zhai and X. Min, “Perceptual image quality assessment: a survey,” Science China Information Sciences, vol. 63, pp. 1–52, 2020.
  3. B. Yan, B. Bare, and W. Tan, “Naturalness-aware deep no-reference image quality assessment,” IEEE Transactions on Multimedia, vol. 21, no. 10, pp. 2603–2615, 2019.
  4. W. Zhang, K. Ma, J. Yan, D. Deng, and Z. Wang, “Blind image quality assessment using a deep bilinear convolutional neural network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 1, pp. 36–47, 2018.
  5. S. Lee and S. J. Park, “A new image quality assessment method to detect and measure strength of blocking artifacts,” Signal Processing: Image Communication, vol. 27, no. 1, pp. 31–38, 2012.
  6. Z. P. Sazzad, Y. Kawayoke, and Y. Horita, “Spatial features based no reference image quality assessment for jpeg2000,” in 2007 IEEE International Conference on Image Processing, vol. 3.   IEEE, 2007, pp. III–517.
  7. E. Ong, W. Lin, Z. Lu, X. Yang, S. Yao, F. Pan, L. Jiang, and F. Moschetti, “A no-reference quality metric for measuring image blur,” in Seventh International Symposium on Signal Processing and Its Applications, 2003. Proceedings., vol. 1.   Ieee, 2003, pp. 469–472.
  8. S. Gu, J. Bao, D. Chen, and F. Wen, “Giqa: Generated image quality assessment,” in European conference on computer vision.   Springer, 2020, pp. 369–385.
  9. Q. Jiang, Y. Gu, C. Li, R. Cong, and F. Shao, “Underwater image enhancement quality evaluation: Benchmark dataset and objective metric,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 9, pp. 5959–5974, 2022.
  10. P. Chen, L. Li, J. Wu, W. Dong, and G. Shi, “Unsupervised curriculum domain adaptation for no-reference video quality assessment,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5178–5187.
  11. B. Chen, H. Li, H. Fan, and S. Wang, “No-reference screen content image quality assessment with unsupervised domain adaptation,” IEEE Transactions on Image Processing, vol. 30, pp. 5463–5476, 2021.
  12. J. Liu, X. Li, S. An, and Z. Chen, “Source-free unsupervised domain adaptation for blind image quality assessment,” arXiv preprint arXiv:2207.08124, 2022.
  13. Y. Lu, X. Li, J. Liu, and Z. Chen, “Styleam: Perception-oriented unsupervised domain adaption for non-reference image quality assessment,” arXiv preprint arXiv:2207.14489, 2022.
  14. Z. Qin, P. Zhang, F. Wu, and X. Li, “Fcanet: Frequency channel attention networks,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 783–792.
  15. X. Marichal, W.-Y. Ma, and H. Zhang, “Blur determination in the compressed domain using dct information,” in Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348), vol. 2.   IEEE, 1999, pp. 386–390.
  16. R. Muijs and I. Kirenko, “A no-reference blocking artifact measure for adaptive video processing,” in 2005 13th European signal processing conference.   IEEE, 2005, pp. 1–4.
  17. M. A. Saad, A. C. Bovik, and C. Charrier, “A dct statistics-based blind image quality index,” IEEE Signal Processing Letters, vol. 17, no. 6, pp. 583–586, 2010.
  18. ——, “Blind image quality assessment: A natural scene statistics approach in the dct domain,” IEEE transactions on Image Processing, vol. 21, no. 8, pp. 3339–3352, 2012.
  19. A. K. Moorthy and A. C. Bovik, “Blind image quality assessment: From natural scene statistics to perceptual quality,” IEEE transactions on Image Processing, vol. 20, no. 12, pp. 3350–3364, 2011.
  20. Y. Rao, W. Zhao, Z. Zhu, J. Lu, and J. Zhou, “Global filter networks for image classification,” Advances in Neural Information Processing Systems, vol. 34, pp. 980–993, 2021.
  21. Z.-Q. J. Xu, Y. Zhang, T. Luo, Y. Xiao, and Z. Ma, “Frequency principle: Fourier analysis sheds light on deep neural networks,” arXiv preprint arXiv:1901.06523, 2019.
  22. H. Wang, X. Wu, Z. Huang, and E. P. Xing, “High-frequency component helps explain the generalization of convolutional neural networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 8684–8694.
  23. R. Zhang, L. Xu, Z. Yu, Y. Shi, C. Mu, and M. Xu, “Deep-irtarget: An automatic target detector in infrared imagery using dual-domain feature extraction and allocation,” IEEE Transactions on Multimedia, vol. 24, pp. 1735–1749, 2021.
  24. J. Xie, W. Li, X. Zhan, Z. Liu, Y. S. Ong, and C. C. Loy, “Masked frequency modeling for self-supervised visual pre-training,” arXiv preprint arXiv:2206.07706, 2022.
  25. W. Xie, D. Song, C. Xu, C. Xu, H. Zhang, and Y. Wang, “Learning frequency-aware dynamic network for efficient super-resolution,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 4308–4317.
  26. Y. Pang, X. Li, X. Jin, Y. Wu, J. Liu, S. Liu, and Z. Chen, “Fan: Frequency aggregation network for real image super-resolution,” in European Conference on Computer Vision.   Springer, 2020, pp. 468–483.
  27. W. Zou, L. Chen, Y. Wu, Y. Zhang, Y. Xu, and J. Shao, “Joint wavelet sub-bands guided network for single image super-resolution,” IEEE Transactions on Multimedia, 2022.
  28. P. C. Madhusudana, N. Birkbeck, Y. Wang, B. Adsumilli, and A. C. Bovik, “Image quality assessment using contrastive learning,” IEEE Transactions on Image Processing, vol. 31, pp. 4149–4161, 2022.
  29. S. Sun, T. Yu, J. Xu, W. Zhou, and Z. Chen, “Graphiqa: Learning distortion graph representations for blind image quality assessment,” IEEE Transactions on Multimedia, 2022.
  30. J. Liu, W. Zhou, J. Xu, X. Li, S. An, and Z. Chen, “Liqa: Lifelong blind image quality assessment,” arXiv preprint arXiv:2104.14115, 2021.
  31. Y. Liu, K. Gu, Y. Zhang, X. Li, G. Zhai, D. Zhao, and W. Gao, “Unsupervised blind image quality evaluation via statistical measurements of structure, naturalness, and perception,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 4, pp. 929–943, 2019.
  32. L. Kang, P. Ye, Y. Li, and D. Doermann, “Convolutional neural networks for no-reference image quality assessment,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 1733–1740.
  33. H. Zhu, L. Li, J. Wu, W. Dong, and G. Shi, “Metaiqa: Deep meta-learning for no-reference image quality assessment,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14 143–14 152.
  34. W. Zhang, D. Li, C. Ma, G. Zhai, X. Yang, and K. Ma, “Continual learning for blind image quality assessment,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  35. X. Yang, F. Li, and H. Liu, “Ttl-iqa: Transitive transfer learning based no-reference image quality assessment,” IEEE Transactions on Multimedia, vol. 23, pp. 4326–4340, 2020.
  36. A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,” IEEE Signal processing letters, vol. 20, no. 3, pp. 209–212, 2012.
  37. A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Transactions on image processing, vol. 21, no. 12, pp. 4695–4708, 2012.
  38. G. Yin, W. Wang, Z. Yuan, C. Han, W. Ji, S. Sun, and C. Wang, “Content-variant reference image quality assessment via knowledge distillation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 3, 2022, pp. 3134–3142.
  39. A. Mittal, G. S. Muralidhar, J. Ghosh, and A. C. Bovik, “Blind image quality assessment without human training using latent quality factors,” IEEE Signal Processing Letters, vol. 19, no. 2, pp. 75–78, 2011.
  40. D. L. Ruderman, “The statistics of natural images,” Network: computation in neural systems, vol. 5, no. 4, p. 517, 1994.
  41. A. Srivastava, A. B. Lee, E. P. Simoncelli, and S.-C. Zhu, “On advances in statistical modeling of natural images,” Journal of mathematical imaging and vision, vol. 18, pp. 17–33, 2003.
  42. H. R. Sheikh, M. F. Sabir, and A. C. Bovik, “A statistical evaluation of recent full reference image quality assessment algorithms,” IEEE Transactions on image processing, vol. 15, no. 11, pp. 3440–3451, 2006.
  43. N. Ponomarenko, V. Lukin, A. Zelensky, K. Egiazarian, M. Carli, and F. Battisti, “Tid2008-a database for evaluation of full-reference visual quality assessment metrics,” Advances of modern radioelectronics, vol. 10, no. 4, pp. 30–45, 2009.
  44. S. Su, Q. Yan, Y. Zhu, C. Zhang, X. Ge, J. Sun, and Y. Zhang, “Blindly assess image quality in the wild guided by a self-adaptive hyper network,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3667–3676.
  45. J. Ke, Q. Wang, Y. Wang, P. Milanfar, and F. Yang, “Musiq: Multi-scale image quality transformer,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5148–5157.
  46. Z. Pan, F. Yuan, J. Lei, Y. Fang, X. Shao, and S. Kwong, “Vcrnet: Visual compensation restoration network for no-reference image quality assessment,” IEEE Transactions on Image Processing, vol. 31, pp. 1613–1627, 2022.
  47. W. Zhang, K. Ma, G. Zhai, and X. Yang, “Uncertainty-aware blind image quality assessment in the laboratory and wild,” IEEE Transactions on Image Processing, vol. 30, pp. 3474–3486, 2021.
  48. H. Lin, V. Hosu, and D. Saupe, “Kadid-10k: A large-scale artificially distorted iqa database,” in 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX).   IEEE, 2019, pp. 1–3.
  49. E. C. Larson and D. M. Chandler, “Most apparent distortion: full-reference image quality assessment and the role of strategy,” Journal of electronic imaging, vol. 19, no. 1, pp. 011 006–011 006, 2010.
  50. V. Hosu, H. Lin, T. Sziranyi, and D. Saupe, “Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment,” IEEE Transactions on Image Processing, vol. 29, pp. 4041–4056, 2020.
  51. D. Ghadiyaram and A. C. Bovik, “Massive online crowdsourced study of subjective and objective picture quality,” IEEE Transactions on Image Processing, vol. 25, no. 1, pp. 372–387, 2015.
  52. A. Rozantsev, M. Salzmann, and P. Fua, “Beyond sharing weights for deep domain adaptation,” IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 4, pp. 801–814, 2018.
  53. E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell, “Adversarial discriminative domain adaptation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7167–7176.
  54. B. Sun and K. Saenko, “Deep coral: Correlation alignment for deep domain adaptation,” in European conference on computer vision.   Springer, 2016, pp. 443–450.
  55. Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” The journal of machine learning research, vol. 17, no. 1, pp. 2096–2030, 2016.
  56. G. Kang, L. Jiang, Y. Yang, and A. G. Hauptmann, “Contrastive adaptation network for unsupervised domain adaptation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4893–4902.
  57. Y. Zhu, F. Zhuang, J. Wang, J. Chen, Z. Shi, W. Wu, and Q. He, “Multi-representation adaptation network for cross-domain image classification,” Neural Networks, vol. 119, pp. 214–221, 2019.
  58. G. Wei, C. Lan, W. Zeng, and Z. Chen, “Metaalign: Coordinating domain alignment and classification for unsupervised domain adaptation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 16 643–16 653.
  59. X. Jin, C. Lan, W. Zeng, and Z. Chen, “Re-energizing domain discriminator with sample relabeling for adversarial domain adaptation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9174–9183.
  60. J. Yang, J. Liu, N. Xu, and J. Huang, “Tvt: Transferable vision transformer for unsupervised domain adaptation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 520–530.
  61. Z. Yu, J. Li, L. Zhu, K. Lu, and H. T. Shen, “Classification certainty maximization for unsupervised domain adaptation,” IEEE Transactions on Circuits and Systems for Video Technology, 2023.
  62. C. Zhang, Z. Li, J. Liu, P. Peng, Q. Ye, S. Lu, T. Huang, and Y. Tian, “Self-guided adaptation: Progressive representation alignment for domain adaptive object detection,” IEEE Transactions on Multimedia, vol. 24, pp. 2246–2258, 2021.
  63. Y. Zhang, “A survey of unsupervised domain adaptation for visual recognition,” arXiv preprint arXiv:2112.06745, 2021.
  64. M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning transferable features with deep adaptation networks,” in International conference on machine learning.   PMLR, 2015, pp. 97–105.
  65. B. B. Damodaran, B. Kellenberger, R. Flamary, D. Tuia, and N. Courty, “Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 447–463.
  66. J. Yang, S. Shi, Z. Wang, H. Li, and X. Qi, “St3d: Self-training for unsupervised domain adaptation on 3d object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 10 368–10 378.
  67. X. Li, C. Lan, G. Wei, and Z. Chen, “Semantic-aware message broadcasting for efficient unsupervised domain adaptation,” arXiv preprint arXiv:2212.02739, 2022.
  68. H. Liu, J. Wang, and M. Long, “Cycle self-training for domain adaptation,” Advances in Neural Information Processing Systems, vol. 34, pp. 22 968–22 981, 2021.
  69. P. Wang, C. Ding, W. Tan, M. Gong, K. Jia, and D. Tao, “Uncertainty-aware clustering for unsupervised domain adaptive object re-identification,” IEEE Transactions on Multimedia, 2022.
  70. Z. Wang, Z.-R. Tang, J. Zhang, and Y. Fang, “Learning from synthetic data for opinion-free blind image quality assessment in the wild,” arXiv preprint arXiv:2106.14076, 2021.
  71. M. Tliba, A. Sekhri, M. A. Kerkouri, and A. Chetouani, “Deep-based quality assessment of medical images through domain adaptation,” in 2022 IEEE International Conference on Image Processing (ICIP).   IEEE, 2022, pp. 3692–3696.
  72. S. Nabavi, H. Simchi, M. E. Moghaddam, A. F. Frangi, and A. A. Abin, “Automatic multi-class cardiovascular magnetic resonance image quality assessment using unsupervised domain adaptation in spatial and frequency domains,” arXiv preprint arXiv:2112.06806, 2021.
  73. Q. Yang, Y. Liu, S. Chen, Y. Xu, and J. Sun, “No-reference point cloud quality assessment via domain adaptation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 21 179–21 188.
  74. X. Li, X. Jin, T. Yu, S. Sun, Y. Pang, Z. Zhang, and Z. Chen, “Learning omni-frequency region-adaptive representations for real image super-resolution,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 3, 2021, pp. 1975–1983.
  75. K. Zhang, M. Long, J. Chen, M. Liu, and J. Li, “Cfpnet: a denoising network for complex frequency band signal processing,” IEEE Transactions on Multimedia, 2023.
  76. J. Zhang, J. Huang, Z. Tian, and S. Lu, “Spectral unsupervised domain adaptation for visual recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9829–9840.
  77. J. Huang, D. Guan, A. Xiao, and S. Lu, “Rda: Robust domain adaptation via fourier adversarial attacking,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 8988–8999.
  78. Y. Yang and S. Soatto, “Fda: Fourier domain adaptation for semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 4085–4095.
  79. J. N. Kundu, A. R. Kulkarni, S. Bhambri, V. Jampani, and V. B. Radhakrishnan, “Amplitude spectrum transformation for open compound domain adaptive semantic segmentation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 2, 2022, pp. 1220–1227.
  80. M. Lin, Q. Chen, and S. Yan, “Network in network,” arXiv preprint arXiv:1312.4400, 2013.
  81. R. Shang, J. Zhang, L. Jiao, Y. Li, N. Marturi, and R. Stolkin, “Multi-scale adaptive feature fusion network for semantic segmentation in remote sensing images,” Remote Sensing, vol. 12, no. 5, p. 872, 2020.
  82. Z. Chen, Q. Xu, R. Cong, and Q. Huang, “Global context-aware progressive aggregation network for salient object detection,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 07, 2020, pp. 10 599–10 606.
  83. C. Luo, Q. Lin, W. Xie, B. Wu, J. Xie, and L. Shen, “Frequency-driven imperceptible adversarial attack on semantic similarity,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 15 315–15 324.
  84. Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in International conference on machine learning.   PMLR, 2015, pp. 1180–1189.
  85. A. Ciancio, E. A. da Silva, A. Said, R. Samadani, P. Obrador et al., “No-reference blur assessment of digital pictures based on multifeature classifiers,” IEEE Transactions on image processing, vol. 20, no. 1, pp. 64–75, 2010.
  86. N. Venkatanath, D. Praneeth, M. C. Bh, S. S. Channappayya, and S. S. Medasani, “Blind image quality evaluation using perception based features,” in 2015 Twenty First National Conference on Communications (NCC).   IEEE, 2015, pp. 1–6.
  87. X. Liu, J. Van De Weijer, and A. D. Bagdanov, “Rankiqa: Learning from rankings for no-reference image quality assessment,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1040–1049.
  88. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition.   Ieee, 2009, pp. 248–255.
  89. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019.
  90. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  91. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  92. X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, “Moment matching for multi-source domain adaptation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1406–1415.
  93. E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” arXiv preprint arXiv:1611.01144, 2016.
  94. N. Murray, L. Marchesotti, and F. Perronnin, “Ava: A large-scale database for aesthetic visual analysis,” in 2012 IEEE conference on computer vision and pattern recognition.   IEEE, 2012, pp. 2408–2415.
  95. S. He, Y. Zhang, R. Xie, D. Jiang, and A. Ming, “Rethinking image aesthetics assessment: Models, datasets and benchmarks,” in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, 2022, pp. 942–948.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now