Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deep Feature Statistics Mapping for Generalized Screen Content Image Quality Assessment (2209.05321v4)

Published 12 Sep 2022 in cs.CV

Abstract: The statistical regularities of natural images, referred to as natural scene statistics, play an important role in no-reference image quality assessment. However, it has been widely acknowledged that screen content images (SCIs), which are typically computer generated, do not hold such statistics. Here we make the first attempt to learn the statistics of SCIs, based upon which the quality of SCIs can be effectively determined. The underlying mechanism of the proposed approach is based upon the mild assumption that the SCIs, which are not physically acquired, still obey certain statistics that could be understood in a learning fashion. We empirically show that the statistics deviation could be effectively leveraged in quality assessment, and the proposed method is superior when evaluated in different settings. Extensive experimental results demonstrate the Deep Feature Statistics based SCI Quality Assessment (DFSS-IQA) model delivers promising performance compared with existing NR-IQA models and shows a high generalization capability in the cross-dataset settings. The implementation of our method is publicly available at https://github.com/Baoliang93/DFSS-IQA.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (88)
  1. S. Wang, L. Ma, Y. Fang, W. Lin, S. Ma, and W. Gao, “Just noticeable difference estimation for screen content images,” IEEE Transactions on Image Processing, vol. 25, no. 8, pp. 3838–3851, 2016.
  2. S. Wang, K. Gu, X. Zhang, W. Lin, S. Ma, and W. Gao, “Reduced-reference quality assessment of screen content images,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 1, pp. 1–14, 2016.
  3. S. Wang, K. Gu, S. Ma, and W. Gao, “Joint chroma downsampling and upsampling for screen content image,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 26, no. 9, pp. 1595–1609, 2015.
  4. Q. Xu, J. Xiong, X. Cao, and Y. Yao, “Parsimonious mixed-effects hodgerank for crowdsourced preference aggregation,” in ACM International Conference on Multimedia (ACM MM), 2016, pp. 841–850.
  5. Q. Xu, J. Xiong, Q. Huang, and Y. Yao, “Online hodgerank on random graphs for crowdsourceable QoE evaluation,” IEEE Transactions on Multimedia, vol. 16, no. 2, pp. 373–386, 2013.
  6. Q. Xu, J. Xiong, Q. Huang, and Y. Yao, “Robust evaluation for quality of experience in crowdsourcing,” in ACM International Conference Multimedia (ACM MM), 2013, pp. 43–52.
  7. Q. Xu, Q. Huang, and Y. Yao, “Online crowdsourcing subjective image quality assessment,” in ACM International Conference Multimedia (ACM MM), 2012, pp. 359–368.
  8. Q. Xu, Q. Huang, T. Jiang, B. Yan, W. Lin, and Y. Yao, “HodgeRank on random graphs for subjective video quality assessment,” IEEE Transactions on Multimedia, vol. 14, no. 3, pp. 844–857, 2012.
  9. X. Min, K. Gu, G. Zhai, X. Yang, W. Zhang, P. Le Callet, and C. W. Chen, “Screen content quality assessment: overview, benchmark, and beyond,” ACM Computing Surveys, vol. 54, no. 9, pp. 1–36, 2021.
  10. Y. Fang, J. Yan, J. Liu, S. Wang, Q. Li, and Z. Guo, “Objective quality assessment of screen content images by uncertainty weighting,” IEEE Transactions on Image Processing, vol. 26, no. 4, pp. 2016–2027, 2017.
  11. Z. Ni, L. Ma, H. Zeng, J. Chen, C. Cai, and K.-K. Ma, “ESIM: Edge similarity for screen content image quality assessment,” IEEE Transactions on Image Processing, vol. 26, no. 10, pp. 4818–4831, 2017.
  12. Z. Ni, H. Zeng, L. Ma, J. Hou, J. Chen, and K.-K. Ma, “A gabor feature-based quality assessment model for the screen content images,” IEEE Transactions on Image Processing, vol. 27, no. 9, pp. 4516–4528, 2018.
  13. Y. Zhang, D. M. Chandler, and X. Mou, “Quality assessment of screen content images via convolutional-neural-network-based synthetic/natural segmentation,” IEEE Transactions on Image Processing, vol. 27, no. 10, pp. 5113–5128, 2018.
  14. J. Yang, Z. Bian, Y. Zhao, W. Lu, and X. Gao, “Full-reference quality assessment for screen content images based on the concept of global-guidance and local-adjustment,” IEEE Transactions on Broadcasting, vol. 67, no. 3, pp. 696–709, 2021.
  15. K. Gu, J. Zhou, J.-F. Qiao, G. Zhai, W. Lin, and A. C. Bovik, “No-reference quality assessment of screen content pictures,” IEEE Transactions on Image Processing, vol. 26, no. 8, pp. 4005–4018, 2017.
  16. Y. Fang, J. Yan, L. Li, J. Wu, and W. Lin, “No reference quality assessment for screen content images with both local and global feature representation,” IEEE Transactions on Image Processing, vol. 27, no. 4, pp. 1600–1610, 2017.
  17. L. Zheng, L. Shen, J. Chen, P. An, and J. Luo, “No-reference quality assessment for screen content images based on hybrid region features fusion,” IEEE Transactions on Multimedia, vol. 21, no. 8, pp. 2057–2070, 2019.
  18. X. Jiang, L. Shen, G. Feng, L. Yu, and P. An, “Deep optimization model for screen content image quality assessment using neural networks,” arXiv preprint arXiv:1903.00705, 2019.
  19. Z. Cheng, M. Takeuchi, K. Kanai, and J. Katto, “A fast no-reference screen content image quality prediction using convolutional neural networks,” in IEEE International Conference on Multimedia & Expo Workshops (ICMEW), 2018, pp. 1–6.
  20. J. Chen, L. Shen, L. Zheng, and X. Jiang, “Naturalization module in neural networks for screen content image quality assessment,” IEEE Signal Processing Letters, vol. 25, no. 11, pp. 1685–1689, 2018.
  21. B. Chen, H. Li, H. Fan, and S. Wang, “No-reference screen content image quality assessment with unsupervised domain adaptation,” IEEE Transactions on Image Processing, vol. 30, pp. 5463–5476, 2021.
  22. J. Yang, Z. Bian, Y. Zhao, W. Lu, and X. Gao, “Staged-learning: Assessing the quality of screen content images from distortion information,” IEEE Signal Processing Letters, vol. 28, pp. 1480–1484, 2021.
  23. N. Ponomarenko, L. Jin, O. Ieremeiev, V. Lukin, K. Egiazarian, J. Astola, B. Vozel, K. Chehdi, M. Carli, F. Battisti et al., “Image database TID2013: Peculiarities, results and perspectives,” Signal Processing: Image Communication, vol. 30, pp. 57–77, 2015.
  24. H. Yang, Y. Fang, and W. Lin, “Perceptual quality assessment of screen content images,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 4408–4421, 2015.
  25. A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,” IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209–212, 2012.
  26. A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Transactions on Image Processing, vol. 21, no. 12, pp. 4695–4708, 2012.
  27. A. K. Moorthy and A. C. Bovik, “A two-step framework for constructing blind image quality indices,” IEEE Signal Processing Letters, vol. 17, no. 5, pp. 513–516, 2010.
  28. Moorthy, Anush Krishna and Bovik, Alan Conrad, “Blind image quality assessment: From natural scene statistics to perceptual quality,” IEEE Transactions on Image Processing, vol. 20, no. 12, pp. 3350–3364, 2011.
  29. K. Gu, S. Wang, H. Yang, W. Lin, G. Zhai, X. Yang, and W. Zhang, “Saliency-guided quality assessment of screen content images,” IEEE Transactions on Multimedia, vol. 18, no. 6, pp. 1098–1110, 2016.
  30. Z. Ni, L. Ma, H. Zeng, Y. Fu, L. Xing, and K.-K. Ma, “SCID: A database for screen content images quality assessment,” in IEEE International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), 2017, pp. 774–779.
  31. H. Tang, N. Joshi, and A. Kapoor, “Learning a blind measure of perceptual image quality,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 305–312.
  32. Q. Wu, H. Li, F. Meng, K. N. Ngan, B. Luo, C. Huang, and B. Zeng, “Blind image quality assessment based on multichannel feature fusion and label transfer,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 26, no. 3, pp. 425–440, 2015.
  33. K. Friston, J. Kilner, and L. Harrison, “A free energy principle for the brain,” Journal of Physiology-Paris, vol. 100, no. 1-3, pp. 70–87, 2006.
  34. K. Friston, “The free-energy principle: a unified brain theory?” Nature Reviews Neuroscience, vol. 11, no. 2, pp. 127–138, 2010.
  35. K. Gu, G. Zhai, X. Yang, W. Zhang, and L. Liang, “No-reference image quality assessment metric by combining free energy theory and structural degradation model,” in IEEE International Conference on Multimedia and Expo (ICME), 2013, pp. 1–6.
  36. K. Gu, G. Zhai, X. Yang, and W. Zhang, “Using free energy principle for blind image quality assessment,” IEEE Transactions on Multimedia, vol. 17, no. 1, pp. 50–63, 2014.
  37. G. Zhai, X. Wu, X. Yang, W. Lin, and W. Zhang, “A psychovisual quality metric in free-energy principle,” IEEE Transactions on Image Processing, vol. 21, no. 1, pp. 41–52, 2011.
  38. W. Xue, X. Mou, L. Zhang, A. C. Bovik, and X. Feng, “Blind image quality assessment using joint statistics of gradient magnitude and laplacian features,” IEEE Transactions on Image Processing, vol. 23, no. 11, pp. 4850–4862, 2014.
  39. P. Ye, J. Kumar, L. Kang, and D. Doermann, “Unsupervised feature learning framework for no-reference image quality assessment,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 1098–1105.
  40. P. Ye and D. Doermann, “No-reference image quality assessment using visual codebooks,” IEEE Transactions on Image Processing, vol. 21, no. 7, pp. 3129–3138, 2012.
  41. P. Zhang, W. Zhou, L. Wu, and H. Li, “SOM: Semantic obviousness metric for image quality assessment,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 2394–2402.
  42. L. Kang, P. Ye, Y. Li, and D. Doermann, “Convolutional neural networks for no-reference image quality assessment,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1733–1740.
  43. S. Bianco, L. Celona, P. Napoletano, and R. Schettini, “On the use of deep learning for blind image quality assessment,” Signal, Image and Video Processing, vol. 12, no. 2, pp. 355–362, 2018.
  44. L. Kang, P. Ye, Y. Li, and D. Doermann, “Simultaneous estimation of image quality and distortion via multi-task convolutional neural networks,” in IEEE International Conference on Image Processing (ICIP), 2015, pp. 2791–2795.
  45. X. Liu, J. van de Weijer, and A. D. Bagdanov, “RankIQA: Learning from rankings for no-reference image quality assessment,” in IEEE International Conference on Computer Vision (ICCV), 2017, pp. 1040–1049.
  46. Y. Niu, D. Huang, Y. Shi, and X. Ke, “Siamese-network-based learning to rank for no-reference 2D and 3D image quality assessment,” IEEE Access, vol. 7, pp. 101 583–101 595, 2019.
  47. Z. Ying, D. Pan, and P. Shi, “Quality difference ranking model for smartphone camera photo quality assessment,” in International Conference on Multimedia & Expo Workshops (ICMEW), 2020, pp. 1–6.
  48. W. Zhang, K. Ma, J. Yan, D. Deng, and Z. Wang, “Blind image quality assessment using a deep bilinear convolutional neural network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 1, pp. 36–47, 2018.
  49. K. Ma, W. Liu, K. Zhang, Z. Duanmu, Z. Wang, and W. Zuo, “End-to-end blind image quality assessment using deep neural networks,” IEEE Transactions on Image Processing, vol. 27, no. 3, pp. 1202–1213, 2017.
  50. Y. Fang, H. Zhu, Y. Zeng, K. Ma, and Z. Wang, “Perceptual quality assessment of smartphone photography,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 3677–3686.
  51. R. Ma, Q. Wu, K. N. Ngan, H. Li, F. Meng, and L. Xu, “Forgetting to remember: A scalable incremental learning framework for cross-task blind image quality assessment,” IEEE Transactions on Multimedia, 2023.
  52. R. Ma, H. Luo, Q. Wu, K. N. Ngan, H. Li, F. Meng, and L. Xu, “Remember and reuse: Cross-task blind image quality assessment via relevance-aware incremental learning,” in ACM International Conference on Multimedia (ACM MM), 2021, pp. 5248–5256.
  53. P. Chen, L. Li, Q. Wu, and J. Wu, “SPIQ: A self-supervised pre-trained model for image quality assessment,” IEEE Signal Processing Letters, vol. 29, pp. 513–517, 2022.
  54. G. Yue, D. Cheng, L. Li, T. Zhou, H. Liu, and T. Wang, “Semi-supervised authentically distorted image quality assessment with consistency-preserving dual-branch convolutional neural network,” IEEE Transactions on Multimedia, vol. 25, pp. 6499–6511, 2022.
  55. K.-Y. Lin and G. Wang, “Hallucinated-IQA: No-reference image quality assessment via adversarial learning,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 732–741.
  56. B. Chen, L. Zhu, C. Kong, H. Zhu, S. Wang, and Z. Li, “No-reference image quality assessment by hallucinating pristine features,” IEEE Transactions on Image Processing, vol. 31, pp. 6139–6151, 2022.
  57. G. Yue, C. Hou, W. Yan, L. K. Choi, T. Zhou, and Y. Hou, “Blind quality assessment for screen content images via convolutional neural network,” Digital Signal Processing, vol. 91, pp. 21–30, 2019.
  58. X. Jiang, L. Shen, L. Yu, M. Jiang, and G. Feng, “No-reference screen content image quality assessment based on multi-region features,” Neurocomputing, vol. 386, pp. 30–41, 2020.
  59. X. Min, K. Gu, G. Zhai, M. Hu, and X. Yang, “Saliency-induced reduced-reference quality index for natural scene and screen content images,” Signal Processing, vol. 145, pp. 127–136, 2018.
  60. X. Min, K. Ma, K. Gu, G. Zhai, Z. Wang, and W. Lin, “Unified blind quality assessment of compressed natural, graphic, and screen content images,” IEEE Transactions on Image Processing, vol. 26, no. 11, pp. 5462–5474, 2017.
  61. D. Li, T. Jiang, and M. Jiang, “Exploiting high-level semantics for no-reference image quality assessment of realistic blur images,” in ACM International Conference on Multimedia (ACM MM), 2017, pp. 378–386.
  62. D. Li, T. Jiang, W. Lin, and M. Jiang, “Which has better visual quality: The clear blue sky or a blurry animal?” IEEE Transactions on Multimedia, vol. 21, no. 5, pp. 1221–1234, 2018.
  63. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 586–595.
  64. S. Bosse, D. Maniry, K.-R. Müller, T. Wiegand, and W. Samek, “Deep neural networks for no-reference and full-reference image quality assessment,” IEEE Transactions on Image Processing, vol. 27, no. 1, pp. 206–219, 2017.
  65. K. Ding, K. Ma, S. Wang, and E. P. Simoncelli, “Image quality assessment: Unifying structure and texture similarity,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 5, pp. 2567–2581, 2020.
  66. K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1904–1916, 2015.
  67. J. Kim and S. Lee, “Fully deep blind image quality predictor,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 1, pp. 206–220, 2016.
  68. B. Chen, L. Zhu, G. Li, F. Lu, H. Fan, and S. Wang, “Learning generalized spatial-temporal deep feature representation for no-reference video quality assessment,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 4, pp. 1903–1916, 2021.
  69. H. Zhu, B. Chen, L. Zhu, and S. Wang, “Learning spatiotemporal interactions for user-generated video quality assessment,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 3, pp. 1031–1042, 2022.
  70. A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola, “A kernel two-sample test,” Journal of Machine Learning Research, vol. 13, no. Mar, pp. 723–773, 2012.
  71. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “PyTorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems, 2019, pp. 8026–8037.
  72. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  73. L. Zhang, L. Zhang, and A. C. Bovik, “A feature-enriched completely blind image quality evaluator,” IEEE Transactions on Image Processing, vol. 24, no. 8, pp. 2579–2591, 2015.
  74. J. Xu, P. Ye, Q. Li, H. Du, Y. Liu, and D. Doermann, “Blind image quality assessment based on high order statistics aggregation,” IEEE Transactions on Image Processing, vol. 25, no. 9, pp. 4444–4457, 2016.
  75. K. Gu, G. Zhai, W. Lin, X. Yang, and W. Zhang, “Learning a blind quality evaluation engine of screen content images,” Neurocomputing, vol. 196, pp. 140–149, 2016.
  76. Y. Fang, R. Du, Y. Zuo, W. Wen, and L. Li, “Perceptual quality assessment for screen content images by spatial continuity,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 11, pp. 4050–4063, 2019.
  77. S. Sun, T. Yu, J. Xu, W. Zhou, and Z. Chen, “GraphIQA: Learning distortion graph representations for blind image quality assessment,” IEEE Transactions on Multimedia, 2022.
  78. Z. Pan, F. Yuan, J. Lei, Y. Fang, X. Shao, and S. Kwong, “VCRNet: Visual compensation restoration network for no-reference image quality assessment,” IEEE Transactions on Image Processing, vol. 31, pp. 1613–1627, 2022.
  79. W. Xue, L. Zhang, and X. Mou, “Learning without human scores for blind image quality assessment,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 995–1002.
  80. J. Wu, Z. Xia, H. Zhang, and H. Li, “Blind quality assessment for screen content images by combining local and global features,” Digital Signal Processing, vol. 91, pp. 31–40, 2019.
  81. R. Li, H. Yang, T. Yu, and Z. Pan, “CNN model for screen content image quality assessment based on region difference,” in International Conference on Signal and Image Processing (ICSIP), 2019, pp. 1010–1014.
  82. J. Yang, Y. Zhao, J. Liu, B. Jiang, Q. Meng, W. Lu, and X. Gao, “No reference quality assessment for screen content images using stacked autoencoders in pictorial and textual regions,” IEEE Transactions on Cybernetics, vol. 52, no. 5, pp. 2798–2810, 2020.
  83. J. Yang, Z. Bian, J. Liu, B. Jiang, W. Lu, X. Gao, and H. Song, “No-reference quality assessment for screen content images using visual edge model and adaboosting neural network,” IEEE Transactions on Image Processing, vol. 30, pp. 6801–6814, 2021.
  84. H. Lin, V. Hosu, and D. Saupe, “KADID-10k: A large-scale artificially distorted IQA database,” in International Conference on Quality of Multimedia Experience (QoMEX), 2019.
  85. K. Ma, Z. Duanmu, Q. Wu, Z. Wang, H. Yong, H. Li, and L. Zhang, “Waterloo exploration database: New challenges for image quality assessment models,” IEEE Transactions on Image Processing, vol. 26, no. 2, pp. 1004–1016, 2016.
  86. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255.
  87. L. v. d. Maaten and G. Hinton, “Visualizing data using t-SNE,” Journal of Machine Learning Research, vol. 9, no. Nov, pp. 2579–2605, 2008.
  88. M. A. Saad, A. C. Bovik, and C. Charrier, “Blind image quality assessment: A natural scene statistics approach in the dct domain,” IEEE Transactions on Image Processing, vol. 21, no. 8, pp. 3339–3352, 2012.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now