Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 71 tok/s
Gemini 2.5 Pro 48 tok/s Pro
GPT-5 Medium 23 tok/s Pro
GPT-5 High 17 tok/s Pro
GPT-4o 111 tok/s Pro
Kimi K2 161 tok/s Pro
GPT OSS 120B 412 tok/s Pro
Claude Sonnet 4 35 tok/s Pro
2000 character limit reached

Comparing the Robustness of Modern No-Reference Image- and Video-Quality Metrics to Adversarial Attacks (2310.06958v4)

Published 10 Oct 2023 in cs.CV, cs.LG, cs.MM, and eess.IV

Abstract: Nowadays, neural-network-based image- and video-quality metrics perform better than traditional methods. However, they also became more vulnerable to adversarial attacks that increase metrics' scores without improving visual quality. The existing benchmarks of quality metrics compare their performance in terms of correlation with subjective quality and calculation time. Nonetheless, the adversarial robustness of image-quality metrics is also an area worth researching. This paper analyses modern metrics' robustness to different adversarial attacks. We adapted adversarial attacks from computer vision tasks and compared attacks' efficiency against 15 no-reference image- and video-quality metrics. Some metrics showed high resistance to adversarial attacks, which makes their usage in benchmarks safer than vulnerable metrics. The benchmark accepts submissions of new metrics for researchers who want to make their metrics more robust to attacks or to find such metrics for their needs. The latest results can be found online: https://videoprocessing.ai/benchmarks/metrics-robustness.html.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. 2001. Xiph.org Video Test Media [derf’s collection]. https://media.xiph.org/video/derf/.
  2. 2017. NIPS 2017: Adversarial Learning Development Set. https://www.kaggle.com/datasets/google-brain/nips-2017-adversarial-learning-development-set.
  3. Comparing the robustness of modern no-reference image- and video-quality metrics to adversarial attacks. arXiv:2310.06958.
  4. Video compression dataset and benchmark of learning-based video-quality metrics. In Advances in Neural Information Processing Systems, volume 35, 13814–13825.
  5. Bing, M. 2013. A Behind the Scenes Look at How Bing is Improving Image Search Quality. https://blogs.bing.com/search-quality-insights/2013/08/23/a-behind-the-scenes-look-at-how-bing-is-improving-image-search-quality.
  6. Accuracy and cross-calibration of video quality metrics: new methods from ATIS/T1A1. Signal Processing: Image Communication, 19(2): 101–107.
  7. No-Reference Image Quality Assessment by Hallucinating Pristine Features. IEEE Transactions on Image Processing, 31: 6139–6151.
  8. Supplemental subjective testing to evaluate the performance of image and video quality estimators. In Human Vision and Electronic Imaging XVI, volume 7865, 249–257. SPIE.
  9. Systematic stress testing of image quality estimators. In 2011 18th IEEE International Conference on Image Processing, 3101–3104. IEEE.
  10. Comparison, M. V. C. 2021. MSU Video Codecs Comparison 2021 Part 2: Subjective. http://www.compression.ru/video/codec˙comparison/2021/subjective˙report.html.
  11. Vmaf based rate-distortion optimization for video coding. In 2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP), 1–6. IEEE.
  12. Comparison of full-reference image quality models for optimization of image processing systems. International Journal of Computer Vision, 129: 1258–1281.
  13. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, 9185–9193.
  14. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.
  15. Perceptual quality assessment of smartphone photography. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3677–3686.
  16. Attacking Perceptual Similarity Metrics. arXiv preprint arXiv:2305.08840.
  17. No-reference image quality assessment via transformers, relative ranking, and self-consistency. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 1220–1230.
  18. Explaining and Harnessing Adversarial Examples. In Bengio, Y.; and LeCun, Y., eds., 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  19. NTIRE 2022 challenge on perceptual image quality assessment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 951–967.
  20. KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions on Image Processing, 29: 4041–4056.
  21. Kantorovich, L. V. 1960. Mathematical Methods of Organizing and Planning Production. Management Science, 6(4): 366–422.
  22. E-lpips: robust perceptual image similarity via random transformation ensembles. arXiv preprint arXiv:1906.03973.
  23. Klebanov, L. 2005. N-Distances and their Applications.
  24. Adversarial Attacks Against Blind Image Quality Assessment Models. In Proceedings of the 2nd Workshop on Quality of Experience in Visual Multimedia Applications, 3–11.
  25. Neural Optimal Transport. In International Conference on Learning Representations.
  26. Adversarial examples in the physical world. In Artificial intelligence safety and security, 99–112. Chapman and Hall/CRC.
  27. Quality assessment of in-the-wild videos. In Proceedings of the 27th ACM International Conference on Multimedia, 2351–2359.
  28. Norm-in-norm loss with faster convergence and better performance for image quality assessment. In Proceedings of the 28th ACM International Conference on Multimedia, 789–797.
  29. Unified quality assessment of in-the-wild videos with mixed datasets training. International Journal of Computer Vision, 129: 1238–1257.
  30. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, 740–755. Springer.
  31. Software to stress test image quality estimators. In 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX), 1–6. IEEE.
  32. Rankiqa: Learning from rankings for no-reference image quality assessment. In Proceedings of the IEEE international conference on computer vision, 1040–1049.
  33. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083.
  34. MediaEval. 2020. Pixel Privacy: Quality Camouflage for Social Images. https://multimediaeval.github.io/editions/2020/tasks/pixelprivacy/.
  35. Making a “completely blind” image quality analyzer. IEEE Signal processing letters, 20(3): 209–212.
  36. On the Generation of Adversarial Samples for Image Quality Assessment. Available at SSRN 4112969.
  37. Universal Perturbation Attack on Differentiable No-Reference Image- and Video-Quality Metrics. In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022. BMVA Press.
  38. Fast Adversarial CNN-based Perturbation Attack of No-Reference Image Quality Metrics.
  39. Hacking VMAF and VMAF NEG: vulnerability to different preprocessing methods. In 2021 4th Artificial Intelligence and Cloud Computing Conference, 89–96.
  40. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, 23(5): 828–841.
  41. Blindly assess image quality in the wild guided by a self-adaptive hyper network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3667–3676.
  42. Blind natural image quality prediction using convolutional neural networks and weighted spatial pooling. In 2020 IEEE International Conference on Image Processing (ICIP), 191–195. IEEE.
  43. NIMA: Neural image assessment. IEEE transactions on image processing, 27(8): 3998–4011.
  44. Video Quality Assessment of User Generated Content: A Benchmark Study and a New Model. In 2021 IEEE International Conference on Image Processing (ICIP), 1409–1413. IEEE.
  45. V-Nova. 2023. FFmpeg with LCEVC. https://docs.v-nova.com/.
  46. Exploring CLIP for Assessing the Look and Feel of Images. In AAAI.
  47. Image Quality Assessment: From Error Visibility to Structural Similarity. Image Processing, IEEE Transactions on, 13: 600 – 612.
  48. Maximum differentiation (MAD) competition: A methodology for comparing computational models of perceptual quantities. Journal of Vision, 8(12): 8–8.
  49. Video Enhancement with Task-Oriented Flow. International Journal of Computer Vision (IJCV), 127(8): 1106–1125.
  50. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1191–1200.
  51. From patches to pictures (PaQ-2-PiQ): Mapping the perceptual space of picture quality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3575–3585.
  52. Benchmarking ultra-high-definition image super-resolution. In Proceedings of the IEEE/CVF international conference on computer vision, 14769–14778.
  53. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, 586–595.
  54. Perceptual Attacks of No-Reference Image Quality Models with Human-in-the-Loop. arXiv preprint arXiv:2210.00933.
  55. MetaIQA: Deep meta-learning for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14143–14152.
  56. Hacking VMAF with video color and contrast distortion. In CEUR Workshop Proceedings, 53–57.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up