Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Hierarchical Forgery Classifier On Multi-modality Face Forgery Clues (2212.14629v2)

Published 30 Dec 2022 in cs.CV

Abstract: Face forgery detection plays an important role in personal privacy and social security. With the development of adversarial generative models, high-quality forgery images become more and more indistinguishable from real to humans. Existing methods always regard as forgery detection task as the common binary or multi-label classification, and ignore exploring diverse multi-modality forgery image types, e.g. visible light spectrum and near-infrared scenarios. In this paper, we propose a novel Hierarchical Forgery Classifier for Multi-modality Face Forgery Detection (HFC-MFFD), which could effectively learn robust patches-based hybrid domain representation to enhance forgery authentication in multiple-modality scenarios. The local spatial hybrid domain feature module is designed to explore strong discriminative forgery clues both in the image and frequency domain in local distinct face regions. Furthermore, the specific hierarchical face forgery classifier is proposed to alleviate the class imbalance problem and further boost detection performance. Experimental results on representative multi-modality face forgery datasets demonstrate the superior performance of the proposed HFC-MFFD compared with state-of-the-art algorithms. The source code and models are publicly available at https://github.com/EdWhites/HFC-MFFD.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (77)
  1. D. Güera and E. J. Delp, “Deepfake video detection using recurrent neural networks,” in 2018 15th IEEE international conference on advanced video and signal based surveillance (AVSS).   IEEE, 2018, pp. 1–6.
  2. Y. Li, X. Yang, P. Sun, H. Qi, and S. Lyu, “Celeb-df: A large-scale challenging dataset for deepfake forensics,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 3207–3216.
  3. S. Karaoğlu, T. Gevers et al., “Self-supervised face image manipulation by conditioning gan on face decomposition,” IEEE Transactions on Multimedia, vol. 24, pp. 377–385, 2021.
  4. J. Song, J. Zhang, L. Gao, Z. Zhao, and H. T. Shen, “Agegan++: Face aging and rejuvenation with dual conditional gans,” IEEE Transactions on Multimedia, vol. 24, pp. 791–804, 2021.
  5. Y. He, B. Gan, S. Chen, Y. Zhou, G. Yin, L. Song, L. Sheng, J. Shao, and Z. Liu, “Forgerynet: A versatile benchmark for comprehensive forgery analysis,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 4360–4369.
  6. Y. Wang, C. Peng, D. Liu, N. Wang, and X. Gao, “Forgerynir: Deep face forgery and detection in near-infrared scenario,” IEEE Transactions on Information Forensics and Security, vol. 17, pp. 500–515, 2022.
  7. F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1251–1258.
  8. N. Yu, L. S. Davis, and M. Fritz, “Attributing fake images to gans: Learning and analyzing gan fingerprints,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 7556–7566.
  9. Q. Huang, J. Zhang, W. Zhou, W. Zhang, and N. Yu, “Initiative defense against facial manipulation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 2, 2021, pp. 1619–1627.
  10. J. Frank, T. Eisenhofer, L. Schönherr, A. Fischer, D. Kolossa, and T. Holz, “Leveraging frequency analysis for deep fake image recognition,” in International conference on machine learning.   PMLR, 2020, pp. 3247–3258.
  11. Y. Qian, G. Yin, L. Sheng, Z. Chen, and J. Shao, “Thinking in frequency: Face forgery detection by mining frequency-aware clues,” in European conference on computer vision.   Springer, 2020, pp. 86–103.
  12. H. Liu, X. Li, W. Zhou, Y. Chen, Y. He, H. Xue, W. Zhang, and N. Yu, “Spatial-phase shallow learning: rethinking face forgery detection in frequency domain,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 772–781.
  13. J. Li, H. Xie, J. Li, Z. Wang, and Y. Zhang, “Frequency-aware discriminative feature learning supervised by single-center loss for face forgery detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 6458–6467.
  14. D. Liu, J. Li, N. Wang, C. Peng, and X. Gao, “Composite components-based face sketch recognition,” Neurocomputing, vol. 302, pp. 46–54, 2018.
  15. D. Liu, X. Gao, N. Wang, C. Peng, and J. Li, “Iterative local re-ranking with attribute guided synthesis for face sketch recognition,” Pattern Recognition, vol. 109, 2021.
  16. L. Chai, D. Bau, S.-N. Lim, and P. Isola, “What makes fake images detectable? understanding properties that generalize,” in European conference on computer vision.   Springer, 2020, pp. 103–120.
  17. S. Chen, T. Yao, Y. Chen, S. Ding, J. Li, and R. Ji, “Local relation learning for face forgery detection,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 2, 2021, pp. 1081–1088.
  18. T. Ojala, M. Pietikainen, and T. Maenpaa, “Multiresolution gray-scale and rotation invariant texture classification with local binary patterns,” IEEE Transactions on pattern analysis and machine intelligence, vol. 24, no. 7, pp. 971–987, 2002.
  19. D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International journal of computer vision, vol. 60, no. 2, pp. 91–110, 2004.
  20. N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol. 1.   Ieee, 2005, pp. 886–893.
  21. C. Cortes and V. Vapnik, “Support-vector networks,” Machine learning, vol. 20, no. 3, pp. 273–297, 1995.
  22. J. Platt et al., “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” Advances in large margin classifiers, vol. 10, no. 3, pp. 61–74, 1999.
  23. T.-F. Wu, C.-J. Lin, and R. Weng, “Probability estimates for multi-class classification by pairwise coupling,” Advances in Neural Information Processing Systems, vol. 16, 2003.
  24. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  25. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  26. H. Mo, B. Chen, and W. Luo, “Fake faces identification via convolutional neural network,” in Proceedings of the 6th ACM workshop on information hiding and multimedia security, 2018, pp. 43–47.
  27. S.-Y. Wang, O. Wang, R. Zhang, A. Owens, and A. A. Efros, “Cnn-generated images are surprisingly easy to spot… for now,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 8695–8704.
  28. D. Liu, Z. Dang, C. Peng, Y. Zheng, S. Li, N. Wang, and X. Gao, “Fedforgery: generalized face forgery detection with residual federated learning,” IEEE Transactions on Information Forensics and Security, 2023.
  29. H. Li, S. Wang, P. He, and A. Rocha, “Face anti-spoofing with deep neural network distillation,” IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 5, pp. 933–946, 2020.
  30. Z. Mi, X. Jiang, T. Sun, and K. Xu, “Gan-generated image detection with self-attention mechanism against gan generator defect,” IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 5, pp. 969–981, 2020.
  31. Y. Fei, C. Huang, C. Jinkun, M. Li, Y. Zhang, and C. Lu, “Attribute restoration framework for anomaly detection,” IEEE Transactions on Multimedia, 2020.
  32. Y. Chen, Z. Wang, Z. J. Wang, and X. Kang, “Automated design of neural network architectures with reinforcement learning for detection of global manipulations,” IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 5, pp. 997–1011, 2020.
  33. L. Su, C. Li, Y. Lai, and J. Yang, “A fast forgery detection algorithm based on exponential-fourier moments for video region duplication,” IEEE Transactions on Multimedia, vol. 20, no. 4, pp. 825–840, 2017.
  34. S. Chen, S. Tan, B. Li, and J. Huang, “Automatic detection of object-based forgery in advanced video,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 26, no. 11, pp. 2138–2151, 2015.
  35. C. M. LIY and L. InIctuOculi, “Exposingaicreated fakevideosbydetectingeyeblinking,” in 2018IEEEInterG national Workshop on Information Forensics and Security (WIFS). IEEE, 2018.
  36. C. Peng, W. Zhang, D. Liu, N. Wang, and X. Gao, “Temporal consistency based deep face forgery detection network,” in International Conference on Machine Learning for Cyber Security.   Springer, 2020, pp. 55–63.
  37. A. Chintha, B. Thai, S. J. Sohrawardi, K. Bhatt, A. Hickerson, M. Wright, and R. Ptucha, “Recurrent convolutional structures for audio spoof and video deepfake detection,” IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 5, pp. 1024–1037, 2020.
  38. P. Yang, D. Baracchi, M. Iuliani, D. Shullani, R. Ni, Y. Zhao, and A. Piva, “Efficient video integrity analysis through container characterization,” IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 5, pp. 947–954, 2020.
  39. L. D’Amiano, D. Cozzolino, G. Poggi, and L. Verdoliva, “A patchmatch-based dense-field algorithm for video copy–move detection and localization,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 3, pp. 669–682, 2018.
  40. M. Aloraini, M. Sharifzadeh, and D. Schonfeld, “Sequential and patch analyses for object removal video forgery detection and localization,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 3, pp. 917–930, 2020.
  41. J. Hu, X. Liao, W. Wang, and Z. Qin, “Detecting compressed deepfake videos in social networks using frame-temporality two-stream convolutional network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 3, pp. 1089–1102, 2021.
  42. P. He, H. Li, B. Li, H. Wang, and L. Liu, “Exposing fake bitrate videos using hybrid deep-learning network from recompression error,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 11, pp. 4034–4049, 2019.
  43. L. Zhang, T. Qiao, M. Xu, N. Zheng, and S. Xie, “Unsupervised learning-based framework for deepfake video detection,” IEEE Transactions on Multimedia, 2022.
  44. L. Song, X. Li, Z. Fang, Z. Jin, Y. Chen, and C. Xu, “Face forgery detection via symmetric transformer,” in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 4102–4111.
  45. X. Zhang, S. Karaman, and S.-F. Chang, “Detecting and simulating artifacts in gan fake images,” in 2019 IEEE international workshop on information forensics and security (WIFS).   IEEE, 2019, pp. 1–6.
  46. T. Dzanic, K. Shah, and F. Witherden, “Fourier spectrum discrepancies in deep network generated images,” Advances in neural information processing systems, vol. 33, pp. 3022–3032, 2020.
  47. K. Chandrasegaran, N.-T. Tran, and N.-M. Cheung, “A closer look at fourier spectrum discrepancies for cnn-generated images detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 7200–7209.
  48. O. Giudice, L. Guarnera, and S. Battiato, “Fighting deepfakes by detecting gan dct anomalies,” Journal of Imaging, vol. 7, no. 8, p. 128, 2021.
  49. L. Song, Z. Fang, X. Li, X. Dong, Z. Jin, Y. Chen, and S. Lyu, “Adaptive face forgery detection in cross domain,” in Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIV.   Springer, 2022, pp. 467–484.
  50. J. Wang, Z. Wu, W. Ouyang, X. Han, J. Chen, Y.-G. Jiang, and S.-N. Li, “M2tr: Multi-modal multi-scale transformers for deepfake detection,” in Proceedings of the 2022 International Conference on Multimedia Retrieval, 2022, pp. 615–623.
  51. X. Tang and X. Wang, “Face sketch synthesis and recognition,” in Proceedings of IEEE International Conference on Computer Vision, 2003, pp. 687–694.
  52. Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma, “A nonlinear approach for face sketch synthesis and recognition,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2005, pp. 1005–1010.
  53. M. Yang, L. Zhang, X. Feng, and D. Zhang, “Sparse representation based fisher discrimination dictionary learning for image classification,” International Journal of Computer Vision, vol. 109, no. 3, pp. 209––232, 2014.
  54. L. Zhang, L. Lin, X. Wu, S. Ding, and L. Zhang, “End-to-end photo-sketch generation via fully convolutional representation learning,” in Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, 2015, pp. 627–634.
  55. M. Kan, S. Shan, H. Zhang, S. Lao, and X. Chen, “Multi-view discriminant analysis,” in Proc. Eur. Conf. Comput. Vis., 2012, pp. 808–821.
  56. A. Sharma and D. W. Jacobs, “Bypassing synthesis: Pls for face recognition with pose, low-resolution and sketch,” in CVPR 2011.   IEEE, 2011, pp. 593–600.
  57. J. Huo, Y. Gao, Y. Shi, W. Yang, and H. Yin, “Heterogeneous face recognition by margin-based cross-modality metric learning,” IEEE Trans. Cybernetics, 2017.
  58. D. Liu, X. Gao, N. Wang, J. Li, and C. Peng, “Coupled attribute learning for heterogeneous face recognition,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 11, pp. 4699–4712, 2020.
  59. W. Hu and H. Hu, “Adversarial disentanglement spectrum variations and cross-modality attention networks for nir-vis face recognition,” IEEE Transactions on Multimedia, vol. 23, pp. 145–160, 2020.
  60. D. Liu, X. Gao, C. Peng, N. Wang, and J. Li, “Heterogeneous face interpretable disentangled representation for joint face recognition and synthesis,” IEEE transactions on neural networks and learning systems, vol. 33, no. 10, pp. 5611–5625, 2021.
  61. Z. Lei, D. Yi, and S. Li, “Discriminant image filter learning for face recognition with local binary pattern like representation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2512–2517.
  62. A. Alex, V. Asari, and A. Mathew, “Local difference of gaussian binary pattern: robust features for face sketch recognition,” in Proc. IEEE Int. Conf. Syst. Man &\&& Cybern., 2013, pp. 1211–1216.
  63. H. Han, B. Klare, K. Bonnen, and A. Jain, “Matching composite sketches to face photos: a component-based approach,” IEEE Trans. Inf. Forens. Security, vol. 8, no. 1, pp. 191–204, Jan. 2013.
  64. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  65. H. Zhao, W. Zhou, D. Chen, T. Wei, W. Zhang, and N. Yu, “Multi-attentional deepfake detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 2185–2194.
  66. H. Zhao, W. Zhou, D. Chen, W. Zhang, and N. Yu, “Self-supervised transformer for deepfake detection,” arXiv preprint arXiv:2203.01265, 2022.
  67. P. Wang, K. Liu, W. Zhou, H. Zhou, H. Liu, W. Zhang, and N. Yu, “Adt: Anti-deepfake transformer,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2022, pp. 2899–1903.
  68. S. Li, D. Yi, Z. Lei, and S. Liao, “The casia nir-vis 2.0 face database,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2013, pp. 348–353.
  69. B. Zi, M. Chang, J. Chen, X. Ma, and Y.-G. Jiang, “Wilddeepfake: A challenging real-world dataset for deepfake detection,” in Proceedings of the 28th ACM international conference on multimedia, 2020, pp. 2382–2390.
  70. M. Wolter, F. Blanke, C. T. Hoyt, and J. Garcke, “Wavelet-packet powered deepfake image detection,” arXiv preprint arXiv:2106.09369, 2021.
  71. J. Cao, C. Ma, T. Yao, S. Chen, S. Ding, and X. Yang, “End-to-end reconstruction-classification learning for face forgery detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4113–4122.
  72. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, 2012.
  73. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.
  74. D. Afchar, V. Nozick, J. Yamagishi, and I. Echizen, “Mesonet: a compact facial video forgery detection network,” in 2018 IEEE international workshop on information forensics and security (WIFS).   IEEE, 2018, pp. 1–7.
  75. H. H. Nguyen, J. Yamagishi, and I. Echizen, “Capsule-forensics: Using capsule networks to detect forged images and videos,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2019, pp. 2307–2311.
  76. A. Khormali and J.-S. Yuan, “Add: Attention-based deepfake detection approach,” Big Data and Cognitive Computing, vol. 5, no. 4, p. 49, 2021.
  77. C. Wang and W. Deng, “Representative forgery mining for fake face detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14 923–14 932.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Decheng Liu (22 papers)
  2. Zeyang Zheng (3 papers)
  3. Chunlei Peng (20 papers)
  4. Yukai Wang (10 papers)
  5. Nannan Wang (106 papers)
  6. Xinbo Gao (194 papers)
Citations (11)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - EdWhites/HFC-MFFD (6 stars)