LAA-Net: Localized Artifact Attention Network for Quality-Agnostic and Generalizable Deepfake Detection (2401.13856v2)
Abstract: This paper introduces a novel approach for high-quality deepfake detection called Localized Artifact Attention Network (LAA-Net). Existing methods for high-quality deepfake detection are mainly based on a supervised binary classifier coupled with an implicit attention mechanism. As a result, they do not generalize well to unseen manipulations. To handle this issue, two main contributions are made. First, an explicit attention mechanism within a multi-task learning framework is proposed. By combining heatmap-based and self-consistency attention strategies, LAA-Net is forced to focus on a few small artifact-prone vulnerable regions. Second, an Enhanced Feature Pyramid Network (E-FPN) is proposed as a simple and effective mechanism for spreading discriminative low-level features into the final feature output, with the advantage of limiting redundancy. Experiments performed on several benchmarks show the superiority of our approach in terms of Area Under the Curve (AUC) and Average Precision (AP). The code is available at https://github.com/10Ring/LAA-Net.
- Mesonet: a compact facial video forgery detection network. CoRR, abs/1809.00888, 2018.
- Regularizing deep neural networks by enhancing diversity in feature extraction. IEEE transactions on neural networks and learning systems, 30(9):2650–2661, 2019.
- Aunet: Learning relations between action units for face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24709–24719, 2023.
- Sarah Cahlan. How misinformation helped spark an attempted coup in Gabon. https://wapo.st/3KZARDF, 2020. [Online; accessed 7-March-2023].
- Marlin: Masked autoencoder for facial video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1493–1504, 2023.
- End-to-end reconstruction-classification learning for face forgery detection. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4103–4112, 2022.
- Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection, 2022.
- Local relation learning for face forgery detection. In AAAI Conference on Artificial Intelligence, 2021.
- François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017.
- Combining efficientnet and vision transformers for video deepfake detection. CoRR, abs/2107.02612, 2021.
- Deepfakes. Faceswapdevs. https://github.com/deepfakes/faceswap, 2019.
- Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
- The deepfake detection challenge (DFDC) preview dataset. CoRR, abs/1910.08854, 2019.
- Implicit identity leakage: The stumbling block to improving deepfake detection generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3994–4004, 2023.
- Contributing data to deepfake detection research. https://ai.googleblog.com/2019/09/contributing-data-to-deepfake-detection.html, 2019.
- Sharpness-aware minimization for efficiently improving generalization. CoRR, abs/2010.01412, 2020.
- Controllable guide-space for generalizable face forgery detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20818–20827, 2023.
- Lips don’t lie: A generalisable and robust approach to face forgery detection. CoRR, abs/2012.07657, 2020.
- Leveraging real talking faces via self-supervision for robust forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14950–14962, 2022.
- Implicit identity driven deepfake face swapping detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4490–4499, 2023.
- Davis E. King. Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res., 10:1755–1758, 2009.
- Marek Kowalski. Faceswap. https://github.com/MarekKowalski/FaceSwap, 2018.
- Cornernet: Detecting objects as paired keypoints. International Journal of Computer Vision, 128:642–656, 2018.
- Face x-ray for more general face forgery detection. CoRR, abs/1912.13458, 2019a.
- Celeb-df: A new dataset for deepfake forensics. CoRR, abs/1909.12962, 2019b.
- Focal loss for dense object detection. CoRR, abs/1708.02002, 2017a.
- Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017b.
- Ti2net: Temporal identity inconsistency network for deepfake detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4691–4700, 2023.
- Pose guided person image generation. Advances in neural information processing systems, 30, 2017.
- Zero-shot noise2noise: Efficient image denoising without any data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14018–14027, 2023.
- Exploiting visual artifacts to expose deepfakes and face manipulations. In 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), pages 83–92, 2019.
- Leveraging high-frequency components for deepfake detection. In 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP), pages 1–6, 2021.
- Untag: Learning generic features for unsupervised type-agnostic deepfake detection. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5, 2023.
- When does label smoothing help? CoRR, abs/1906.02629, 2019.
- Capsule-forensics: Using capsule networks to detect forged images and videos. CoRR, abs/1810.11215, 2018.
- Deep learning for deepfakes creation and detection. CoRR, abs/1909.11573, 2019.
- FaceForensics++: Learning to detect manipulated facial images. In International Conference on Computer Vision (ICCV), 2019.
- Feature pyramid network for multi-class land segmentation. CoRR, abs/1806.03510, 2018.
- Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. CoRR, abs/1610.02391, 2016.
- Structure aggregation for cross-spectral stereo image guided denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13997–14006, 2023.
- Detecting deepfakes with self-blended images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18720–18729, 2022.
- Multi-label deepfake classification. IEEE Workshop on Multimedia Signal Processing, 2023.
- Improving the efficiency and robustness of deepfakes detection through precise geometric features. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3608–3617, 2021.
- Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR, abs/1905.11946, 2019.
- Deferred neural rendering: Image synthesis using neural textures. CoRR, abs/1904.12356, 2019.
- Face2face: Real-time face capture and reenactment of RGB videos. CoRR, abs/2007.14808, 2020.
- FCOS: fully convolutional one-stage object detection. CoRR, abs/1904.01355, 2019.
- Jane Wakefield. Deepfake presidents used in Russia-Ukraine war. https://www.bbc.com/news/technology-60780142, 2022. [Online; accessed 7-March-2023].
- Fakespotter: A simple baseline for spotting ai-synthesized fake faces. CoRR, abs/1909.06122, 2019.
- Dynamic graph learning with content-guided spatial-frequency relation reasoning for deepfake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7278–7287, 2023a.
- Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612, 2004.
- Altfreezing for more general video face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4129–4138, 2023b.
- Ucf: Uncovering common features for generalizable deepfake detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22412–22423, 2023.
- Learning self-consistency for deepfake detection. In ICCV 2021, 2021a.
- Multi-attentional deepfake detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2185–2194, 2021b.
- Random erasing data augmentation. CoRR, abs/1708.04896, 2017.
- Wilddeepfake: A challenging real-world dataset for deepfake detection. Proceedings of the 28th ACM International Conference on Multimedia, 2020.