Exploring Temporal Coherence for More General Video Face Forgery Detection

Abstract: Although current face manipulation techniques achieve impressive performance regarding quality and controllability, they are struggling to generate temporal coherent face videos. In this work, we explore to take full advantage of the temporal coherence for video face forgery detection. To achieve this, we propose a novel end-to-end framework, which consists of two major stages. The first stage is a fully temporal convolution network (FTCN). The key insight of FTCN is to reduce the spatial convolution kernel size to 1, while maintaining the temporal convolution kernel size unchanged. We surprisingly find this special design can benefit the model for extracting the temporal features as well as improve the generalization capability. The second stage is a Temporal Transformer network, which aims to explore the long-term temporal coherence. The proposed framework is general and flexible, which can be directly trained from scratch without any pre-training models or external datasets. Extensive experiments show that our framework outperforms existing methods and remains effective when applied to detect new sorts of face forgery videos.

Temporal Coherence in Face Forgery Detection

The paper "Exploring Temporal Coherence for More General Video Face Forgery Detection" addresses a critical challenge in the domain of video-based face forgery detection: the exploitation of temporal coherence. Given that current face manipulation techniques often generate videos in a frame-by-frame manner, they struggle to maintain temporal coherence across frames, which manifests as flickering and discontinuities. This paper proposes a novel framework leveraging temporal coherence for improving the generalization and robustness of video face forgery detection methods.

Framework Overview

The authors introduce an end-to-end framework consisting of two core components: a Fully Temporal Convolution Network (FTCN) and a Temporal Transformer network. The FTCN focuses on learning temporal features by using a unique convolutional strategy—reducing spatial convolution kernel sizes to 1 and maintaining temporal kernel sizes, thus encouraging the model to rely on temporal incoherence rather than spatial anomalies for detection. This design choice is instrumental in boosting the generalization capability across different manipulation techniques.

The Temporal Transformer further enhances this by capturing long-range temporal dependencies, addressing discontinuities that might not be adjacent in the temporal sequence. This component leverages self-attention mechanisms to model such dependencies effectively.

Evaluation and Results

Extensive experiments on several datasets highlight the framework's strong performance. On the FaceForensics++ dataset, the proposed method outperforms traditional methods in terms of both precision and generalization capability. When evaluated using the leave-one-out strategy, the framework achieves substantial AUC improvements compared to prevailing techniques such as LipForensics and Face X-ray, without the need for pre-trained models or external datasets. This demonstrates its potential as a general and flexible solution to face forgery detection.

Generalization and Robustness

Significantly, the framework exhibits excellent cross-dataset generalization, as evidenced by robust results on Celeb-DF, Deepfake Detection Challenge, FaceShifter, and DeeperForensics datasets. Moreover, the method demonstrates resilience to various perturbations such as compression, noise, and color changes, which are typical in real-world scenarios. These empirical evaluations signify its practical applicability and robustness under diverse conditions.

Implications and Future Directions

The implications of this research extend both practically and theoretically. Practically, it offers a scalable solution that can potentially be deployed in real-time detection systems without reliance on resource-intensive pre-training phases. Theoretically, this work underscores the importance of temporal dynamics over spatial artifacts, guiding future research towards models that prioritize temporal feature learning.

Looking ahead, there are several possible avenues for future work. The integration of multimodal data—such as audio cues—could enrich the temporal coherence framework further. Additionally, exploring lightweight architectures for mobile and edge deployments remains an open area of exploration. Lastly, extending this approach to detect other types of temporal anomalies across different domains could broaden its applications beyond face forgery detection.

In conclusion, the paper proposes an innovative approach to face forgery detection by focusing on temporal coherence. The empirical results demonstrate substantial improvements in generalization and robustness, laying a strong foundation for future developments in this critical area of multimedia forensics.