How Do the Hearts of Deep Fakes Beat? Deep Fake Source Detection via Interpreting Residuals with Biological Signals (2008.11363v1)

Abstract: Fake portrait video generation techniques have been posing a new threat to the society with photorealistic deep fakes for political propaganda, celebrity imitation, forged evidences, and other identity related manipulations. Following these generation techniques, some detection approaches have also been proved useful due to their high classification accuracy. Nevertheless, almost no effort was spent to track down the source of deep fakes. We propose an approach not only to separate deep fakes from real videos, but also to discover the specific generative model behind a deep fake. Some pure deep learning based approaches try to classify deep fakes using CNNs where they actually learn the residuals of the generator. We believe that these residuals contain more information and we can reveal these manipulation artifacts by disentangling them with biological signals. Our key observation yields that the spatiotemporal patterns in biological signals can be conceived as a representative projection of residuals. To justify this observation, we extract PPG cells from real and fake videos and feed these to a state-of-the-art classification network for detecting the generative model per video. Our results indicate that our approach can detect fake videos with 97.29% accuracy, and the source model with 93.39% accuracy.

• The paper presents a novel framework that leverages spatiotemporal PPG signals to achieve 97.29% accuracy in deep fake detection and 93.39% in source attribution.
• It employs a pipeline of face detection, PPG signal extraction, and CNN classification to interpret residuals linked to specific generative models.
• The approach offers practical implications for media verification and misinformation mitigation by reliably identifying the origins of deep fake videos.

Overview of Deep Fake Source Detection using Biological Signals

The paper "How Do the Hearts of Deep Fakes Beat? Deep Fake Source Detection via Interpreting Residuals with Biological Signals" presents a novel methodology within the field of computer vision and artificial intelligence for discerning the source of deep fake videos. Unlike traditional binary classification methods that merely distinguish between real and fake media, this paper introduces a framework that not only detects whether a video is fake but also identifies the specific generative model that produced it. The authors leverage the spatiotemporal inconsistencies in biological signals, specifically Photoplethysmography (PPG), to capture residual artifacts from various generative models, thereby achieving source attribution.

Methodology

The authors propose a novel system architecture that involves generating PPG cells from detected facial regions in video frames. This process involves several steps:

1. Face Detection and ROI Extraction: The system first applies face detection techniques to identify facial regions of interest (ROIs) that are least susceptible to movement and lighting variations.
2. PPG Signal Extraction: Raw PPG signals are extracted from these ROIs. The PPG captures variations in skin reflectance due to blood flow, indicative of underlying biological signals.
3. Spatiotemporal Aggregation into PPG Cells: The PPG data are organized into structured spatiotemporal blocks, termed PPG cells, which incorporate both raw signals and their frequency spectra.
4. Classification Using Neural Networks: These PPG cells are fed into convolutional neural networks (CNNs), specifically employing VGG architectures, to classify video authenticity and source effectively.

Results

The empirical studies conducted by the authors demonstrate that their approach achieves significant accuracy, detecting the authenticity of videos with a success rate of 97.29% and identifying the generative model behind the fakes with an accuracy of 93.39% on the FaceForensics++ dataset. These results are significant, reflecting the potential utility of biological signals as a discriminative feature for detecting deep fake artifacts across multiple types of generative models.

Implications and Future Prospects

The paper’s contribution lies in advancing the capability to identify not only whether content is fake but also elucidating the underlying generative model, which is crucial for understanding the propagation of misinformation and digital manipulation. The methodology extends the application of biological signals from authenticity detection to source attribution, marking a key advancement in deep fake detection research.

Practical implications of this research include its deployment in automated systems for media verification, content moderation, and security purposes. On a theoretical level, it lays the foundation for the development of more sophisticated algorithms that can handle an increasing variety of generative adversarial networks (GANs) and other AI-driven content creation tools.

Looking to the future, this framework could be expanded by incorporating additional biometric markers or advanced signal processing techniques to further enhance signature detection. Integrative approaches combining the proposed residual interpretation with traditional image analysis techniques could also be explored to boost detection capabilities, particularly for real-time applications.

In summary, this paper introduces an innovative application of biological signals in machine learning for deep fake detection and source attribution, offering a critical new tool in the arsenal against digital misinformation and identity manipulation.