TalkingHeadBench: A Multi-Modal Benchmark & Analysis of Talking-Head DeepFake Detection (2505.24866v1)

Abstract: The rapid advancement of talking-head deepfake generation fueled by advanced generative models has elevated the realism of synthetic videos to a level that poses substantial risks in domains such as media, politics, and finance. However, current benchmarks for deepfake talking-head detection fail to reflect this progress, relying on outdated generators and offering limited insight into model robustness and generalization. We introduce TalkingHeadBench, a comprehensive multi-model multi-generator benchmark and curated dataset designed to evaluate the performance of state-of-the-art detectors on the most advanced generators. Our dataset includes deepfakes synthesized by leading academic and commercial models and features carefully constructed protocols to assess generalization under distribution shifts in identity and generator characteristics. We benchmark a diverse set of existing detection methods, including CNNs, vision transformers, and temporal models, and analyze their robustness and generalization capabilities. In addition, we provide error analysis using Grad-CAM visualizations to expose common failure modes and detector biases. TalkingHeadBench is hosted on https://huggingface.co/datasets/luchaoqi/TalkingHeadBench with open access to all data splits and protocols. Our benchmark aims to accelerate research towards more robust and generalizable detection models in the face of rapidly evolving generative techniques.

• The paper introduces TalkingHeadBench, a novel multi-modal benchmark utilizing state-of-the-art talking-head deepfake generators to rigorously evaluate and understand the robustness of detection systems.
• Evaluations using TalkingHeadBench reveal significant weaknesses in current deepfake detectors, showing poor generalization performance when facing new identities or synthetic video generation methods.
• TalkingHeadBench aims to drive research toward developing more robust and adaptable deepfake detection models capable of handling sophisticated, modern generation techniques and evolving alongside them.

TalkingHeadBench: A Multi-Modal Benchmark Analysis of Talking-Head DeepFake Detection

The paper introduces TalkingHeadBench, a novel benchmark specifically designed to evaluate deepfake detection systems using state-of-the-art talking-head deepfake generators. The authors emphasize the need for this benchmark due to advancements in generative models that generate highly realistic synthetic videos posing risks in sensitive domains such as politics, finance, and media. Traditional benchmarks rely on outdated generator models, which do not capture the complexities of modern deepfakes. TalkingHeadBench, therefore, offers a comprehensive testbed featuring diverse synthetic videos generated by advanced diffusion techniques and commercial models.

Core Contributions

TalkingHeadBench incorporates six academic and one commercial talking-head generators using both audio and video signals. This dataset offers numerous advantages:

• Multi-Modal and Multi-Generator Benchmark: The dataset refines the understanding of detector robustness across different synthesis methods. By assessing the generalization capabilities under distribution shifts in identities and generator characteristics, the benchmark pushes detectors to enhance their performance across varied synthetic video modalities.
• Contemporary Deepfake Methods: Unlike previous deepfake datasets, TalkingHeadBench incorporates diffusion-based techniques to synthesize the entire facial region, controlling pose and expression more comprehensively.
• Challenging Evaluation Protocols: The paper introduces protocols explicitly designed to paper the robustness and generalization of detection methods across train-test distribution shifts arising from identity and generator properties changes.

Evaluation of State-of-the-Art Detectors

The benchmark is utilized to test diverse detection methods, including CNN-based, vision transformers, and temporal models:

• Protocol Analysis: Tests reveal weaknesses in current state-of-the-art detectors when facing new generators and identities. The performance drop across protocols highlights the need for detectors to adapt beyond identity shifts.
• Failure Modes: Using Grad-CAM visualizations, notable biases and failure modes are identified, pointing out areas where existing models falter. Models often misclassify due to distractions from background features rather than focusing on facial cues.

Implications and Future Directions

The primary objective of TalkingHeadBench is to galvanize research in developing detectors capable of handling increasingly sophisticated deepfakes. Key implications include:

• Enhancing Robustness: Detectors must account for the intricacies introduced by modern diffusion-based models, requiring improved architectures, strategic training designs, and stronger evaluation schemas.
• Future Adaptations: The establishment of adaptive benchmarks that update with new generator techniques can provide real-time feedback to the community, facilitating advancements both in detection capabilities and synthetic generation methodologies.
• Practical Deployment: With the sophistication of talking-head deepfakes exemplified by the benchmark, real-world applications of detectors in critical sectors demand heightened reliability and predictive accuracy under low false-positive thresholds.

Conclusion

TalkingHeadBench represents a significant stride towards effectively mapping the landscape of facial deepfake detection. By leveraging cutting-edge synthesis methods in its data creation and establishing rigorous evaluation protocols, this benchmark provides invaluable resources necessary for advancing detector models. It anticipates future challenges and sets a precedent for evolving benchmarks, aiming to foster cross-collaboration between deepfake generation and detection communities, ultimately strengthening societal defenses against manipulative artificial media.